diff options
Diffstat (limited to 'docs/testing/developer')
| -rw-r--r-- | docs/testing/developer/design/apidocs/index.rst | 13 | ||||
| -rw-r--r-- | docs/testing/developer/design/apidocs/qtip_restful_api.rst | 10 | ||||
| -rw-r--r-- | docs/testing/developer/design/cli.rst | 92 | ||||
| -rw-r--r-- | docs/testing/developer/design/compute_QPI.rst | 69 | ||||
| -rw-r--r-- | docs/testing/developer/design/dashboard.rst | 151 | ||||
| -rw-r--r-- | docs/testing/developer/design/index.rst | 15 | ||||
| -rw-r--r-- | docs/testing/developer/design/integration_with_yardstick.rst | 92 | ||||
| -rw-r--r-- | docs/testing/developer/overview/.gitkeep | 0 | ||||
| -rw-r--r-- | docs/testing/developer/requirement/.gitkeep | 0 |
9 files changed, 442 insertions, 0 deletions
diff --git a/docs/testing/developer/design/apidocs/index.rst b/docs/testing/developer/design/apidocs/index.rst new file mode 100644 index 00000000..241a2680 --- /dev/null +++ b/docs/testing/developer/design/apidocs/index.rst @@ -0,0 +1,13 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2015 Dell Inc. +.. (c) 2016 ZTE Corp. + +################ +QTIP Configguide +################ + +.. toctree:: + :maxdepth: 2 + + ./qtip_restful_api.rst diff --git a/docs/testing/developer/design/apidocs/qtip_restful_api.rst b/docs/testing/developer/design/apidocs/qtip_restful_api.rst new file mode 100644 index 00000000..06c01292 --- /dev/null +++ b/docs/testing/developer/design/apidocs/qtip_restful_api.rst @@ -0,0 +1,10 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2015 Dell Inc. +.. (c) 2016 ZTE Corp. + +**************** +Qtip restful api +**************** + +You can get all the Qtip restful api by http://restful_api.qtip.openzero.net/api/spec.html. diff --git a/docs/testing/developer/design/cli.rst b/docs/testing/developer/design/cli.rst new file mode 100644 index 00000000..72d1fbaf --- /dev/null +++ b/docs/testing/developer/design/cli.rst @@ -0,0 +1,92 @@ +*************************** +QTIP Command Line Interface +*************************** + +Abstract +######## + +QTIP consists of different tools(metrics) to benchmark the NFVI. These metrics fall under different NFVI +subsystems(QPI's) such as compute, storage and network. A plan consists of one or more QPI's, depending upon how +the end user would want to measure performance. CLI is designed to help the user, execute benchmarks and +view respective scores. + +Framework +========= + +QTIP CLI has been created using the Python package `Click`_, Command Line Interface Creation Kit. It has been +chosen for number of reasons. It presents the user with a very simple yet powerful API to build complex +applications. One of the most striking features is command nesting. + +As explained, QTIP consists of metrics, QPI's and plans. CLI is designed to provide interface to all +these components. It is responsible for execution, as well as provide listing and details of each individual +element making up these components. + +Design +====== + +CLI's entry point extends Click's built in MultiCommand class object. It provides two methods, which are +overridden to provide custom configurations. + +.. code-block:: python + + class QtipCli(click.MultiCommand): + + def list_commands(self, ctx): + rv = [] + for filename in os.listdir(cmd_folder): + if filename.endswith('.py') and \ + filename.startswith('cmd_'): + rv.append(filename[4:-3]) + rv.sort() + return rv + + def get_command(self, ctx, name): + try: + if sys.version_info[0] == 2: + name = name.encode('ascii', 'replace') + mod = __import__('qtip.cli.commands.cmd_' + name, + None, None, ['cli']) + except ImportError: + return + return mod.cli + +Commands and subcommands will then be loaded by the ``get_command`` method above. + +Extending the Framework +======================= + +Framework can be easily extended, as per the users requirements. One such example can be to override the builtin +configurations with user defined ones. These can be written in a file, loaded via a Click Context and passed +through to all the commands. + +.. code-block:: python + + class Context: + + def __init__(): + + self.config = ConfigParser.ConfigParser() + self.config.read('path/to/configuration_file') + + def get_paths(): + + paths = self.config.get('section', 'path') + return paths + +The above example loads configuration from user defined paths, which then need to be provided to the actual +command definitions. + +.. code-block:: python + + from qtip.cli.entry import Context + + pass_context = click.make_pass_decorator(Context, ensure=False) + + @cli.command('list', help='List the Plans') + @pass_context + def list(ctx): + plans = Plan.list_all(ctx.paths()) + table = utils.table('Plans', plans) + click.echo(table) + +.. _Click: http://click.pocoo.org/5/ diff --git a/docs/testing/developer/design/compute_QPI.rst b/docs/testing/developer/design/compute_QPI.rst new file mode 100644 index 00000000..2e5aa87c --- /dev/null +++ b/docs/testing/developer/design/compute_QPI.rst @@ -0,0 +1,69 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2016 ZTE Corp. + + +*********** +Compute QPI +*********** + +The compute QPI gives user an overall score for system compute performace. + +Summary +======= + +The compute QPI are calibrated a ZTE `E9000 <http://www.zte.com.cn/global/products/cocloud/cloud_computing/cloud_infrastructure/cloud_hw/429552>`_ server as a baseline with score of 2500 points. +Higher scores are better, with double the score indicating double the performance. +The compute QPI provides three different kinds of scores: + +* Workload Scores +* Section Scores +* Compute QPI Scores + +Baseline +======== + +ZTE E9000 server with an 2 Deca core Intel Xeon CPU processor,128560.0MB Memory. + +Workload Scores +=============== + +Each time a workload is executed QTIP calculates a score based on the computer's performance +compared to the baseline performance. + +Section Scores +============== + +QTIP uses a number of different tests, or workloads, to measure performance. +The workloads are divided into five different sections: + ++-----------------+--------------------------------------------------------------+------------------------------------------+ +| Section | Detail | Indication | ++=================+==============================================================+==========================================+ +| Integer | Integer workloads measure the integer instruction performace | All app relies on integer | +| | of host or vm by performing Dhrystone test. | performance | ++-----------------+--------------------------------------------------------------+------------------------------------------+ +| Floating point | Floating point workloads measure the floating pointperfo | Floating point performance is especially | +| | rmance by performing Whetstone test. | important in video games,digital content | +| | | creation applications. | ++-----------------+--------------------------------------------------------------+------------------------------------------+ +| Memory | Memory workloads measure memory bandwidth by performing | Software working with cipher large | +| | RamSpeed test. | amounts data relies on SSL Performace. | ++-----------------+--------------------------------------------------------------+------------------------------------------+ +| DPI | DPI workloads measure deep-packet inspection speed by | Software working with network packet | +| | performing nDPI test. | anlysis relies on DPI performance. | ++-----------------+--------------------------------------------------------------+------------------------------------------+ +| SSL | SSL Performance workloads measure cipher speeds by | Software working with cipher large | +| | using the OpenSSL tool. | amounts data relies on SSL Performace | ++-----------------+--------------------------------------------------------------+------------------------------------------+ + +A section score is the `geometric mean <https://en.wikipedia.org/wiki/Geometric_mean>`_ of all the workload scores for workloads +that are part of the section. These scores are useful for determining the performance of +the computer in a particular area. + +Compute QPI Scores +================== + +The compute QPI score is the `weighted arithmetic mean <https://en.wikipedia.org/wiki/Weighted_arithmetic_mean>`_ of the five section scores. +The compute QPI score provides a way to quickly compare performance across different +computers and different platforms without getting bogged down in details. diff --git a/docs/testing/developer/design/dashboard.rst b/docs/testing/developer/design/dashboard.rst new file mode 100644 index 00000000..60c4720d --- /dev/null +++ b/docs/testing/developer/design/dashboard.rst @@ -0,0 +1,151 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2016 ZTE Corp. + + +********* +Dashboard +********* + +The dashboard gives user an intuitive view of benchmark result. + +Purpose +======= + +The basic element to be displayed is QPI a.k.a. QTIP Performance Index. But it +is also important to show user + +#. How is the final score calculated? +#. Under what condition is the test plan executed? +#. How many runs of a performance tests have been executed and is there any deviation? +#. Comparison of benchmark result from different PODs or configuration + +Templates +========= + +Different board templates are created to satisfy the above requirements. + +Composition +----------- + +QTIP gives a simple score but there must be a complex formula behind it. This +view explains the composition of the QPI. + +Condition +--------- + +The condition of a benchmark result includes + +* System Under Test + + * Hardware environment + * Hypervisor version + * Operation System release version + * System Configuration + +* Test Tools + + * Release version + * Configuration + +* Test Facility + + * Laboratory + * Engineer + * Date + +Conditions that do NOT have an obvious affect on the test result may be ignored, +e.g. temperature, power supply. + +Stats +----- + +Performance tests are actually measurement of specific metrics. All measurement +comes with uncertainty. The final result is normally one or a group of metrics +calculated from many repeats. + +For each metric, the stats board shall consist of a diagram of all measured +values and a box of stats:: + + ^ +------------+ + | | count: ? | + | |average: ? | + | | min: ? | + | X | max: ? | + | XXXX XXXX X XXXXX | | + |X XX XX XX XXX XXX XX | | + | XXXXXX X XXXXX XX | | + | | | + | | | + | | | + | | | + | | | + +---------------------------------------------> +------------+ + +The type of diagram and selection of stats shall depend on what metric to show. + +Comparison +---------- + +Comparison can be done between different PODs or different configuration on the +same PODs. + +In a comparison view, the metrics are displayed in the same diagram. And the +parameters are listed side by side. + +Both common parameters and different parameters are listed. Common values are +merged to the same cell. And user may configure the view to hide common rows. + +A draft design is as following:: + + ^ + | + | + | + | XXXXXXXX + | XXX XX+-+ XXXXXXXXXX + | XXX +XXXX XXXXX + +-+XX X +--+ ++ XXXXXX +-+ + | X+-+X +----+ +-+ +----+X + |X +--+ +---+ XXXXXX X + | +-------+ X + | + | + +-----------------------------------------------------> + + +--------------------+----------------+---------------+ + | different param 1 | | | + | | | | + +-----------------------------------------------------+ + | different param 2 | | | + | | | | + +-------------------------------------+---------------+ + | common param 1 | | + | | | + +-------------------------------------+---------------+ + | different param 3 | | | + | | | | + +-------------------------------------+---------------+ + | common param 2 | | + | | | + +--------------------+--------------------------------+ + +------------+ + | HIDE COMMON| + +------------+ + +Time line +--------- + +Time line diagram for analysis of time critical performance test:: + + +-----------------+-----------+-------------+-------------+-----+ + | | | | | | + +-----------------> | | | | + | +-----------> | | | + | ? ms +-------------> | | + | ? ms +------------>+ | + | ? ms ? ms | + | | + +---------------------------------------------------------------+ + +The time cost between checkpoints shall be displayed in the diagram. diff --git a/docs/testing/developer/design/index.rst b/docs/testing/developer/design/index.rst new file mode 100644 index 00000000..b6dd0c01 --- /dev/null +++ b/docs/testing/developer/design/index.rst @@ -0,0 +1,15 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2016 ZTE Corp. + + +########################## +QTIP Design Specifications +########################## + +.. toctree:: + :maxdepth: 2 + + dashboard.rst + compute_QPI.rst + integration_with_yardstick.rst diff --git a/docs/testing/developer/design/integration_with_yardstick.rst b/docs/testing/developer/design/integration_with_yardstick.rst new file mode 100644 index 00000000..a8298d6f --- /dev/null +++ b/docs/testing/developer/design/integration_with_yardstick.rst @@ -0,0 +1,92 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2016 ZTE Corp. + + +************************** +Integration with Yardstick +************************** + +Problem description +=================== + +For each specified QPI [1]_, QTIP needs to select a suite of test cases and collect +required test results. Based on these results, Qtip calculates the score. + +Proposed change +=============== +Qtip has a flexible architecture [2]_ to support different mode: standalone and agent. +It is recommended to use **agent mode** to work with existing test runners. Yardstick will +act as a runner to generate test result and trigger Qtip agent on the completion of test. + + +Work Items in Yardstick +----------------------- + +1. Create a customized suite in Yardstick + +Yardstick not only has many existing suites but also support customized suites. Qtip could +create a suite named **Qtip-PoC** in Yardstick repo to verify workflow of Qtip agent mode. + +2. Launch Qtip in Yardstick + +Whether to launch Qtip will be determined by checking the existence of OS environment +variable *QTIP*. If it exists, Qtip will be launched by using Yardstick CLI +`yardstick plugin install` [3]_. + +3. Yardstick interacts with Qtip + +See +`Yardstick-Qtip+integration <https://wiki.opnfv.org/display/yardstick/Yardstick-Qtip+integration>`_ +for details. + +Work Items in Qtip +------------------ + +1. Provide an API for Yardstick to post test result and environment info + +After completing test execution, Yardstick will post test result and enviroment info with +JSON format via QTIP API. See +`Yardstick-Qtip+integration <https://wiki.opnfv.org/display/yardstick/Yardstick-Qtip+integration>`_ +for details. + +2. Parse yardstick test result + +When Qtip agent receive Yarstick test result and enviroment info, Qtip agent will extract +metrics which is definded in metric spec configuration file. Based on these metrics, Qtip +agent will caculate QPI. + +3. Provide an API for querying QPI + +Qtip will provide an API for querying QPI. See +`Yardstick-Qtip+integration <https://wiki.opnfv.org/display/yardstick/Yardstick-Qtip+integration>`_ +for details. + +Implementation +============== + +Assignee(s) +----------- + +*Primary assignee:* + wu.zhihui + +*Other contributors* + TBD + +Testing +======= + +The changes will be covered by new unit test. + +Documentation +============= + +TBD + +Reference +========= + +.. [1] QTIP performance index +.. [2] https://wiki.opnfv.org/display/qtip/Architecture +.. [3] https://wiki.opnfv.org/display/yardstick/How+to+install+a+plug-in+into+Yardstick diff --git a/docs/testing/developer/overview/.gitkeep b/docs/testing/developer/overview/.gitkeep new file mode 100644 index 00000000..e69de29b --- /dev/null +++ b/docs/testing/developer/overview/.gitkeep diff --git a/docs/testing/developer/requirement/.gitkeep b/docs/testing/developer/requirement/.gitkeep new file mode 100644 index 00000000..e69de29b --- /dev/null +++ b/docs/testing/developer/requirement/.gitkeep |