Rename docs/testing/developer/design to docs/testing/developer/devguide

See https://git.opnfv.org/opnfvdocs/tree/docs/testing/testing-dev.rst

Change-Id: Iac14adf4a2876fd74972dcf6279f7022e92cb9b5
Signed-off-by: Yujun Zhang <zhang.yujunz@zte.com.cn>

7 files changed, 203 insertions, 0 deletions

diff --git a/docs/testing/developer/devguide/_assets/sequence.png b/docs/testing/developer/devguide/_assets/sequence.png
new file mode 100644
index 00000000..b15458bf
--- /dev/null
+++ b/docs/testing/developer/devguide/_assets/sequence.png
diff --git a/docs/testing/developer/devguide/_assets/standalone.png b/docs/testing/developer/devguide/_assets/standalone.png
new file mode 100644
index 00000000..50222b5b
--- /dev/null
+++ b/docs/testing/developer/devguide/_assets/standalone.png
diff --git a/docs/testing/developer/devguide/api.rst b/docs/testing/developer/devguide/api.rst
new file mode 100644
index 00000000..eb2b0d67
--- /dev/null
+++ b/docs/testing/developer/devguide/api.rst
@@ -0,0 +1,2 @@
+- Which framework has been used and why
+- How to extend to more api
diff --git a/docs/testing/developer/devguide/arch.rst b/docs/testing/developer/devguide/arch.rst
new file mode 100644
index 00000000..d95faba6
--- /dev/null
+++ b/docs/testing/developer/devguide/arch.rst
@@ -0,0 +1,24 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) 2017 ZTE Corp.
+
+
+########################
+QTIP Architecture Design
+########################
+
+In Danube, QTIP releases its standalone mode, which is also know as ``solo``:
+
+.. figure:: _assets/standalone.png
+   :alt: QTIP standalone mode
+
+The runner could be launched from CLI (command line interpreter) or API
+(application programming interface) and drives the testing jobs. The generated
+data including raw performance data and testing environment are fed to collector.
+Performance metrics will be parsed from the raw data and used for QPI calculation.
+Then the benchmark report is rendered with the benchmarking results.
+
+The execution can be detailed in the diagram below:
+
+.. figure:: _assets/sequence.png
+   :alt: QTIP execution sequence
diff --git a/docs/testing/developer/devguide/cli.rst b/docs/testing/developer/devguide/cli.rst
new file mode 100644
index 00000000..72d1fbaf
--- /dev/null
+++ b/docs/testing/developer/devguide/cli.rst
@@ -0,0 +1,92 @@
+***************************
+QTIP Command Line Interface
+***************************
+
+Abstract
+########
+
+QTIP consists of different tools(metrics) to benchmark the NFVI. These metrics fall under different NFVI
+subsystems(QPI's) such as compute, storage and network. A plan consists of one or more QPI's, depending upon how
+the end user would want to measure performance. CLI is designed to help the user, execute benchmarks and
+view respective scores.
+
+Framework
+=========
+
+QTIP CLI has been created using the Python package `Click`_, Command Line Interface Creation Kit. It has been
+chosen for number of reasons. It presents the user with a very simple yet powerful API to build complex
+applications. One of the most striking features is command nesting.
+
+As explained, QTIP consists of metrics, QPI's and plans. CLI is designed to provide interface to all
+these components. It is responsible for execution, as well as provide listing and details of each individual
+element making up these components.
+
+Design
+======
+
+CLI's entry point extends Click's built in MultiCommand class object. It provides two methods, which are
+overridden to provide custom configurations.
+
+.. code-block:: python
+
+    class QtipCli(click.MultiCommand):
+
+        def list_commands(self, ctx):
+            rv = []
+            for filename in os.listdir(cmd_folder):
+                if filename.endswith('.py') and \
+                    filename.startswith('cmd_'):
+                    rv.append(filename[4:-3])
+            rv.sort()
+            return rv
+
+        def get_command(self, ctx, name):
+            try:
+                if sys.version_info[0] == 2:
+                    name = name.encode('ascii', 'replace')
+                mod = __import__('qtip.cli.commands.cmd_' + name,
+                                 None, None, ['cli'])
+            except ImportError:
+                return
+            return mod.cli
+
+Commands and subcommands will then be loaded by the ``get_command`` method above.
+
+Extending the Framework
+=======================
+
+Framework can be easily extended, as per the users requirements. One such example can be to override the builtin
+configurations with user defined ones. These can be written in a file, loaded via a Click Context and passed
+through to all the commands.
+
+.. code-block:: python
+
+   class Context:
+
+       def __init__():
+
+           self.config = ConfigParser.ConfigParser()
+           self.config.read('path/to/configuration_file')
+
+       def get_paths():
+
+           paths = self.config.get('section', 'path')
+           return paths
+
+The above example loads configuration from user defined paths, which then need to be provided to the actual
+command definitions.
+
+.. code-block:: python
+
+   from qtip.cli.entry import Context
+
+   pass_context = click.make_pass_decorator(Context, ensure=False)
+
+   @cli.command('list', help='List the Plans')
+   @pass_context
+   def list(ctx):
+       plans = Plan.list_all(ctx.paths())
+       table = utils.table('Plans', plans)
+       click.echo(table)
+
+.. _Click: http://click.pocoo.org/5/
diff --git a/docs/testing/developer/devguide/compute-qpi.rst b/docs/testing/developer/devguide/compute-qpi.rst
new file mode 100644
index 00000000..2e5aa87c
--- /dev/null
+++ b/docs/testing/developer/devguide/compute-qpi.rst
@@ -0,0 +1,69 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) 2016 ZTE Corp.
+
+
+***********
+Compute QPI
+***********
+
+The compute QPI gives user an overall score for system compute performace.
+
+Summary
+=======
+
+The compute QPI are calibrated a ZTE `E9000 <http://www.zte.com.cn/global/products/cocloud/cloud_computing/cloud_infrastructure/cloud_hw/429552>`_ server as a baseline with score of 2500 points.
+Higher scores are better, with double the score indicating double the performance.
+The compute QPI provides three different kinds of scores:
+
+* Workload Scores
+* Section Scores
+* Compute QPI Scores
+
+Baseline
+========
+
+ZTE E9000 server with an 2 Deca core Intel Xeon CPU processor,128560.0MB Memory.
+
+Workload Scores
+===============
+
+Each time a workload is executed QTIP calculates a score based on the computer's performance
+compared to the baseline performance.
+
+Section Scores
+==============
+
+QTIP uses a number of different tests, or workloads, to measure performance.
+The workloads are divided into five different sections:
+
++-----------------+--------------------------------------------------------------+------------------------------------------+
+| Section         | Detail                                                       | Indication                               |
++=================+==============================================================+==========================================+
+| Integer         | Integer workloads measure the integer instruction performace | All app relies on integer                |
+|                 | of host or vm by performing Dhrystone test.                  | performance                              |
++-----------------+--------------------------------------------------------------+------------------------------------------+
+| Floating point  | Floating point workloads measure the floating pointperfo     | Floating point performance is especially |
+|                 | rmance by performing Whetstone test.                         | important in video games,digital content |
+|                 |                                                              | creation applications.                   |
++-----------------+--------------------------------------------------------------+------------------------------------------+
+| Memory          | Memory workloads measure memory bandwidth by performing      | Software working with cipher large       |
+|                 | RamSpeed test.                                               | amounts data relies on SSL Performace.   |
++-----------------+--------------------------------------------------------------+------------------------------------------+
+| DPI             | DPI workloads measure deep-packet inspection speed by        | Software working with network packet     |
+|                 | performing nDPI test.                                        | anlysis relies on DPI performance.       |
++-----------------+--------------------------------------------------------------+------------------------------------------+
+| SSL             | SSL Performance workloads measure cipher speeds by           | Software working with cipher large       |
+|                 | using the OpenSSL tool.                                      | amounts data relies on SSL Performace    |
++-----------------+--------------------------------------------------------------+------------------------------------------+
+
+A section score is the `geometric mean <https://en.wikipedia.org/wiki/Geometric_mean>`_ of all the workload scores for workloads
+that are part of the section. These scores are useful for determining the performance of
+the computer in a particular area.
+
+Compute QPI Scores
+==================
+
+The compute QPI score is the `weighted arithmetic mean <https://en.wikipedia.org/wiki/Weighted_arithmetic_mean>`_ of the five section scores.
+The compute QPI score provides a way to quickly compare performance across different
+computers and different platforms without getting bogged down in details.
diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst
new file mode 100644
index 00000000..2b4bd9b0
--- /dev/null
+++ b/docs/testing/developer/devguide/index.rst
@@ -0,0 +1,16 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) 2016 ZTE Corp.
+
+
+##########################
+QTIP Design Specifications
+##########################
+
+.. toctree::
+   :maxdepth: 2
+
+   arch.rst
+   cli.rst
+   api.rst
+   compute-qpi.rst