Merge "enhance test cases description and provide more info(in progress)"

1 files changed, 94 insertions, 22 deletions

diff --git a/docs/userguide/opnfv_yardstick_tc010.rst b/docs/userguide/opnfv_yardstick_tc010.rst
index ab793de76..202307de6 100644
--- a/docs/userguide/opnfv_yardstick_tc010.rst
+++ b/docs/userguide/opnfv_yardstick_tc010.rst
@@ -7,21 +7,71 @@
 Yardstick Test Case Description TC010
 *************************************
 
-.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/lat_mem_rd.8.html
+.. _lat_mem_rd: http://manpages.ubuntu.com/manpages/trusty/lat_mem_rd.8.html
 
 +-----------------------------------------------------------------------------+
 |Memory Latency                                                               |
 |                                                                             |
 +--------------+--------------------------------------------------------------+
-|test case id  | OPNFV_YARDSTICK_TC010_Memory Latency                         |
+|test case id  | OPNFV_YARDSTICK_TC010_MEMORY LATENCY                         |
 |              |                                                              |
 +--------------+--------------------------------------------------------------+
-|metric        | Latency in nanoseconds                                       |
+|metric        | Memory read latency (nanoseconds)                            |
 |              |                                                              |
 +--------------+--------------------------------------------------------------+
-|test purpose  | Measure the memory read latency for varying memory sizes and |
-|              | strides. Whole memory hierarchy is measured including all    |
-|              | levels of cache.                                             |
+|test purpose  | The purpose of TC010 is to evaluate the IaaS compute         |
+|              | performance with regards to memory read latency.             |
+|              | It measures the memory read latency for varying memory sizes |
+|              | and strides. Whole memory hierarchy is measured.             |
+|              |                                                              |
+|              | The purpose is also to be able to spot the trends.           |
+|              | Test results, graphs and similar shall be stored for         |
+|              | comparison reasons and product evolution understanding       |
+|              | between different OPNFV versions and/or configurations.      |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|test tool     | Lmbench                                                      |
+|              |                                                              |
+|              | Lmbench is a suite of operating system microbenchmarks. This |
+|              | test uses lat_mem_rd tool from that suite including:         |
+|              |  * Context switching                                         |
+|              |  * Networking: connection establishment, pipe, TCP, UDP, and |
+|              |    RPC hot potato                                            |
+|              |  * File system creates and deletes                           |
+|              |  * Process creation                                          |
+|              |  * Signal handling                                           |
+|              |  * System call overhead                                      |
+|              |  * Memory read latency                                       |
+|              |                                                              |
+|              | (LMbench is not always part of a Linux distribution, hence   |
+|              | it needs to be installed. As an example see the              |
+|              | /yardstick/tools/ directory for how to generate a Linux      |
+|              | image with LMbench included.)                                |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|test          | LMbench lat_mem_rd benchmark measures memory read latency    |
+|description   | for varying memory sizes and strides.                        |
+|              |                                                              |
+|              | The benchmark runs as two nested loops. The outer loop is    |
+|              | the stride size. The inner loop is the array size. For each  |
+|              | array size, the benchmark creates a ring of pointers that    |
+|              | point backward one stride.Traversing the array is done by:   |
+|              |                                                              |
+|              |         p = (char **)*p;                                     |
+|              |                                                              |
+|              | in a for loop (the over head of the for loop is not          |
+|              | significant; the loop is an unrolled loop 100 loads long).   |
+|              | The size of the array varies from 512 bytes to (typically)   |
+|              | eight megabytes. For the small sizes, the cache will have an |
+|              | effect, and the loads will be much faster. This becomes much |
+|              | more apparent when the data is plotted.                      |
+|              |                                                              |
+|              | Only data accesses are measured; the instruction cache is    |
+|              | not measured.                                                |
+|              |                                                              |
+|              | The results are reported in nanoseconds per load and have    |
+|              | been verified accurate to within a few nanoseconds on an SGI |
+|              | Indy.                                                        |
 |              |                                                              |
 +--------------+--------------------------------------------------------------+
 |configuration | File: opnfv_yardstick_tc010.yaml                             |
@@ -33,20 +83,13 @@ Yardstick Test Case Description TC010
 |              | * Interval: 1 - there is 1 second delay between each         |
 |              |   iteration.                                                 |
 |              |                                                              |
-+--------------+--------------------------------------------------------------+
-|test tool     | Lmbench                                                      |
-|              |                                                              |
-|              | Lmbench is a suite of operating system microbenchmarks. This |
-|              | test uses lat_mem_rd tool from that suite.                   |
-|              | Lmbench is not always part of a Linux distribution, hence it |
-|              | needs to be installed in the test image                      |
-|              |                                                              |
-+--------------+--------------------------------------------------------------+
-|references    | man-pages_                                                   |
-|              |                                                              |
-|              | McVoy, Larry W.,and Carl Staelin. "lmbench: Portable Tools   |
-|              | for Performance Analysis." USENIX annual technical           |
-|              | conference 1996.                                             |
+|              | SLA is optional. The SLA in this test case serves as an      |
+|              | example. Considerably lower read latency is expected.        |
+|              | However, to cover most configurations, both baremetal and    |
+|              | fully virtualized  ones, this value should be possible to    |
+|              | achieve and acceptable for black box testing.                |
+|              | Many heavy IO applications start to suffer badly if the      |
+|              | read latency is higher than this.                            |
 |              |                                                              |
 +--------------+--------------------------------------------------------------+
 |applicability | Test can be configured with different:                       |
@@ -55,12 +98,21 @@ Yardstick Test Case Description TC010
 |              | * stop_size;                                                 |
 |              | * iterations and intervals.                                  |
 |              |                                                              |
-|              | There are default values for each above-mentioned option.    |
+|              | Default values exist.                                        |
 |              |                                                              |
 |              | SLA (optional) : max_latency: The maximum memory latency     |
 |              | that is accepted.                                            |
 |              |                                                              |
 +--------------+--------------------------------------------------------------+
+|usability     | This test case is one of Yardstick's generic test. Thus it   |
+|              | is runnable on most of the scenarios.                        |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|references    | LMbench lat_mem_rd_                                          |
+|              |                                                              |
+|              | ETSI-NFV-TST001                                              |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
 |pre-test      | The test case image needs to be installed into Glance        |
 |conditions    | with Lmbench included in the image.                          |
 |              |                                                              |
@@ -70,12 +122,32 @@ Yardstick Test Case Description TC010
 |test sequence | description and expected result                              |
 |              |                                                              |
 +--------------+--------------------------------------------------------------+
-|step 1        | The host is installed as client. Lmbench's lat_mem_rd tool   |
+|step 1        | The host is installed as client. LMbench's lat_mem_rd tool   |
 |              | is invoked and logs are produced and stored.                 |
 |              |                                                              |
 |              | Result: logs are stored.                                     |
 |              |                                                              |
 +--------------+--------------------------------------------------------------+
+|step 1        | A host VM with LMbench installed is booted.                  |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|step 2        | Yardstick is connected with the host VM by using ssh.        |
+|              | 'lmbench_latency_benchmark' bash script is copyied from Jump |
+|              | Host to the host VM via the ssh tunnel.                      |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|step 3        | 'lmbench_latency_benchmark' script is invoked. LMbench's     |
+|              | lat_mem_rd benchmark starts to measures memory read latency  |
+|              | for varying memory sizes and strides. Memory read latency    |
+|              | are recorded and checked against the SLA. Logs are produced  |
+|              | and stored.                                                  |
+|              |                                                              |
+|              | Result: Logs are stored.                                     |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|step 4        | The host VM is deleted.                                      |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
 |test verdict  | Test fails if the measured memory latency is above the SLA   |
 |              | value or if there is a test case execution problem.          |
 |              |                                                              |