Merge "Yardstick Preliminary Documentation"

1 files changed, 135 insertions, 0 deletions

diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc012.rst b/docs/testing/user/userguide/opnfv_yardstick_tc012.rst
new file mode 100644
index 000000000..b56e829f5
--- /dev/null
+++ b/docs/testing/user/userguide/opnfv_yardstick_tc012.rst
@@ -0,0 +1,135 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Ericsson AB and others.
+
+*************************************
+Yardstick Test Case Description TC012
+*************************************
+
+.. _bw_mem: http://manpages.ubuntu.com/manpages/trusty/bw_mem.8.html
+
++-----------------------------------------------------------------------------+
+|Memory Bandwidth                                                             |
+|                                                                             |
++--------------+--------------------------------------------------------------+
+|test case id  | OPNFV_YARDSTICK_TC012_MEMORY BANDWIDTH                       |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|metric        | Memory read/write bandwidth (MBps)                           |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|test purpose  | The purpose of TC012 is to evaluate the IaaS compute         |
+|              | performance with regards to memory throughput.               |
+|              | It measures the rate at which data can be read from and      |
+|              | written to the memory (this includes all levels of memory).  |
+|              |                                                              |
+|              | The purpose is also to be able to spot the trends.           |
+|              | Test results, graphs and similar shall be stored for         |
+|              | comparison reasons and product evolution understanding       |
+|              | between different OPNFV versions and/or configurations.      |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|test tool     | LMbench                                                      |
+|              |                                                              |
+|              | LMbench is a suite of operating system microbenchmarks.      |
+|              | This test uses bw_mem tool from that suite including:        |
+|              |  * Cached file read                                          |
+|              |  * Memory copy (bcopy)                                       |
+|              |  * Memory read                                               |
+|              |  * Memory write                                              |
+|              |  * Pipe                                                      |
+|              |  * TCP                                                       |
+|              |                                                              |
+|              | (LMbench is not always part of a Linux distribution, hence   |
+|              | it needs to be installed. As an example see the              |
+|              | /yardstick/tools/ directory for how to generate a Linux      |
+|              | image with LMbench included.)                                |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|test          | LMbench bw_mem benchmark allocates twice the specified       |
+|description   | amount of memory, zeros it, and then times the copying of    |
+|              | the first half to the second half. The benchmark is invoked  |
+|              | in a host VM on a compute blade. Results are reported in     |
+|              | megabytes moved per second.                                  |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|configuration | File: opnfv_yardstick_tc012.yaml                             |
+|              |                                                              |
+|              | * SLA (optional): 15000 (MBps) min_bw: The minimum amount of |
+|              |   memory bandwidth that is accepted.                         |
+|              | * Size: 10 240 kB - test allocates twice that size           |
+|              |   (20 480kB) zeros it and then measures the time it takes to |
+|              |   copy from one side to another.                             |
+|              | * Benchmark: rdwr - measures the time to read data into      |
+|              |   memory and then write data to the same location.           |
+|              | * Warmup: 0 - the number of iterations to perform before     |
+|              |   taking actual measurements.                                |
+|              | * Iterations: 10 - test is run 10 times iteratively.         |
+|              | * Interval: 1 - there is 1 second delay between each         |
+|              |   iteration.                                                 |
+|              |                                                              |
+|              | SLA is optional. The SLA in this test case serves as an      |
+|              | example. Considerably higher bandwidth is expected.          |
+|              | However, to cover most configurations, both baremetal and    |
+|              | fully virtualized  ones, this value should be possible to    |
+|              | achieve and acceptable for black box testing.                |
+|              | Many heavy IO applications start to suffer badly if the      |
+|              | read/write bandwidths are lower than this.                   |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|applicability | Test can be configured with different:                       |
+|              |                                                              |
+|              |  * memory sizes;                                             |
+|              |  * memory operations (such as rd, wr, rdwr, cp, frd, fwr,    |
+|              |    fcp, bzero, bcopy);                                       |
+|              |  * number of warmup iterations;                              |
+|              |  * iterations and intervals.                                 |
+|              |                                                              |
+|              | Default values exist.                                        |
+|              |                                                              |
+|              | SLA (optional) : min_bandwidth: The minimun memory bandwidth |
+|              | that is accepted.                                            |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|usability     | This test case is one of Yardstick's generic test. Thus it   |
+|              | is runnable on most of the scenarios.                        |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|references    | LMbench bw_mem_                                              |
+|              |                                                              |
+|              | ETSI-NFV-TST001                                              |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|pre-test      | The test case image needs to be installed into Glance        |
+|conditions    | with Lmbench included in the image.                          |
+|              |                                                              |
+|              | No POD specific requirements have been identified.           |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|test sequence | description and expected result                              |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|step 1        | A host VM with LMbench installed is booted.                  |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|step 2        | Yardstick is connected with the host VM by using ssh.        |
+|              | "lmbench_bandwidth_benchmark" bash script is copied from     |
+|              | Jump Host to the host VM via ssh tunnel.                     |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|step 3        | 'lmbench_bandwidth_benchmark' script is invoked. LMbench's   |
+|              | bw_mem benchmark starts to measures memory read/write        |
+|              | bandwidth. Memory read/write bandwidth results are recorded  |
+|              | and checked against the SLA. Logs are produced and stored.   |
+|              |                                                              |
+|              | Result: Logs are stored.                                     |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|step 4        | The host VM is deleted.                                      |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|test verdict  | Test fails if the measured memory bandwidth is below the SLA |
+|              | value or if there is a test case execution problem.          |
+|              |                                                              |
++--------------+--------------------------------------------------------------+