Merge "Add spec cpu2006 test case"

1 files changed, 133 insertions, 0 deletions

diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc078.rst b/docs/testing/user/userguide/opnfv_yardstick_tc078.rst
new file mode 100644
index 000000000..560d55e07
--- /dev/null
+++ b/docs/testing/user/userguide/opnfv_yardstick_tc078.rst
@@ -0,0 +1,133 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Huawei Technologies Co.,Ltd and others.
+
+*************************************
+Yardstick Test Case Description TC078
+*************************************
+
+.. _spec_cpu2006: https://www.spec.org/cpu2006/
+
++-----------------------------------------------------------------------------+
+|Compute Performance                                                          |
+|                                                                             |
++--------------+--------------------------------------------------------------+
+|test case id  | OPNFV_YARDSTICK_TC078_SPEC CPU 2006                          |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|metric        | compute-intensive performance                                |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|test purpose  | The purpose of TC078 is to evaluate the IaaS compute         |
+|              | performance by using SPEC CPU 2006 benchmark. The SPEC CPU   |
+|              | 2006 benchmark has several different ways to measure         |
+|              | computer performance. One way is to measure how fast the     |
+|              | computer completes a single task; this is called a speed     |
+|              | measurement. Another way is to measure how many tasks        |
+|              | computer can accomplish in a certain amount of time; this is |
+|              | called a throughput, capacity or rate measurement.           |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|test tool     | SPEC CPU 2006                                                |
+|              |                                                              |
+|              | The SPEC CPU 2006 benchmark is SPEC's industry-standardized, |
+|              | CPU-intensive benchmark suite, stressing a system's          |
+|              | processor, memory subsystem and compiler. This benchmark     |
+|              | suite includes the SPECint benchmarks and the SPECfp         |
+|              | benchmarks. The SPECint 2006 benchmark contains 12 different |
+|              | enchmark tests and the SPECfp 2006 benchmark contains 19     |
+|              | different benchmark tests.                                   |
+|              |                                                              |
+|              | SPEC CPU 2006 is not always part of a Linux distribution.    |
+|              | SPEC requires that users purchase a license and agree with   |
+|              | their terms and conditions. For this test case, users must   |
+|              | manually download cpu2006-1.2.iso from the SPEC website and  |
+|              | save it under the yardstick/resources folder (e.g. /home/    |
+|              | opnfv/repos/yardstick/yardstick/resources/cpu2006-1.2.iso)   |
+|              | SPEC CPU® 2006 benchmark is available for purchase via the   |
+|              | SPEC order form (https://www.spec.org/order.html).           |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|test          | This test case uses SPEC CPU 2006 benchmark to measure       |
+|description   | compute-intensive performance of hosts.                      |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|configuration | file: spec_cpu.yaml (in the 'samples' directory)             |
+|              |                                                              |
+|              | benchmark_subset is set to int.                              |
+|              |                                                              |
+|              | SLA is not available in this test case.                      |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|applicability | Test can be configured with different:                       |
+|              |                                                              |
+|              |  * benchmark_subset - a subset of SPEC CPU2006 benchmarks to |
+|              |    run;                                                      |
+|              |  * SPECint_benchmark - a SPECint benchmark to run;           |
+|              |  * SPECint_benchmark - a SPECfp benchmark to run;            |
+|              |  * output_format - desired report format;                    |
+|              |  * runspec_config - SPEC CPU2006 config file provided to the |
+|              |    runspec binary;                                           |
+|              |  * runspec_iterations - the number of benchmark iterations   |
+|              |    to execute. For a reportable run, must be 3;              |
+|              |  * runspec_tune - tuning to use (base, peak, or all). For a  |
+|              |    reportable run, must be either base or all. Reportable    |
+|              |    runs do base first, then (optionally) peak;               |
+|              |  * runspec_size - size of input data to run (test, train, or |
+|              |    ref). Reportable runs ensure that your binaries can       |
+|              |    produce correct results with the test and train workloads |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|usability     | This test case is used for executing SPEC CPU 2006 benchmark |
+|              | physical servers. The SPECint 2006 benchmark takes           |
+|              | approximately 5 hours.                                       |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|references    | spec_cpu2006_                                                |
+|              |                                                              |
+|              | ETSI-NFV-TST001                                              |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|pre-test      | To run and install SPEC CPU2006, the following are required: |
+|conditions    |  * For SPECint2006: Both C99 and C++98 compilers;            |
+|              |  * For SPECfp2006: All three of C99, C++98 and Fortran-95    |
+|              |    compilers;                                                |
+|              |  * At least 8GB of disk space availabile on the system.      |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|test sequence | description and expected result                              |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|step 1        | cpu2006-1.2.iso has been saved under the yardstick/resources |
+|              | folder (e.g. /home/opnfv/repos/yardstick/yardstick/resources |
+|              | /cpu2006-1.2.iso). Additional, to use your custom runspec    |
+|              | config file you can save it under the yardstick/resources/   |
+|              | files folder and specify the config file name in the         |
+|              | runspec_config parameter.                                    |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|step 2        | Upload SPEC CPU2006 ISO to the target server and install     |
+|              | SPEC CPU2006 via ansible.                                    |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|step 3        | Yardstick is connected with the target server by using ssh.  |
+|              | If custom runspec config file is used, this file is copyied  |
+|              | from yardstick to the target server via the ssh tunnel.      |
+--------------+---------------------------------------------------------------+
+|step 4        | SPEC CPU2006 benchmark is invoked and SPEC CPU 2006 metrics  |
+|              | are generated.                                               |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|step 5        | Text, HTML, CSV, PDF, and Configuration file outputs for the |
+|              | SPEC CPU 2006 metrics are fetch from the server and stored   |
+|              | under /tmp/result folder.                                    |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|step 6        | uninstall SPEC CPU2006 and remove cpu2006-1.2.iso from the   |
+|              | target server .                                              |
+|              |                                                              |
++--------------+--------------------------------------------------------------+
+|test verdict  | None. SPEC CPU2006 results are collected and stored.         |
+|              |                                                              |
++--------------+--------------------------------------------------------------+