diff options
Diffstat (limited to 'docs/testing')
| -rw-r--r-- | docs/testing/.gitkeep | 0 | ||||
| -rw-r--r-- | docs/testing/developer/design/apidocs/index.rst | 13 | ||||
| -rw-r--r-- | docs/testing/developer/design/apidocs/qtip_restful_api.rst | 10 | ||||
| -rw-r--r-- | docs/testing/developer/design/compute_QPI.rst | 69 | ||||
| -rw-r--r-- | docs/testing/developer/design/dashboard.rst | 151 | ||||
| -rw-r--r-- | docs/testing/developer/design/integration_with_yardstick.rst | 92 | ||||
| -rw-r--r-- | docs/testing/user/.gitkeep | 0 | ||||
| -rw-r--r-- | docs/testing/user/installation/.gitkeep | 0 | ||||
| -rw-r--r-- | docs/testing/user/scenarios/.gitkeep | 0 | ||||
| -rw-r--r-- | docs/testing/user/userguide/_01-compute.rst | 104 | ||||
| -rw-r--r-- | docs/testing/user/userguide/_02-network.rst | 61 | ||||
| -rw-r--r-- | docs/testing/user/userguide/_03-storage.rst | 31 | ||||
| -rw-r--r-- | docs/testing/user/userguide/_testcase_description.rst | 46 | ||||
| -rw-r--r-- | docs/testing/user/userguide/annex.rst | 18 | ||||
| -rw-r--r-- | docs/testing/user/userguide/benchmark-suites.rst | 15 | ||||
| -rw-r--r-- | docs/testing/user/userguide/introduction.rst | 381 |
16 files changed, 0 insertions, 991 deletions
diff --git a/docs/testing/.gitkeep b/docs/testing/.gitkeep deleted file mode 100644 index e69de29b..00000000 --- a/docs/testing/.gitkeep +++ /dev/null diff --git a/docs/testing/developer/design/apidocs/index.rst b/docs/testing/developer/design/apidocs/index.rst deleted file mode 100644 index 241a2680..00000000 --- a/docs/testing/developer/design/apidocs/index.rst +++ /dev/null @@ -1,13 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2015 Dell Inc. -.. (c) 2016 ZTE Corp. - -################ -QTIP Configguide -################ - -.. toctree:: - :maxdepth: 2 - - ./qtip_restful_api.rst diff --git a/docs/testing/developer/design/apidocs/qtip_restful_api.rst b/docs/testing/developer/design/apidocs/qtip_restful_api.rst deleted file mode 100644 index 06c01292..00000000 --- a/docs/testing/developer/design/apidocs/qtip_restful_api.rst +++ /dev/null @@ -1,10 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2015 Dell Inc. -.. (c) 2016 ZTE Corp. - -**************** -Qtip restful api -**************** - -You can get all the Qtip restful api by http://restful_api.qtip.openzero.net/api/spec.html. diff --git a/docs/testing/developer/design/compute_QPI.rst b/docs/testing/developer/design/compute_QPI.rst deleted file mode 100644 index 2e5aa87c..00000000 --- a/docs/testing/developer/design/compute_QPI.rst +++ /dev/null @@ -1,69 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2016 ZTE Corp. - - -*********** -Compute QPI -*********** - -The compute QPI gives user an overall score for system compute performace. - -Summary -======= - -The compute QPI are calibrated a ZTE `E9000 <http://www.zte.com.cn/global/products/cocloud/cloud_computing/cloud_infrastructure/cloud_hw/429552>`_ server as a baseline with score of 2500 points. -Higher scores are better, with double the score indicating double the performance. -The compute QPI provides three different kinds of scores: - -* Workload Scores -* Section Scores -* Compute QPI Scores - -Baseline -======== - -ZTE E9000 server with an 2 Deca core Intel Xeon CPU processor,128560.0MB Memory. - -Workload Scores -=============== - -Each time a workload is executed QTIP calculates a score based on the computer's performance -compared to the baseline performance. - -Section Scores -============== - -QTIP uses a number of different tests, or workloads, to measure performance. -The workloads are divided into five different sections: - -+-----------------+--------------------------------------------------------------+------------------------------------------+ -| Section | Detail | Indication | -+=================+==============================================================+==========================================+ -| Integer | Integer workloads measure the integer instruction performace | All app relies on integer | -| | of host or vm by performing Dhrystone test. | performance | -+-----------------+--------------------------------------------------------------+------------------------------------------+ -| Floating point | Floating point workloads measure the floating pointperfo | Floating point performance is especially | -| | rmance by performing Whetstone test. | important in video games,digital content | -| | | creation applications. | -+-----------------+--------------------------------------------------------------+------------------------------------------+ -| Memory | Memory workloads measure memory bandwidth by performing | Software working with cipher large | -| | RamSpeed test. | amounts data relies on SSL Performace. | -+-----------------+--------------------------------------------------------------+------------------------------------------+ -| DPI | DPI workloads measure deep-packet inspection speed by | Software working with network packet | -| | performing nDPI test. | anlysis relies on DPI performance. | -+-----------------+--------------------------------------------------------------+------------------------------------------+ -| SSL | SSL Performance workloads measure cipher speeds by | Software working with cipher large | -| | using the OpenSSL tool. | amounts data relies on SSL Performace | -+-----------------+--------------------------------------------------------------+------------------------------------------+ - -A section score is the `geometric mean <https://en.wikipedia.org/wiki/Geometric_mean>`_ of all the workload scores for workloads -that are part of the section. These scores are useful for determining the performance of -the computer in a particular area. - -Compute QPI Scores -================== - -The compute QPI score is the `weighted arithmetic mean <https://en.wikipedia.org/wiki/Weighted_arithmetic_mean>`_ of the five section scores. -The compute QPI score provides a way to quickly compare performance across different -computers and different platforms without getting bogged down in details. diff --git a/docs/testing/developer/design/dashboard.rst b/docs/testing/developer/design/dashboard.rst deleted file mode 100644 index 60c4720d..00000000 --- a/docs/testing/developer/design/dashboard.rst +++ /dev/null @@ -1,151 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2016 ZTE Corp. - - -********* -Dashboard -********* - -The dashboard gives user an intuitive view of benchmark result. - -Purpose -======= - -The basic element to be displayed is QPI a.k.a. QTIP Performance Index. But it -is also important to show user - -#. How is the final score calculated? -#. Under what condition is the test plan executed? -#. How many runs of a performance tests have been executed and is there any deviation? -#. Comparison of benchmark result from different PODs or configuration - -Templates -========= - -Different board templates are created to satisfy the above requirements. - -Composition ------------ - -QTIP gives a simple score but there must be a complex formula behind it. This -view explains the composition of the QPI. - -Condition ---------- - -The condition of a benchmark result includes - -* System Under Test - - * Hardware environment - * Hypervisor version - * Operation System release version - * System Configuration - -* Test Tools - - * Release version - * Configuration - -* Test Facility - - * Laboratory - * Engineer - * Date - -Conditions that do NOT have an obvious affect on the test result may be ignored, -e.g. temperature, power supply. - -Stats ------ - -Performance tests are actually measurement of specific metrics. All measurement -comes with uncertainty. The final result is normally one or a group of metrics -calculated from many repeats. - -For each metric, the stats board shall consist of a diagram of all measured -values and a box of stats:: - - ^ +------------+ - | | count: ? | - | |average: ? | - | | min: ? | - | X | max: ? | - | XXXX XXXX X XXXXX | | - |X XX XX XX XXX XXX XX | | - | XXXXXX X XXXXX XX | | - | | | - | | | - | | | - | | | - | | | - +---------------------------------------------> +------------+ - -The type of diagram and selection of stats shall depend on what metric to show. - -Comparison ----------- - -Comparison can be done between different PODs or different configuration on the -same PODs. - -In a comparison view, the metrics are displayed in the same diagram. And the -parameters are listed side by side. - -Both common parameters and different parameters are listed. Common values are -merged to the same cell. And user may configure the view to hide common rows. - -A draft design is as following:: - - ^ - | - | - | - | XXXXXXXX - | XXX XX+-+ XXXXXXXXXX - | XXX +XXXX XXXXX - +-+XX X +--+ ++ XXXXXX +-+ - | X+-+X +----+ +-+ +----+X - |X +--+ +---+ XXXXXX X - | +-------+ X - | - | - +-----------------------------------------------------> - - +--------------------+----------------+---------------+ - | different param 1 | | | - | | | | - +-----------------------------------------------------+ - | different param 2 | | | - | | | | - +-------------------------------------+---------------+ - | common param 1 | | - | | | - +-------------------------------------+---------------+ - | different param 3 | | | - | | | | - +-------------------------------------+---------------+ - | common param 2 | | - | | | - +--------------------+--------------------------------+ - +------------+ - | HIDE COMMON| - +------------+ - -Time line ---------- - -Time line diagram for analysis of time critical performance test:: - - +-----------------+-----------+-------------+-------------+-----+ - | | | | | | - +-----------------> | | | | - | +-----------> | | | - | ? ms +-------------> | | - | ? ms +------------>+ | - | ? ms ? ms | - | | - +---------------------------------------------------------------+ - -The time cost between checkpoints shall be displayed in the diagram. diff --git a/docs/testing/developer/design/integration_with_yardstick.rst b/docs/testing/developer/design/integration_with_yardstick.rst deleted file mode 100644 index a8298d6f..00000000 --- a/docs/testing/developer/design/integration_with_yardstick.rst +++ /dev/null @@ -1,92 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2016 ZTE Corp. - - -************************** -Integration with Yardstick -************************** - -Problem description -=================== - -For each specified QPI [1]_, QTIP needs to select a suite of test cases and collect -required test results. Based on these results, Qtip calculates the score. - -Proposed change -=============== -Qtip has a flexible architecture [2]_ to support different mode: standalone and agent. -It is recommended to use **agent mode** to work with existing test runners. Yardstick will -act as a runner to generate test result and trigger Qtip agent on the completion of test. - - -Work Items in Yardstick ------------------------ - -1. Create a customized suite in Yardstick - -Yardstick not only has many existing suites but also support customized suites. Qtip could -create a suite named **Qtip-PoC** in Yardstick repo to verify workflow of Qtip agent mode. - -2. Launch Qtip in Yardstick - -Whether to launch Qtip will be determined by checking the existence of OS environment -variable *QTIP*. If it exists, Qtip will be launched by using Yardstick CLI -`yardstick plugin install` [3]_. - -3. Yardstick interacts with Qtip - -See -`Yardstick-Qtip+integration <https://wiki.opnfv.org/display/yardstick/Yardstick-Qtip+integration>`_ -for details. - -Work Items in Qtip ------------------- - -1. Provide an API for Yardstick to post test result and environment info - -After completing test execution, Yardstick will post test result and enviroment info with -JSON format via QTIP API. See -`Yardstick-Qtip+integration <https://wiki.opnfv.org/display/yardstick/Yardstick-Qtip+integration>`_ -for details. - -2. Parse yardstick test result - -When Qtip agent receive Yarstick test result and enviroment info, Qtip agent will extract -metrics which is definded in metric spec configuration file. Based on these metrics, Qtip -agent will caculate QPI. - -3. Provide an API for querying QPI - -Qtip will provide an API for querying QPI. See -`Yardstick-Qtip+integration <https://wiki.opnfv.org/display/yardstick/Yardstick-Qtip+integration>`_ -for details. - -Implementation -============== - -Assignee(s) ------------ - -*Primary assignee:* - wu.zhihui - -*Other contributors* - TBD - -Testing -======= - -The changes will be covered by new unit test. - -Documentation -============= - -TBD - -Reference -========= - -.. [1] QTIP performance index -.. [2] https://wiki.opnfv.org/display/qtip/Architecture -.. [3] https://wiki.opnfv.org/display/yardstick/How+to+install+a+plug-in+into+Yardstick diff --git a/docs/testing/user/.gitkeep b/docs/testing/user/.gitkeep deleted file mode 100644 index e69de29b..00000000 --- a/docs/testing/user/.gitkeep +++ /dev/null diff --git a/docs/testing/user/installation/.gitkeep b/docs/testing/user/installation/.gitkeep deleted file mode 100644 index e69de29b..00000000 --- a/docs/testing/user/installation/.gitkeep +++ /dev/null diff --git a/docs/testing/user/scenarios/.gitkeep b/docs/testing/user/scenarios/.gitkeep deleted file mode 100644 index e69de29b..00000000 --- a/docs/testing/user/scenarios/.gitkeep +++ /dev/null diff --git a/docs/testing/user/userguide/_01-compute.rst b/docs/testing/user/userguide/_01-compute.rst deleted file mode 100644 index 56be5488..00000000 --- a/docs/testing/user/userguide/_01-compute.rst +++ /dev/null @@ -1,104 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2015 Dell Inc. -.. (c) 2016 ZTE Corp. - - -Compute Suite -============= - -Introduction ------------- - -The QTIP testing suite aims to benchmark the compute components of an OPNFV platform. -Such components include, the CPU performance, the memory performance. -Additionally virtual computing performance provided by the Hypervisor (KVM) installed as part of OPNFV platforms would be benchmarked too. - -The test suite consists of both synthetic and application specific benchmarks to test compute components. - -All the compute benchmarks could be run in 2 scenarios: - -1. On Baremetal Machines provisioned by an OPNFV installer (Host machines) -2. On Virtual Machines brought up through OpenStack on an OPNFV platform - -Note: The Compute benchmank suite constains relatively old benchmarks such as dhrystone and whetstone. The suite would be updated for better benchmarks such as Linbench for the OPNFV C release. - -Benchmarks ----------- - -The benchmarks include: - -Dhrystone 2.1 -^^^^^^^^^^^^^^^^ - -Dhrystone is a synthetic benchmark for measuring CPU performance. It uses integer calculations to evaluate CPU capabilities. -Both Single CPU performance is measured along multi-cpu performance. - - -Dhrystone, however, is a dated benchmark and has some short comings. -Written in C, it is a small program that doesn't test the CPU memory subsystem. -Additionally, dhrystone results could be modified by optimizing the compiler and insome cases hardware configuration. - -References: http://www.eembc.org/techlit/datasheets/dhrystone_wp.pdf - -Whetstone -^^^^^^^^^^^^ - -Whetstone is a synthetic benchmark to measure CPU floating point operation performance. -Both Single CPU performance is measured along multi-cpu performance. - -Like Dhrystone, Whetstone is a dated benchmark and has short comings. - -References: - -http://www.netlib.org/benchmark/whetstone.c - -OpenSSL Speed -^^^^^^^^^^^^^^^^ - -OpenSSL Speed can be used to benchmark compute performance of a machine. In QTIP, two OpenSSL Speed benchmarks are incorporated: -1. RSA signatunes/sec signed by a machine -2. AES 128-bit encryption throughput for a machine for cipher block sizes - -References: - -https://www.openssl.org/docs/manmaster/apps/speed.html - -RAMSpeed -^^^^^^^^ - -RAMSpeed is used to measure a machine's memory perfomace. -The problem(array)size is large enough to ensure Cache Misses so that the main machine memory is used. -INTmem and FLOATmem benchmarks are executed in 4 different scenarios: - -a. Copy: a(i)=b(i) -b. Add: a(i)=b(i)+c(i) -c. Scale: a(i)=b(i)*d -d. Tniad: a(i)=b(i)+c(i)*d - -INTmem uses integers in these four benchmarks whereas FLOATmem uses floating points for these benchmarks. - -References: - -http://alasir.com/software/ramspeed/ - -https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/Untangling+memory+access+measurements - -DPI -^^^ - -nDPI is a modified variant of OpenDPI, Open source Deep packet Inspection, that is maintained by ntop. -An example application called *pcapreader* has been developed and is available for use along nDPI. - -A sample .pcap file is passed to the *pcapreader* application. -nDPI classifies traffic in the pcap file into different categories based on string matching. -The *pcapreader* application provides a throughput number for the rate at which traffic was classified, indicating a machine's computational performance. -The results are run 10 times and an average is taken for the obtained number. - -*nDPI may provide non consistent results and was added to Brahmaputra for experimental purposes* - -References: - -http://www.ntop.org/products/deep-packet-inspection/ndpi/ - -http://www.ntop.org/wp-content/uploads/2013/12/nDPI_QuickStartGuide.pdf diff --git a/docs/testing/user/userguide/_02-network.rst b/docs/testing/user/userguide/_02-network.rst deleted file mode 100644 index 00fe5b0a..00000000 --- a/docs/testing/user/userguide/_02-network.rst +++ /dev/null @@ -1,61 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2015 Dell Inc. -.. (c) 2016 ZTE Corp. - - -Network Suite -============= - -QTIP uses IPerf3 as the main tool for testing the network throughput. -There are three tests that are run through the QTIP framework. - -**1. Network throughput between two compute nodes** - -**2. Network Throughput between two VMs on the same compute node** - -**3. Network Throughput between two VMs on different compute nodes** - - -Network throughput between two compute nodes ------------------------------------------------ - -For the throughput between two compute nodes, Iperf3 is installed on the compute nodes comprising the systems-under-test. -One of the compute nodes is used as a server and the other as a client. -The client pushes traffic to the server for a duration specified by the user in the configuration file for Iperf3. - - -These files can be found in the "benchmarks/testplan/{POD}/network/" directory. -The bandwidth is limited by the physical link layer speed connecting the two compute nodes. -The result file includes the b/s bandwidth and the CPU usage for both the client and server. - -Network throughput between two VMs on the same compute node --------------------------------------------------------------- - -QTIP framework sets up a stack with a private network, security groups, routers and attaches two VMs to this network. -Iperf3 is installed on the VMs and one is assigned the role of client while the other VM serves as a server. -Traffic is pushed over the QTIP private network between the two VMs. -A closer look is needed to see how the traffic actually flows between the VMs in this configuration to understand what is happening to the packet as it traverses the OpenStack virtual network. - -The packet originates from VM1 and its sent to the Linux bridge via a tap interface where the security groups are written. -Afterwards the packet is forwarded to the Integration bridge (br-int) via a patch port. -Since VM2 is also connected to the Integration bridge in a similar manner as VM1, the packet gets forwarded to the linux bridge connecting VM2. -After the Linux bridge the packet is sent to VM2 and is received by the Iperf3 server. -Since no physical link is involved in this topology, only the OVS (Integration bridge) (br-int) is being benchmarked. - - -Network throughput between two VMs on different compute nodes --------------------------------------------------------------- - - -As in case 2, QTIP framework sets up a stack with a private network, security groups, routers, and two VMs which are attached to the created network. However, the two VMs are spawned up on different compute nodes. - -Since the VMs are spawned on different nodes, the traffic involves additional paths. - -The traffic packet leaves the client VM and makes its way to the Integration Bridge (br-int) as in the previous case through a linux bridge and a patch port. -The integration bridge (br-int) forwards the packet to the the tunneling bridge (br-tun) where the packet is encapsulated based on the tunneling protocol used (GRE/VxLAN). -The packet then moves onto the physical link through the ethernet bridge (br-eth). - -On the receiving compute node, the packet arrives at ethernet bridge(br-eth) through the physical link. -This packet then moves to the tunneling bridge (br-tun) where the packet is decapsulated. -The packet then moves onto the internal bridge (br-int) and finally moves through a patch port into the linux bridge and eventually to the VM where it is received by the Iperf server application. diff --git a/docs/testing/user/userguide/_03-storage.rst b/docs/testing/user/userguide/_03-storage.rst deleted file mode 100644 index b1490432..00000000 --- a/docs/testing/user/userguide/_03-storage.rst +++ /dev/null @@ -1,31 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2015 Dell Inc. -.. (c) 2016 ZTE Corp. - - -Storage Suite -============= - -The QTIP benchmark suite aims to evaluate storage components within an OPNFV platform. -For Brahamaputra release, FIO would evaluate File System performance for the host machine. -It will also test the I/O performance provided by the hypervisor(KVM) when Storage benchmarks are run inside VMs. - -QTIP storage test cases consist of: - -**1. FIO Job to benchmark baremetal file system performance** - -**2. FIO Job to bechmark virtual machine file system performance** - -**Note: For Brahmaputra release, only the Ephemeral Storage is being tested. For C release persistent block and object storage would be tested.** - -The FIO Job would consist of: - -1. A file size of 5GB -2. Random Read 50%, Random Write 50% -3. Direct I/O -4. Asynch I/O Engine -5. I/O Queue depth of 2 -6. Block size :4K - -For this Job, I/O per second would be measured along mean I/O latency to provide storage performance numbers. diff --git a/docs/testing/user/userguide/_testcase_description.rst b/docs/testing/user/userguide/_testcase_description.rst deleted file mode 100644 index d60ca949..00000000 --- a/docs/testing/user/userguide/_testcase_description.rst +++ /dev/null @@ -1,46 +0,0 @@ -.. Template to be used for test case descriptions in QTIP Project. - Write one .rst per test case. - Borrowed Heavily from Yardstick - Upload the .rst for the test case in /docs/ directory. - Review in Gerrit. - -Test Case Description -===================== - -+-----------------------------------------------------------------------------+ -|test case slogan e.g. Network throughput | -+==============+==============================================================+ -|test case id | e.g. qtip_throughput | -+--------------+--------------------------------------------------------------+ -|metric | what will be measured, e.g. latency | -+--------------+--------------------------------------------------------------+ -|test purpose | describe what is the purpose of the test case | -+--------------+--------------------------------------------------------------+ -|configuration | what .yaml file to use, state SLA if applicable, state | -| | test duration, list and describe the scenario options used in| -| | this TC and also list the options using default values. | -+--------------+--------------------------------------------------------------+ -|test tool | e.g. ping | -+--------------+--------------------------------------------------------------+ -|references | e.g. RFCxxx, ETSI-NFVyyy | -+--------------+--------------------------------------------------------------+ -|applicability | describe variations of the test case which can be | -| | performend, e.g. run the test for different packet sizes | -+--------------+--------------------------------------------------------------+ -|pre-test | describe configuration in the tool(s) used to perform | -|conditions | the measurements (e.g. fio, pktgen), POD-specific | -| | configuration required to enable running the test | -+--------------+------+----------------------------------+--------------------+ -|test sequence | step | description | result | -| +------+----------------------------------+--------------------+ -| | 1 | use this to describe tests that | what happens in | -| | | require several steps e.g. | this step | -| | | step 1 collect logs | e.g. logs collected| -| +------+----------------------------------+--------------------+ -| | 2 | remove interface | interface down | -| +------+----------------------------------+--------------------+ -| | N | what is done in step N | what happens | -+--------------+------+----------------------------------+--------------------+ -|test verdict | expected behavior, or SLA, pass/fail criteria | -+--------------+--------------------------------------------------------------+ - diff --git a/docs/testing/user/userguide/annex.rst b/docs/testing/user/userguide/annex.rst deleted file mode 100644 index e8bf5555..00000000 --- a/docs/testing/user/userguide/annex.rst +++ /dev/null @@ -1,18 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2016 ZTE Corp. - - -***** -Annex -***** - -.. toctree:: - :maxdepth: 2 - - _testcase_description.rst - -Downloads -========= - -- :download:`Sample configuration <../download/sample_config.yaml>` diff --git a/docs/testing/user/userguide/benchmark-suites.rst b/docs/testing/user/userguide/benchmark-suites.rst deleted file mode 100644 index 84d1c647..00000000 --- a/docs/testing/user/userguide/benchmark-suites.rst +++ /dev/null @@ -1,15 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2016 ZTE Corp. - - -**************** -Benchmark Suites -**************** - -.. toctree:: - :maxdepth: 2 - - _01-compute.rst - _02-network.rst - _03-storage.rst diff --git a/docs/testing/user/userguide/introduction.rst b/docs/testing/user/userguide/introduction.rst deleted file mode 100644 index 3147f0aa..00000000 --- a/docs/testing/user/userguide/introduction.rst +++ /dev/null @@ -1,381 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2015 Dell Inc. -.. (c) 2016 ZTE Corp. - - -************ -Introduction -************ - -This guide will serve as a first step to familiarize the user with how to -run QTIP the first time when the user pull QTIP image on to their host machine. -In order to install and config QTIP please follow the instructions in the -configuration.rst located in docs/configguide/configuration.rst. - -QTIP Directory structure -======================== - -The QTIP directory has been sectioned off into multiple folders to facilitate - segmenting information into relevant categories. The folders that concern - the end user are `benchmarks/testplan/` and `benchmarks/suite/`. - -**testplan/:** - -This folder is used to store all the config files which are used to setup the -environment prior to a test. This folder is further divided into opnfv pods -which run QTIP. Inside each pod there are folders which contain the config files -segmented based on test cases. Namely, these include, `Compute`, `Network` and -`Storage`. The default folder is there for the end user who is interested in -testing their infrastructure which is installed by fuel or compass but aren't -part of a opnfv pod,and for opnfv CI. - -The structure of the directory for the user appears as follows -:: - - testplan/default/compute - testplan/default/network - testplan/default/storage - -The benchmarks that are part of the QTIP framework are listed under these -folders. An example of the compute folder is shown below. -Their naming convention is <BENCHMARK>_<VM/BM>.yaml -:: - - dhrystone_bm.yaml - dhrystone_vm.yaml - whetstone_vm.yaml - whetstone_bm.yaml - ssl_vm.yaml - ssl_bm.yaml - ramspeed_vm.yaml - ramspeed_bm.yaml - dpi_vm.yaml - dpi_bm.yaml - -The above listed files are used to configure the environment. The VM/BM tag -distinguishes between a test to be run on the Virtual Machine or the compute -node itself, respectively. - - -**benchmarks/suite/:** - -This folder contains three files, namely `compute`, `network` and `storage`. -These files list the benchmarks are to be run by the QTIP framework. Sample -compute test file is shown below -:: - - { - "bm": [ - "dhrystone_bm.yaml", - "whetstone_bm.yaml", - "ramspeed_bm.yaml", - "dpi_bm.yaml", - "ssl_bm.yaml" - ], - "vm": [ - "dhrystone_vm.yaml", - "whetstone_vm.yaml", - "ramspeed_vm.yaml", - "dpi_vm.yaml", - "ssl_vm.yaml" - ] - } - -The compute file will now run all the benchmarks listed above one after -another on the environment. - -Preparing a config file for test: ---------------------------------- - -We will be using dhrystone as a example to list out the changes that the -user will need to do in order to run the benchmark. - -Dhrystone on Compute Nodes: -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -QTIP framework can run benchmarks on the actual compute nodes as well. In -order to run dhrystone on the compute nodes we will be editing the -dhrystone_bm.yaml file. - -:: - - Scenario: - benchmark: dhrystone - host: machine_1, machine_2 - server: - -The `Scenario` field is used by to specify the name of the benchmark to -run as done by `benchmark: dhrystone`. The `host` and `server` tag are -not used for the compute benchmarks but are included here to help the -user `IF` they wish to control the execution. By default both machine_1 -and machine_2 will have dhrystone run on them in parallel but the user -can change this so that machine_1 run dhrystone before machine_2. This -will be elaborated in the `Context` tag. - -:: - - Context: - Host_Machines: - machine_1: - ip: 10.20.0.6 - pw: - role: host - machine_2: - ip: 10.20.0.5 - pw: - role: host - - Virtual_Machines: - -The `Context` tag helps the user list the number of compute nodes they want - to run dhrystone on. The user can list all the compute nodes under the - `Host_Machines` tag. All the machines under test must be listed under the - `Host_Machines` and naming it incrementally higher. The `ip:` tag is used - to specify the IP of the particular compute node.The `ip:` tag can be left - blank when installer type is 'fuel',because QTIP will get ip - from installer. The `pw:` tag can be left blank because QTIP uses its own - key for ssh. In order to run dhrystone on one compute node at a time the user - needs to edit the `role:` tag. `role: host` for machine_1 and `role: server` - for machine_2 will allow for dhrystone to be run on machine_1 and then run - on machine_2. - -:: - - - Test_Description: - Test_category: "Compute" - Benchmark: "dhrystone" - Overview: > - ''' This test will run the dhrystone benchmark in parallel on - machine_1 and machine_2. - -The above field is purely for a description purpose to explain to the user -the working of the test and is not fed to the framework. - -Sample dhrystone_bm.yaml file: ------------------------------- -:: - - Scenario: - benchmark: dhrystone - host: machine_1, machine_2 - server: - - Context: - Host_Machines: - machine_1: - ip: 10.20.0.6 - pw: - role: host - machine_2: - ip: 10.20.0.5 - pw: - role: host - - Virtual_Machines: - - - Test_Description: - Test_category: "Compute" - Benchmark: "dhrystone" - Overview: > - ''' This test will run the dhrystone benchmark in parallel on - machine_1 and machine_2.\n - -Dhrystone on Virtual Machine: -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -To run dhrystone on the VMs we will be editing dhrystone_vm.yaml file. -Snippets on the file are given below. - -:: - - Scenario: - benchmark: dhrystone - host: virtualmachine_1, virtualmachine_2 - server: - - -The `Scenario` field is used by to specify the name of the benchmark to -run as done by `benchmark: dhrystone`. The `host` and `server` tag are -not used for the compute benchmarks but are included here to help the -user `IF` they wish to control the execution. By default both -virtualmachine_1 and virtualmachine_2 will have dhrystone run on them -in parallel but the user can change this so that virtualmachine_1 run -dhrystone before virtualmachine_2. This will be elaborated in the -`Context` tag. -:: - - Context: - Host_Machines: - - Virtual_Machines: - virtualmachine_1: - availability_zone: compute1 - public_network: 'net04_ext' - OS_image: QTIP_CentOS - flavor: m1.large - role: host - virtualmachine_2: - availability_zone: compute2 - public_network: 'net04_ext' - OS_image: QTIP_CentOS - flavor: m1.large - role: host - -The `Context` tag helps the user list the number of VMs and their -characteristic. The user can list all the VMs they want to bring up -under the `Virtual_Machines:` tag. In the above example we will be -bringing up two VMs. One on Compute1 and the other on Compute2. The -user can change this as desired `NOTE: Please ensure you have the -necessary compute nodes before listing under the 'availability_zone:' -tag`. The rest of the options do not need to be modified by the user. - -Running dhrystone sequentially (Optional): -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -In order to run dhrystone on one VM at a time the user needs to edit -the `role:` tag. `role: host` for virtualmachine_1 and `role: server` -for virtualmachine_2 will allow for dhrystone to be run on -virtualmachine_1 and then run on virtualmachine_2. - -:: - - Test_Description: - Test_category: "Compute" - Benchmark: "dhrystone" - Overview: - This test will run the dhrystone benchmark in parallel on - virtualmachine_1 and virtualmachine_2 - -The above field is purely for a decription purpose to explain to -the user the working of the test and is not fed to the framework. - -Running dhrystone with proxy (Optional): -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -In order to run the dhrystone on the hosts or vms which can only access the -public network by proxy, the user needs to add the `Proxy_Environment` info -in `Context` tag. - -:: - - Context: - Host_Machines: - machine_1: - ip: 10.20.0.29 - pw: - role: host - machine_2: - ip: 10.20.0.30 - pw: - role: host - - Virtual_Machines: - - Proxy_Environment: - http_proxy: http://10.20.0.1:8118 - https_proxy: http://10.20.0.1:8118 - no_proxy: localhost,127.0.0.1,10.20.*,192.168.* - -Sample dhrystone_vm.yaml file: ------------------------------- -:: - - Scenario: - benchmark: dhrystone - host: virtualmachine_1, virtualmachine_2 - server: - - Context: - Host_Machines: - - Virtual_Machines: - virtualmachine_1: - availability_zone: compute1 - public_network: 'net04_ext' - OS_image: QTIP_CentOS - flavor: m1.large - role: host - virtualmachine_2: - availability_zone: compute2 - public_network: 'net04_ext' - OS_image: QTIP_CentOS - flavor: m1.large - role: host - - Test_Description: - Test_category: "Compute" - Benchmark: "dhrystone" - Overview: > - This test will run the dhrystone benchmark in parallel on - machine_1 and machine_2.\n - -Commands to run the Framework: ------------------------------- - -In order to start QTIP on the default lab please use the following commands (asssuming your installer -is 'fuel' or 'compass', you use the config files in the benchmarks/testplan/default/ directory and listed the -intended suite in the benchmarks/suite/<RELEVANT-SUITE-FILE>): - -First step is to export the necessary information to the environment and generate QTIP key pair. -Please follow the instructions in the configuration.rst. - -Secondary step download the QTIP image and upload it to the Cloud.QTIP will use this image -to create VM when test VM performance. -:: - - source docker/prepare_qtip_image.sh - -Running QTIP on the using `default` as the pod name and for the `compute` suite by cli. -:: - - python qtip.py -l default -f compute - -Running QTIP on the using 'default' as the pod name and for the 'compute' suite 'bm' type by restful api. -:: - - curl --trace-ascii debug.txt -X POST -d '{ "installer_ip": "10.20.6.2","installer_type":"fuel", "suite_name":"compute", "type": "BM"}' -H "Content-Type: application/json" http://127.0.0.1:5000/api/v1.0/jobs - -Running QTIP on the using 'default' as the pod name and for the 'compute' suite 'vm' type by restful api. -:: - - curl --trace-ascii debug.txt -X POST -d '{ "installer_ip": "10.20.6.2","installer_type":"fuel", "suite_name":"compute", "type": "VM"}' -H "Content-Type: application/json" http://127.0.0.1:5000/api/v1.0/jobs - -Running QTIP on the using `default` as the pod name and for the `network` suite by cli. -:: - - python qtip.py -l default -f network - -Running QTIP on the using 'default' as the pod name and for the 'network' suite 'bm' type by restful api. -:: - - curl --trace-ascii debug.txt -X POST -d '{ "installer_ip": "10.20.6.2","installer_type":"fuel", "suite_name":"network", "type": "BM"}' -H "Content-Type: application/json" http://127.0.0.1:5000/api/v1.0/jobs - -Running QTIP on the using `default` as the pod name and for the `storage` suite by cli. -:: - - python qtip.py -l default -f network - -Running QTIP on the using 'default' as the pod name and for the 'storage' suite 'bm' type by restful api. -:: - - curl --trace-ascii debug.txt -X POST -d '{ "installer_ip": "10.20.6.2","installer_type":"fuel", "suite_name":"storage", "type": "BM"}' -H "Content-Type: application/json" http://127.0.0.1:5000/api/v1.0/jobs - -Get running QTIP job status by restful api -:: - - curl --trace-ascii debug.txt -X GET http://127.0.0.1:5000/api/v1.0/jobs/job-id - For example: - curl --trace-ascii debug.txt -X GET http://127.0.0.1:5000/api/v1.0/jobs/5b71f035-3fd6-425c-9cc7-86acd3a04214 - -Stop running QTIP job by restful api.The job will finish the current benchmark test and stop. -:: - - curl --trace-ascii debug.txt -X DELTET http://127.0.0.1:5000/api/v1.0/jobs/job-id - For example: - curl --trace-ascii debug.txt -X DELETE http://127.0.0.1:5000/api/v1.0/jobs/5b71f035-3fd6-425c-9cc7-86acd3a04214q - -Results: --------- -In QTIP container, QTIP generates results in the `/home/opnfv/qtip/results/` directory are listed down under the particularly benchmark name. So all the results for dhrystone would be listed and time stamped. |