From fc7bfe32ca05a9fa70e49cd7b1571940d9320d2c Mon Sep 17 00:00:00 2001 From: Yujun Zhang Date: Tue, 18 Oct 2016 14:45:01 +0800 Subject: Reorganize existing documents MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Align document structure with Colorado release - follow convention used in "Python’s Style Guide for documentating"[1] [1] https://docs.python.org/devguide/documenting.html#sections Change-Id: Idee6ad6e68a35e3fd84ec36de480722e0259011b JIRA: QTIP-117 Signed-off-by: Yujun Zhang --- docs/configguide/configuration.rst | 6 +- docs/configguide/index.rst | 7 +- docs/download/sample_config.yaml | 50 ++++++++++ docs/overview/index.rst | 13 +++ docs/overview/overview.rst | 21 +++++ docs/templates/index.rst | 13 --- docs/templates/sample_config.yaml | 50 ---------- .../templates/testcase_description_v2_template.rst | 47 --------- docs/userguide/01-compute.rst | 105 --------------------- docs/userguide/02-network.rst | 61 ------------ docs/userguide/03-storage.rst | 31 ------ docs/userguide/_01-compute.rst | 104 ++++++++++++++++++++ docs/userguide/_02-network.rst | 61 ++++++++++++ docs/userguide/_03-storage.rst | 31 ++++++ docs/userguide/_testcase_description.rst | 46 +++++++++ docs/userguide/annex.rst | 19 ++++ docs/userguide/benchmark-suites.rst | 12 +++ docs/userguide/index.rst | 15 +-- docs/userguide/introduction.rst | 6 +- docs/userguide/overview.rst | 22 ----- 20 files changed, 372 insertions(+), 348 deletions(-) create mode 100644 docs/download/sample_config.yaml create mode 100644 docs/overview/index.rst create mode 100644 docs/overview/overview.rst delete mode 100644 docs/templates/index.rst delete mode 100644 docs/templates/sample_config.yaml delete mode 100644 docs/templates/testcase_description_v2_template.rst delete mode 100644 docs/userguide/01-compute.rst delete mode 100644 docs/userguide/02-network.rst delete mode 100644 docs/userguide/03-storage.rst create mode 100644 docs/userguide/_01-compute.rst create mode 100644 docs/userguide/_02-network.rst create mode 100644 docs/userguide/_03-storage.rst create mode 100644 docs/userguide/_testcase_description.rst create mode 100644 docs/userguide/annex.rst create mode 100644 docs/userguide/benchmark-suites.rst delete mode 100644 docs/userguide/overview.rst (limited to 'docs') diff --git a/docs/configguide/configuration.rst b/docs/configguide/configuration.rst index a9bb6b1a..e5228541 100644 --- a/docs/configguide/configuration.rst +++ b/docs/configguide/configuration.rst @@ -3,9 +3,9 @@ .. (c) 2015 Dell Inc. .. (c) 2016 ZTE Corp. - -Qtip configuration -================== +************* +Configuration +************* QTIP currently supports by using a Docker image or by pulling the repo from the upstream repository found at https://git.opnfv.org/qtip. Detailed steps diff --git a/docs/configguide/index.rst b/docs/configguide/index.rst index e8b6f4e1..291a809b 100644 --- a/docs/configguide/index.rst +++ b/docs/configguide/index.rst @@ -3,9 +3,10 @@ .. (c) 2015 Dell Inc. .. (c) 2016 ZTE Corp. -**************** -QTIP Configguide -**************** + +================= +QTIP Config Guide +================= .. toctree:: :maxdepth: 2 diff --git a/docs/download/sample_config.yaml b/docs/download/sample_config.yaml new file mode 100644 index 00000000..13f6d3fe --- /dev/null +++ b/docs/download/sample_config.yaml @@ -0,0 +1,50 @@ + ## This is a sample Config.yaml file + +#Scenario would define the test case scenario. +#The benchmark key contains the benchmark to run such as dhrystone,whetstone,dpi,ssh etc. +# Host and server list the different machines on which the benchmark would run. +# On machines listed within hosts, the benchmarks would run in parallel. +# On machines listed within server, the benchmarks would run when the benechmarks running on hosts have been completed. +# This has been used to control the folow of the testcase. For example, running the testcases on a vm vs hostmachin, we would like to run the test case serially. It should run first on the host machine and then on the vm. This testcase flow control could be used for other testcases to be developed such as those for networking. +Scenario: + benchmark: dhrystone + host: machine_1, machine_2, virtualmachine_1 + server: + + +#Context would define the environment on which to run: +#Host Machine keys would contain Host_Machines/ Baremetal machines to run the benchmarks on +#e.g in Host Machine , machine_1 and machine_2 are the bare metal machines. For each baremetal machine its IP(which should be reachable from the location on which you run QTIP), passwords and its role(host or server). If your installer is 'fuel' or 'compass' and you left baremetal machine IP empty,qtip will get compute node ip from installer automaticly. +Context: + Host_Machines: + machine_1: + ip: 172.18.0.16 + pw: Op3nStack + role: host + Virtual_Machines: + virtualmachine_1: + availability_zone: nova + public_network: 'net04_ext' + OS_image: QTIP_CentOS + flavor: m1.large + role: server +#Proxy_Environment key was optional.If all the Machines could access the public network, no need to define Proxy_Environment.Could be used later for getting http,https proxy infos which would be setted on all the Host_Manchines and Virtual_Machines. + Proxy_Environment: + http_proxy: http://10.20.0.1:8118 + https_proxy: http://10.20.0.1:8118 + no_proxy: localhost,127.0.0.1,10.20.*,192.168.* + +# A general description of the testcase. Could be used later for reports. +Test_Description: + Test_category: "Compute" + Benchmark: "dhrystone" + Overview: > + ''' This test will run the dhrystone benchmark in serial on machine_1 and machine_2.\n + if you wish to add a virtual machine add the following information under the Virtual_Machine tag + + virtualmachine_1: + availability_zone: + public_network: + OS_image: + flavor: + role: ''' diff --git a/docs/overview/index.rst b/docs/overview/index.rst new file mode 100644 index 00000000..731b8d49 --- /dev/null +++ b/docs/overview/index.rst @@ -0,0 +1,13 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2016 ZTE Corp. + + +===================== +QTIP Project Overview +===================== + +.. toctree:: + :maxdepth: 2 + + ./overview.rst diff --git a/docs/overview/overview.rst b/docs/overview/overview.rst new file mode 100644 index 00000000..aa09c7c3 --- /dev/null +++ b/docs/overview/overview.rst @@ -0,0 +1,21 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2015 Dell Inc. +.. (c) 2016 ZTE Corp. + +======== +Overview +======== + +.. _QTIP: https://wiki.opnfv.org/platform_performance_benchmarking + +QTIP_ is an OPNFV Project. + +QTIP aims to benchmark OPNFV platforms through a "Bottom up" approach, testing +bare-metal components first. + +The overall problem this project tries to solve is the general +characterization of an OPNFV platform. It will focus on general performance +questions that are common to the platform itself, or applicable to multiple +OPNFV use cases. QTIP will provide the capability to quantify a platform's +performance behavior in a standardized, rigorous, and open way. diff --git a/docs/templates/index.rst b/docs/templates/index.rst deleted file mode 100644 index 1a8bc649..00000000 --- a/docs/templates/index.rst +++ /dev/null @@ -1,13 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2015 Dell Inc. -.. (c) 2016 ZTE Corp. - -************** -QTIP Templates -************** - -.. toctree:: - :maxdepth: 2 - - ./testcase_description_v2_template.rst diff --git a/docs/templates/sample_config.yaml b/docs/templates/sample_config.yaml deleted file mode 100644 index 13f6d3fe..00000000 --- a/docs/templates/sample_config.yaml +++ /dev/null @@ -1,50 +0,0 @@ - ## This is a sample Config.yaml file - -#Scenario would define the test case scenario. -#The benchmark key contains the benchmark to run such as dhrystone,whetstone,dpi,ssh etc. -# Host and server list the different machines on which the benchmark would run. -# On machines listed within hosts, the benchmarks would run in parallel. -# On machines listed within server, the benchmarks would run when the benechmarks running on hosts have been completed. -# This has been used to control the folow of the testcase. For example, running the testcases on a vm vs hostmachin, we would like to run the test case serially. It should run first on the host machine and then on the vm. This testcase flow control could be used for other testcases to be developed such as those for networking. -Scenario: - benchmark: dhrystone - host: machine_1, machine_2, virtualmachine_1 - server: - - -#Context would define the environment on which to run: -#Host Machine keys would contain Host_Machines/ Baremetal machines to run the benchmarks on -#e.g in Host Machine , machine_1 and machine_2 are the bare metal machines. For each baremetal machine its IP(which should be reachable from the location on which you run QTIP), passwords and its role(host or server). If your installer is 'fuel' or 'compass' and you left baremetal machine IP empty,qtip will get compute node ip from installer automaticly. -Context: - Host_Machines: - machine_1: - ip: 172.18.0.16 - pw: Op3nStack - role: host - Virtual_Machines: - virtualmachine_1: - availability_zone: nova - public_network: 'net04_ext' - OS_image: QTIP_CentOS - flavor: m1.large - role: server -#Proxy_Environment key was optional.If all the Machines could access the public network, no need to define Proxy_Environment.Could be used later for getting http,https proxy infos which would be setted on all the Host_Manchines and Virtual_Machines. - Proxy_Environment: - http_proxy: http://10.20.0.1:8118 - https_proxy: http://10.20.0.1:8118 - no_proxy: localhost,127.0.0.1,10.20.*,192.168.* - -# A general description of the testcase. Could be used later for reports. -Test_Description: - Test_category: "Compute" - Benchmark: "dhrystone" - Overview: > - ''' This test will run the dhrystone benchmark in serial on machine_1 and machine_2.\n - if you wish to add a virtual machine add the following information under the Virtual_Machine tag - - virtualmachine_1: - availability_zone: - public_network: - OS_image: - flavor: - role: ''' diff --git a/docs/templates/testcase_description_v2_template.rst b/docs/templates/testcase_description_v2_template.rst deleted file mode 100644 index d68509f8..00000000 --- a/docs/templates/testcase_description_v2_template.rst +++ /dev/null @@ -1,47 +0,0 @@ -.. Template to be used for test case descriptions in QTIP Project. - Write one .rst per test case. - Borrowed Heavily from Yardstick - Upload the .rst for the test case in /docs/ directory. - Review in Gerrit. - -************************************* -QTIP Test Case Description TCXXX -************************************* - -+-----------------------------------------------------------------------------+ -|test case slogan e.g. Network throughput | -+==============+==============================================================+ -|test case id | e.g. qtip_throughput | -+--------------+--------------------------------------------------------------+ -|metric | what will be measured, e.g. latency | -+--------------+--------------------------------------------------------------+ -|test purpose | describe what is the purpose of the test case | -+--------------+--------------------------------------------------------------+ -|configuration | what .yaml file to use, state SLA if applicable, state | -| | test duration, list and describe the scenario options used in| -| | this TC and also list the options using default values. | -+--------------+--------------------------------------------------------------+ -|test tool | e.g. ping | -+--------------+--------------------------------------------------------------+ -|references | e.g. RFCxxx, ETSI-NFVyyy | -+--------------+--------------------------------------------------------------+ -|applicability | describe variations of the test case which can be | -| | performend, e.g. run the test for different packet sizes | -+--------------+--------------------------------------------------------------+ -|pre-test | describe configuration in the tool(s) used to perform | -|conditions | the measurements (e.g. fio, pktgen), POD-specific | -| | configuration required to enable running the test | -+--------------+------+----------------------------------+--------------------+ -|test sequence | step | description | result | -| +------+----------------------------------+--------------------+ -| | 1 | use this to describe tests that | what happens in | -| | | require several steps e.g. | this step | -| | | step 1 collect logs | e.g. logs collected| -| +------+----------------------------------+--------------------+ -| | 2 | remove interface | interface down | -| +------+----------------------------------+--------------------+ -| | N | what is done in step N | what happens | -+--------------+------+----------------------------------+--------------------+ -|test verdict | expected behavior, or SLA, pass/fail criteria | -+--------------+--------------------------------------------------------------+ - diff --git a/docs/userguide/01-compute.rst b/docs/userguide/01-compute.rst deleted file mode 100644 index 7a2f09f5..00000000 --- a/docs/userguide/01-compute.rst +++ /dev/null @@ -1,105 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2015 Dell Inc. -.. (c) 2016 ZTE Corp. - - -Compute test cases -================== - - -Introduction ------------- - -The QTIP testing suite aims to benchmark the compute components of an OPNFV platform. -Such components include, the CPU performance, the memory performance. -Additionally virtual computing performance provided by the Hypervisor (KVM) installed as part of OPNFV platforms would be benchmarked too. - -The test suite consists of both synthetic and application specific benchmarks to test compute components. - -All the compute benchmarks could be run in 2 scenarios: - -1. On Baremetal Machines provisioned by an OPNFV installer (Host machines) -2. On Virtual Machines brought up through OpenStack on an OPNFV platform - -Note: The Compute benchmank suite constains relatively old benchmarks such as dhrystone and whetstone. The suite would be updated for better benchmarks such as Linbench for the OPNFV C release. - -Benchmarks ----------- - -The benchmarks include: - -Dhrystone 2.1 -^^^^^^^^^^^^^^^^ - -Dhrystone is a synthetic benchmark for measuring CPU performance. It uses integer calculations to evaluate CPU capabilities. -Both Single CPU performance is measured along multi-cpu performance. - - -Dhrystone, however, is a dated benchmark and has some short comings. -Written in C, it is a small program that doesn't test the CPU memory subsystem. -Additionally, dhrystone results could be modified by optimizing the compiler and insome cases hardware configuration. - -References: http://www.eembc.org/techlit/datasheets/dhrystone_wp.pdf - -Whetstone -^^^^^^^^^^^^ - -Whetstone is a synthetic benchmark to measure CPU floating point operation performance. -Both Single CPU performance is measured along multi-cpu performance. - -Like Dhrystone, Whetstone is a dated benchmark and has short comings. - -References: - -http://www.netlib.org/benchmark/whetstone.c - -OpenSSL Speed -^^^^^^^^^^^^^^^^ - -OpenSSL Speed can be used to benchmark compute performance of a machine. In QTIP, two OpenSSL Speed benchmarks are incorporated: -1. RSA signatunes/sec signed by a machine -2. AES 128-bit encnyption throught for a machine for cipher block sizes - -References: - -https://www.openssl.org/docs/manmaster/apps/speed.html - -RAMSpeed -^^^^^^^^ - -RAMSpeed is used to measune a machine's memory perfomace. -The problem(array)size is large enough to ensure Cache Misses so that the main machine memory is used. -INTmem and FLOATmem benchmarks are executed in 4 different scenarios: - -a. Copy: a(i)=b(i) -b. Add: a(i)=b(i)+c(i) -c. Scale: a(i)=b(i)*d -d. Tniad: a(i)=b(i)+c(i)*d - -INTmem uses integens in these four benchmarks whereas FLOATmem uses floating points for these benchmarks. - -References: - -http://alasir.com/software/ramspeed/ - -https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/Untangling+memory+access+measurements - -DPI -^^^ - -nDPI is a modified variant of OpenDPI, Open source Deep packet Inspection, that is maintained by ntop. -An example application called *pcapreader* has been developed and is available for use along nDPI. - -A sample .pcap file is passed to the *pcapreader* application. -nDPI classifies traffic in the pcap file into different categories based on string matching. -The *pcapreader* application provides a throughput number for the rate at which traffic was classified, indicating a machine's computational performance. -The results are run 10 times and an average is taken for the obtained number. - -*nDPI may provide non consistent results and was added to Brahmaputra for experimental purposes* - -References: - -http://www.ntop.org/products/deep-packet-inspection/ndpi/ - -http://www.ntop.org/wp-content/uploads/2013/12/nDPI_QuickStartGuide.pdf diff --git a/docs/userguide/02-network.rst b/docs/userguide/02-network.rst deleted file mode 100644 index c556642b..00000000 --- a/docs/userguide/02-network.rst +++ /dev/null @@ -1,61 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2015 Dell Inc. -.. (c) 2016 ZTE Corp. - - -Network test cases -================== - -QTIP uses IPerf3 as the main tool for testing the network throughput. -There are three tests that are run through the QTIP framework. - -**1. Network throughput between two compute nodes** - -**2. Network Throughput between two VMs on the same compute node** - -**3. Network Throughput between two VMs on different compute nodes** - - -Network throughput between two compute nodes ------------------------------------------------ - -For the throughput between two compute nodes, Iperf3 is installed on the compute nodes comprising the systems-under-test. -One of the compute nodes is used as a server and the other as a client. -The client pushes traffic to the server for a duration specified by the user in the configuration file for Iperf3. - - -These files can be found in the "test_cases/{POD}/network/" directory. -The bandwidth is limited by the physical link layer speed connecting the two compute nodes. -The result file includes the b/s bandwidth and the CPU usage for both the client and server. - -Network throughput between two VMs on the same compute node --------------------------------------------------------------- - -QTIP framework sets up a stack with a private network, security groups, routers and attaches two VMs to this network. -Iperf3 is installed on the VMs and one is assigned the role of client while the other VM serves as a server. -Traffic is pushed over the QTIP private network between the two VMs. -A closer look is needed to see how the traffic actually flows between the VMs in this configuration to understand what is happening to the packet as it traverses the OpenStack virtual network. - -The packet originates from VM1 and its sent to the Linux bridge via a tap interface where the security groups are written. -Afterwards the packet is forwarded to the Integration bridge (br-int) via a patch port. -Since VM2 is also connected to the Integration bridge in a similar manner as VM1, the packet gets forwarded to the linux bridge connecting VM2. -After the Linux bridge the packet is sent to VM2 and is received by the Iperf3 server. -Since no physical link is involved in this topology, only the OVS (Integration bridge) (br-int) is being benchmarked. - - -Network throughput between two VMs on different compute nodes --------------------------------------------------------------- - - -As in case 2, QTIP framework sets up a stack with a private network, security groups, routers, and two VMs which are attached to the created network. However, the two VMs are spawned up on different compute nodes. - -Since the VMs are spawned on different nodes, the traffic involves additional paths. - -The traffic packet leaves the client VM and makes its way to the Integration Bridge (br-int) as in the previous case through a linux bridge and a patch port. -The integration bridge (br-int) forwards the packet to the the tunneling bridge (br-tun) where the packet is encapsulated based on the tunneling protocol used (GRE/VxLAN). -The packet then moves onto the physical link through the ethernet bridge (br-eth). - -On the receiving compute node, the packet arrives at ethernet bridge(br-eth) through the physical link. -This packet then moves to the tunneling bridge (br-tun) where the packet is decapsulated. -The packet then moves onto the internal bridge (br-int) and finally moves through a patch port into the linux bridge and eventually to the VM where it is received by the Iperf server application. diff --git a/docs/userguide/03-storage.rst b/docs/userguide/03-storage.rst deleted file mode 100644 index ea3bb4f3..00000000 --- a/docs/userguide/03-storage.rst +++ /dev/null @@ -1,31 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2015 Dell Inc. -.. (c) 2016 ZTE Corp. - - -Storage test cases -================== - -The QTIP benchmark suite aims to evaluate storage components within an OPNFV platform. -For Brahamaputra release, FIO would evaluate File System performance for the host machine. -It will also test the I/O performance provided by the hypervisor(KVM) when Storage benchmarks are run inside VMs. - -QTIP storage test cases consist of: - -**1. FIO Job to benchmark baremetal file system performance** - -**2. FIO Job to bechmark virtual machine file system performance** - -**Note: For Brahmaputra release, only the Ephemeral Storage is being tested. For C release persistent block and object storage would be tested.** - -The FIO Job would consist of: - -1. A file size of 5GB -2. Random Read 50%, Random Write 50% -3. Direct I/O -4. Asynch I/O Engine -5. I/O Queue depth of 2 -6. Block size :4K - -For this Job, I/O per second would be measured along mean I/O latency to provide storage performance numbers. diff --git a/docs/userguide/_01-compute.rst b/docs/userguide/_01-compute.rst new file mode 100644 index 00000000..7cd4c2ce --- /dev/null +++ b/docs/userguide/_01-compute.rst @@ -0,0 +1,104 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2015 Dell Inc. +.. (c) 2016 ZTE Corp. + + +Compute Suite +============= + +Introduction +------------ + +The QTIP testing suite aims to benchmark the compute components of an OPNFV platform. +Such components include, the CPU performance, the memory performance. +Additionally virtual computing performance provided by the Hypervisor (KVM) installed as part of OPNFV platforms would be benchmarked too. + +The test suite consists of both synthetic and application specific benchmarks to test compute components. + +All the compute benchmarks could be run in 2 scenarios: + +1. On Baremetal Machines provisioned by an OPNFV installer (Host machines) +2. On Virtual Machines brought up through OpenStack on an OPNFV platform + +Note: The Compute benchmank suite constains relatively old benchmarks such as dhrystone and whetstone. The suite would be updated for better benchmarks such as Linbench for the OPNFV C release. + +Benchmarks +---------- + +The benchmarks include: + +Dhrystone 2.1 +^^^^^^^^^^^^^^^^ + +Dhrystone is a synthetic benchmark for measuring CPU performance. It uses integer calculations to evaluate CPU capabilities. +Both Single CPU performance is measured along multi-cpu performance. + + +Dhrystone, however, is a dated benchmark and has some short comings. +Written in C, it is a small program that doesn't test the CPU memory subsystem. +Additionally, dhrystone results could be modified by optimizing the compiler and insome cases hardware configuration. + +References: http://www.eembc.org/techlit/datasheets/dhrystone_wp.pdf + +Whetstone +^^^^^^^^^^^^ + +Whetstone is a synthetic benchmark to measure CPU floating point operation performance. +Both Single CPU performance is measured along multi-cpu performance. + +Like Dhrystone, Whetstone is a dated benchmark and has short comings. + +References: + +http://www.netlib.org/benchmark/whetstone.c + +OpenSSL Speed +^^^^^^^^^^^^^^^^ + +OpenSSL Speed can be used to benchmark compute performance of a machine. In QTIP, two OpenSSL Speed benchmarks are incorporated: +1. RSA signatunes/sec signed by a machine +2. AES 128-bit encnyption throught for a machine for cipher block sizes + +References: + +https://www.openssl.org/docs/manmaster/apps/speed.html + +RAMSpeed +^^^^^^^^ + +RAMSpeed is used to measune a machine's memory perfomace. +The problem(array)size is large enough to ensure Cache Misses so that the main machine memory is used. +INTmem and FLOATmem benchmarks are executed in 4 different scenarios: + +a. Copy: a(i)=b(i) +b. Add: a(i)=b(i)+c(i) +c. Scale: a(i)=b(i)*d +d. Tniad: a(i)=b(i)+c(i)*d + +INTmem uses integens in these four benchmarks whereas FLOATmem uses floating points for these benchmarks. + +References: + +http://alasir.com/software/ramspeed/ + +https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/Untangling+memory+access+measurements + +DPI +^^^ + +nDPI is a modified variant of OpenDPI, Open source Deep packet Inspection, that is maintained by ntop. +An example application called *pcapreader* has been developed and is available for use along nDPI. + +A sample .pcap file is passed to the *pcapreader* application. +nDPI classifies traffic in the pcap file into different categories based on string matching. +The *pcapreader* application provides a throughput number for the rate at which traffic was classified, indicating a machine's computational performance. +The results are run 10 times and an average is taken for the obtained number. + +*nDPI may provide non consistent results and was added to Brahmaputra for experimental purposes* + +References: + +http://www.ntop.org/products/deep-packet-inspection/ndpi/ + +http://www.ntop.org/wp-content/uploads/2013/12/nDPI_QuickStartGuide.pdf diff --git a/docs/userguide/_02-network.rst b/docs/userguide/_02-network.rst new file mode 100644 index 00000000..13697aba --- /dev/null +++ b/docs/userguide/_02-network.rst @@ -0,0 +1,61 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2015 Dell Inc. +.. (c) 2016 ZTE Corp. + + +Network Suite +============= + +QTIP uses IPerf3 as the main tool for testing the network throughput. +There are three tests that are run through the QTIP framework. + +**1. Network throughput between two compute nodes** + +**2. Network Throughput between two VMs on the same compute node** + +**3. Network Throughput between two VMs on different compute nodes** + + +Network throughput between two compute nodes +----------------------------------------------- + +For the throughput between two compute nodes, Iperf3 is installed on the compute nodes comprising the systems-under-test. +One of the compute nodes is used as a server and the other as a client. +The client pushes traffic to the server for a duration specified by the user in the configuration file for Iperf3. + + +These files can be found in the "test_cases/{POD}/network/" directory. +The bandwidth is limited by the physical link layer speed connecting the two compute nodes. +The result file includes the b/s bandwidth and the CPU usage for both the client and server. + +Network throughput between two VMs on the same compute node +-------------------------------------------------------------- + +QTIP framework sets up a stack with a private network, security groups, routers and attaches two VMs to this network. +Iperf3 is installed on the VMs and one is assigned the role of client while the other VM serves as a server. +Traffic is pushed over the QTIP private network between the two VMs. +A closer look is needed to see how the traffic actually flows between the VMs in this configuration to understand what is happening to the packet as it traverses the OpenStack virtual network. + +The packet originates from VM1 and its sent to the Linux bridge via a tap interface where the security groups are written. +Afterwards the packet is forwarded to the Integration bridge (br-int) via a patch port. +Since VM2 is also connected to the Integration bridge in a similar manner as VM1, the packet gets forwarded to the linux bridge connecting VM2. +After the Linux bridge the packet is sent to VM2 and is received by the Iperf3 server. +Since no physical link is involved in this topology, only the OVS (Integration bridge) (br-int) is being benchmarked. + + +Network throughput between two VMs on different compute nodes +-------------------------------------------------------------- + + +As in case 2, QTIP framework sets up a stack with a private network, security groups, routers, and two VMs which are attached to the created network. However, the two VMs are spawned up on different compute nodes. + +Since the VMs are spawned on different nodes, the traffic involves additional paths. + +The traffic packet leaves the client VM and makes its way to the Integration Bridge (br-int) as in the previous case through a linux bridge and a patch port. +The integration bridge (br-int) forwards the packet to the the tunneling bridge (br-tun) where the packet is encapsulated based on the tunneling protocol used (GRE/VxLAN). +The packet then moves onto the physical link through the ethernet bridge (br-eth). + +On the receiving compute node, the packet arrives at ethernet bridge(br-eth) through the physical link. +This packet then moves to the tunneling bridge (br-tun) where the packet is decapsulated. +The packet then moves onto the internal bridge (br-int) and finally moves through a patch port into the linux bridge and eventually to the VM where it is received by the Iperf server application. diff --git a/docs/userguide/_03-storage.rst b/docs/userguide/_03-storage.rst new file mode 100644 index 00000000..b1490432 --- /dev/null +++ b/docs/userguide/_03-storage.rst @@ -0,0 +1,31 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2015 Dell Inc. +.. (c) 2016 ZTE Corp. + + +Storage Suite +============= + +The QTIP benchmark suite aims to evaluate storage components within an OPNFV platform. +For Brahamaputra release, FIO would evaluate File System performance for the host machine. +It will also test the I/O performance provided by the hypervisor(KVM) when Storage benchmarks are run inside VMs. + +QTIP storage test cases consist of: + +**1. FIO Job to benchmark baremetal file system performance** + +**2. FIO Job to bechmark virtual machine file system performance** + +**Note: For Brahmaputra release, only the Ephemeral Storage is being tested. For C release persistent block and object storage would be tested.** + +The FIO Job would consist of: + +1. A file size of 5GB +2. Random Read 50%, Random Write 50% +3. Direct I/O +4. Asynch I/O Engine +5. I/O Queue depth of 2 +6. Block size :4K + +For this Job, I/O per second would be measured along mean I/O latency to provide storage performance numbers. diff --git a/docs/userguide/_testcase_description.rst b/docs/userguide/_testcase_description.rst new file mode 100644 index 00000000..d60ca949 --- /dev/null +++ b/docs/userguide/_testcase_description.rst @@ -0,0 +1,46 @@ +.. Template to be used for test case descriptions in QTIP Project. + Write one .rst per test case. + Borrowed Heavily from Yardstick + Upload the .rst for the test case in /docs/ directory. + Review in Gerrit. + +Test Case Description +===================== + ++-----------------------------------------------------------------------------+ +|test case slogan e.g. Network throughput | ++==============+==============================================================+ +|test case id | e.g. qtip_throughput | ++--------------+--------------------------------------------------------------+ +|metric | what will be measured, e.g. latency | ++--------------+--------------------------------------------------------------+ +|test purpose | describe what is the purpose of the test case | ++--------------+--------------------------------------------------------------+ +|configuration | what .yaml file to use, state SLA if applicable, state | +| | test duration, list and describe the scenario options used in| +| | this TC and also list the options using default values. | ++--------------+--------------------------------------------------------------+ +|test tool | e.g. ping | ++--------------+--------------------------------------------------------------+ +|references | e.g. RFCxxx, ETSI-NFVyyy | ++--------------+--------------------------------------------------------------+ +|applicability | describe variations of the test case which can be | +| | performend, e.g. run the test for different packet sizes | ++--------------+--------------------------------------------------------------+ +|pre-test | describe configuration in the tool(s) used to perform | +|conditions | the measurements (e.g. fio, pktgen), POD-specific | +| | configuration required to enable running the test | ++--------------+------+----------------------------------+--------------------+ +|test sequence | step | description | result | +| +------+----------------------------------+--------------------+ +| | 1 | use this to describe tests that | what happens in | +| | | require several steps e.g. | this step | +| | | step 1 collect logs | e.g. logs collected| +| +------+----------------------------------+--------------------+ +| | 2 | remove interface | interface down | +| +------+----------------------------------+--------------------+ +| | N | what is done in step N | what happens | ++--------------+------+----------------------------------+--------------------+ +|test verdict | expected behavior, or SLA, pass/fail criteria | ++--------------+--------------------------------------------------------------+ + diff --git a/docs/userguide/annex.rst b/docs/userguide/annex.rst new file mode 100644 index 00000000..406d5132 --- /dev/null +++ b/docs/userguide/annex.rst @@ -0,0 +1,19 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2016 ZTE Corp. + + +***** +Annex +***** + +Templates +========= + +.. include:: _testcase_description.rst + + +Downloads +========= + +- :download:`Sample configuration <../download/sample_config.yaml>` \ No newline at end of file diff --git a/docs/userguide/benchmark-suites.rst b/docs/userguide/benchmark-suites.rst new file mode 100644 index 00000000..9a3929cc --- /dev/null +++ b/docs/userguide/benchmark-suites.rst @@ -0,0 +1,12 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2016 ZTE Corp. + + +**************** +Benchmark Suites +**************** + +.. include:: _01-compute.rst +.. include:: _02-network.rst +.. include:: _03-storage.rst diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst index 202bf66e..3032f357 100644 --- a/docs/userguide/index.rst +++ b/docs/userguide/index.rst @@ -4,18 +4,13 @@ .. (c) 2016 ZTE Corp. -************** -QTIP Userguide -************** - -Colorado 1.0 ------------- +=============== +QTIP User Guide +=============== .. toctree:: :maxdepth: 2 - ./overview.rst ./introduction.rst - ./01-compute.rst - ./02-network.rst - ./03-storage.rst + ./benchmark-suites.rst + ./annex.rst diff --git a/docs/userguide/introduction.rst b/docs/userguide/introduction.rst index b874ea46..2655347e 100644 --- a/docs/userguide/introduction.rst +++ b/docs/userguide/introduction.rst @@ -4,9 +4,9 @@ .. (c) 2016 ZTE Corp. -***************** -Qtip Introduction -***************** +************ +Introduction +************ This guide will serve as a first step to familiarize the user with how to run QTIP the first time when the user clones QTIP on to their host machine. diff --git a/docs/userguide/overview.rst b/docs/userguide/overview.rst deleted file mode 100644 index 82b7de91..00000000 --- a/docs/userguide/overview.rst +++ /dev/null @@ -1,22 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2015 Dell Inc. -.. (c) 2016 ZTE Corp. - - -******** -Overview -******** - -.. _QTIP: https://wiki.opnfv.org/platform_performance_benchmarking - -QTIP_ is an OPNFV Project. - -QTIP aims to benchmark OPNFV platforms through a "Bottom up" approach, testing -bare-metal components first. - -The overall problem this project tries to solve is the general -characterization of an OPNFV platform. It will focus on general performance -questions that are common to the platform itself, or applicable to multiple -OPNFV use cases. QTIP will provide the capability to quantify a platform's -performance behavior in a standardized, rigorous, and open way. -- cgit 1.2.3-korg