From 0bbef8513b2ed9166d57e3467d62c823a134d4e6 Mon Sep 17 00:00:00 2001 From: Ana C Date: Thu, 7 Jan 2016 15:00:56 +0100 Subject: Rename folders in yardstick/docs Adapt the naming of the folders under yardstick/docs to the naming adopted by the release documentation defined by opnfvdocs. JIRA: - Change-Id: Icc57720d585abbbf7252bcbf76e2f2a403cb1732 Signed-off-by: Ana C (cherry picked from commit 463a69e878d891b897fe70af15434615baf6a2e9) --- docs/configguide/vTC/README.rst | 87 ++++++++ docs/configguide/vTC/abbreviations.rst | 5 + .../yardstick_testcases/01-introduction.rst | 38 ++++ .../yardstick_testcases/02-methodology.rst | 181 +++++++++++++++++ .../yardstick_testcases/03-list-of-tcs.rst | 44 ++++ .../Yardstick_task_templates.rst | 155 +++++++++++++++ docs/configguide/yardstick_testcases/glossary.rst | 15 ++ docs/configguide/yardstick_testcases/index.rst | 11 + .../yardstick_testcases/opnfv_yardstick_tc001.rst | 79 ++++++++ .../yardstick_testcases/opnfv_yardstick_tc002.rst | 78 ++++++++ .../yardstick_testcases/opnfv_yardstick_tc008.rst | 85 ++++++++ .../yardstick_testcases/opnfv_yardstick_tc009.rst | 84 ++++++++ .../yardstick_testcases/opnfv_yardstick_tc010.rst | 77 +++++++ .../yardstick_testcases/opnfv_yardstick_tc012.rst | 81 ++++++++ .../yardstick_testcases/opnfv_yardstick_tc037.rst | 99 +++++++++ .../yardstick_testcases/opnfv_yardstick_tc038.rst | 99 +++++++++ .../testcase_description_v2_template.rst | 64 ++++++ docs/user_guides/framework/03-installation.rst | 221 --------------------- docs/user_guides/framework/index.rst | 9 - .../yardstick_framework/03-installation.rst | 221 +++++++++++++++++++++ docs/userguide/yardstick_framework/index.rst | 9 + docs/vTC/README.rst | 87 -------- docs/vTC/abbreviations.rst | 5 - docs/yardstick/01-introduction.rst | 38 ---- docs/yardstick/02-methodology.rst | 181 ----------------- docs/yardstick/03-list-of-tcs.rst | 44 ---- docs/yardstick/Yardstick_task_templates.rst | 155 --------------- docs/yardstick/glossary.rst | 15 -- docs/yardstick/index.rst | 11 - docs/yardstick/opnfv_yardstick_tc001.rst | 79 -------- docs/yardstick/opnfv_yardstick_tc002.rst | 78 -------- docs/yardstick/opnfv_yardstick_tc008.rst | 85 -------- docs/yardstick/opnfv_yardstick_tc009.rst | 84 -------- docs/yardstick/opnfv_yardstick_tc010.rst | 77 ------- docs/yardstick/opnfv_yardstick_tc012.rst | 81 -------- docs/yardstick/opnfv_yardstick_tc037.rst | 99 --------- docs/yardstick/opnfv_yardstick_tc038.rst | 99 --------- .../yardstick/testcase_description_v2_template.rst | 64 ------ 38 files changed, 1512 insertions(+), 1512 deletions(-) create mode 100644 docs/configguide/vTC/README.rst create mode 100644 docs/configguide/vTC/abbreviations.rst create mode 100644 docs/configguide/yardstick_testcases/01-introduction.rst create mode 100644 docs/configguide/yardstick_testcases/02-methodology.rst create mode 100644 docs/configguide/yardstick_testcases/03-list-of-tcs.rst create mode 100755 docs/configguide/yardstick_testcases/Yardstick_task_templates.rst create mode 100644 docs/configguide/yardstick_testcases/glossary.rst create mode 100644 docs/configguide/yardstick_testcases/index.rst create mode 100644 docs/configguide/yardstick_testcases/opnfv_yardstick_tc001.rst create mode 100644 docs/configguide/yardstick_testcases/opnfv_yardstick_tc002.rst create mode 100644 docs/configguide/yardstick_testcases/opnfv_yardstick_tc008.rst create mode 100644 docs/configguide/yardstick_testcases/opnfv_yardstick_tc009.rst create mode 100644 docs/configguide/yardstick_testcases/opnfv_yardstick_tc010.rst create mode 100644 docs/configguide/yardstick_testcases/opnfv_yardstick_tc012.rst create mode 100644 docs/configguide/yardstick_testcases/opnfv_yardstick_tc037.rst create mode 100644 docs/configguide/yardstick_testcases/opnfv_yardstick_tc038.rst create mode 100644 docs/configguide/yardstick_testcases/testcase_description_v2_template.rst delete mode 100644 docs/user_guides/framework/03-installation.rst delete mode 100644 docs/user_guides/framework/index.rst create mode 100644 docs/userguide/yardstick_framework/03-installation.rst create mode 100644 docs/userguide/yardstick_framework/index.rst delete mode 100644 docs/vTC/README.rst delete mode 100644 docs/vTC/abbreviations.rst delete mode 100644 docs/yardstick/01-introduction.rst delete mode 100644 docs/yardstick/02-methodology.rst delete mode 100644 docs/yardstick/03-list-of-tcs.rst delete mode 100755 docs/yardstick/Yardstick_task_templates.rst delete mode 100644 docs/yardstick/glossary.rst delete mode 100644 docs/yardstick/index.rst delete mode 100644 docs/yardstick/opnfv_yardstick_tc001.rst delete mode 100644 docs/yardstick/opnfv_yardstick_tc002.rst delete mode 100644 docs/yardstick/opnfv_yardstick_tc008.rst delete mode 100644 docs/yardstick/opnfv_yardstick_tc009.rst delete mode 100644 docs/yardstick/opnfv_yardstick_tc010.rst delete mode 100644 docs/yardstick/opnfv_yardstick_tc012.rst delete mode 100644 docs/yardstick/opnfv_yardstick_tc037.rst delete mode 100644 docs/yardstick/opnfv_yardstick_tc038.rst delete mode 100644 docs/yardstick/testcase_description_v2_template.rst (limited to 'docs') diff --git a/docs/configguide/vTC/README.rst b/docs/configguide/vTC/README.rst new file mode 100644 index 000000000..ae6fefa59 --- /dev/null +++ b/docs/configguide/vTC/README.rst @@ -0,0 +1,87 @@ +========================== +Virtual Traffic Classifier +========================== + +Overview +======== + +The virtual Traffic Classifier VNF [1], comprises in the current version of +1 VNFC [2]. The VNFC contains both the Traffic Inspection module, and the +Traffic forwarding module, needed to run the VNF. The exploitation of DPI +methods for traffic classification is built around two basic assumptions: + +(i) third parties unaffiliated with either source or recipient are able to +inspect each IP packet’s payload and +(ii) the classifier knows the relevant syntax of each application’s packet +payloads (protocol signatures, data patterns, etc.). + +The proposed DPI based approach will only use an indicative, small number of +the initial packets from each flow in order to identify the content and not +inspect each packet. + +In this respect it follows the Packet Based per Flow State (PBFS). +This method uses a table to track each session based on the 5-tuples +(src address,dest address,src port,dest port,transport protocol) +that is maintained for each flow. + +Concepts +======== + +Traffic Inspection: The process of packet analysis and application +identification of network traffic that passes through the vTC. + +Traffic Forwarding: The process of packet forwarding from an incoming +network interface to a pre-defined outgoing network interface. + +Traffic Rule Application: The process of packet tagging, based on a +predefined set of rules. Packet tagging may include e.g. ToS field +modification. + +Architecture +============ + +The Traffic Inspection module is the most computationally intensive component +of the VNF. It implements filtering and packet matching algorithms in order to +support the enhanced traffic forwarding capability of the VNF. The component +supports a flow table (exploiting hashing algorithms for fast indexing of +flows) and an inspection engine for traffic classification. + +The implementation used for these experiments exploits the nDPI library. +The packet capturing mechanism is implemented using libpcap. When the DPI +engine identifies a new flow, the flow register is updated with the +appropriate information and transmitted across the Traffic Forwarding module, +which then applies any required policy updates. + +The Traffic Forwarding moudle is responsible for routing and packet forwarding. +It accepts incoming network traffic, consults the flow table for classification +information for each incoming flow and then applies pre-defined policies +marking e.g. type of Service/Differentiated Services Code Point (TOS/DSCP) +multimedia traffic for QoS enablement on the forwarded traffic. +It is assumed that the traffic is forwarded using the default policy until it +is identified and new policies are enforced. + +The expected response delay is considered to be negligible,as only a small +number of packets are required to identify each flow. + +Graphical Overview +================== + +Install +======= + +run the build.sh with root privileges + +Run +=== + +sudo ./pfbridge -a eth1 -b eth2 + +Custom Image +============ + +TBD + +Development Environment +======================= + +Ubuntu 14.04 >= VM diff --git a/docs/configguide/vTC/abbreviations.rst b/docs/configguide/vTC/abbreviations.rst new file mode 100644 index 000000000..a713ee66b --- /dev/null +++ b/docs/configguide/vTC/abbreviations.rst @@ -0,0 +1,5 @@ +Abbreviations for the virtual Traffic Classifier +================================================ + +[1] VNF - Virtual Network Function +[2] VNFC - Virtual Network Function Component diff --git a/docs/configguide/yardstick_testcases/01-introduction.rst b/docs/configguide/yardstick_testcases/01-introduction.rst new file mode 100644 index 000000000..6cca2875e --- /dev/null +++ b/docs/configguide/yardstick_testcases/01-introduction.rst @@ -0,0 +1,38 @@ +============ +Introduction +============ + +**Welcome to Yardstick's documentation !** + +.. _Yardstick: https://wiki.opnfv.org/yardstick + +Yardstick_ is an OPNFV Project. + +The project's goal is to verify infrastructure compliance, from the perspective +of a :term:`VNF`. + +The Project's scope is the development of a test framework, *Yardstick*, test +cases and test stimuli to enable :term:`NFVI` verification. +The Project also includes a sample :term:`VNF`, the :term:`VTC` and its +experimental framework, *ApexLake* ! + +The chapter :doc:`02-methodology` describes the methodology implemented by the +Yardstick Project for :term:`NFVI` verification. The chapter +:doc:`03-list-of-tcs` includes a list of available Yardstick test cases. + +Yardstick is used for verifying the OPNFV infrastructure and some of the OPNFV +features, listed in :doc:`03-list-of-tcs`. + +The *Yardstick* framework is deployed in several OPNFV community labs. It is +installer, infrastructure and application independent. + +.. _Pharos: https://wiki.opnfv.org/pharos + +.. seealso:: Pharos_ for information on OPNFV community labs. + +Contact Yardstick +================= + +Feedback? `Contact us`_ + +.. _Contact us: opnfv-users@lists.opnfv.org diff --git a/docs/configguide/yardstick_testcases/02-methodology.rst b/docs/configguide/yardstick_testcases/02-methodology.rst new file mode 100644 index 000000000..5097c566b --- /dev/null +++ b/docs/configguide/yardstick_testcases/02-methodology.rst @@ -0,0 +1,181 @@ +=========== +Methodology +=========== + +Abstract +======== + +This chapter describes the methodology implemented by the Yardstick project for +verifying the NFV Infrastructure from the perspective of a VNF. + +ETSI-NFV +======== + +.. _NFV-TST001: https://docbox.etsi.org/ISG/NFV/Open/Drafts/TST001_-_Pre-deployment_Validation/ + +The document ETSI GS NFV-TST001_, "Pre-deployment Testing; Report on Validation +of NFV Environments and Services", recommends methods for pre-deployment +testing of the functional components of an NFV environment. + +The Yardstick project implements the methodology described in chapter 6, "Pre- +deployment validation of NFV infrastructure". + +The methodology consists in decomposing the typical VNF work-load performance +metrics into a number of characteristics/performance vectors, which each can be +represented by distinct test-cases. + +The methodology includes five steps: + +* *Step1:* Define Infrastruture - the HW, SW and corresponding configuration + target for validation; the OPNFV infrastructure, in OPNFV community labs. + +* *Step2:* Identify VNF type - the application for which the infrastructure is + to be validated, and its requirements on the underlying infrastructure. + +* *Step3:* Select test cases - depending on the workload that represents the + application for which the infrastruture is to be validated, the relevant + test cases amongst the list of available Yardstick test cases. + +* *Step4:* Execute tests - define the duration and number of iterations for the + selected test cases, tests runs are automated via OPNFV Jenkins Jobs. + +* *Step5:* Collect results - using the common API for result collection. + +Metrics +======= + +The metrics, as defined by ETSI GS NFV-TST001, are shown in +:ref:`Table1 `, :ref:`Table2 ` and +:ref:`Table3 `. + +In OPNFV Brahmaputra release, generic test cases covering aspects of the listed +metrics are available; further OPNFV releases will provide extended testing of +these metrics. +The view of available Yardstick test cases cross ETSI definitions in +:ref:`Table1 `, :ref:`Table2 ` and :ref:`Table3 ` +is shown in :ref:`Table4 `. +It shall be noticed that the Yardstick test cases are examples, the test +duration and number of iterations are configurable, as are the System Under +Test (SUT) and the attributes (or, in Yardstick nomemclature, the scenario +options). + +.. _table2_1: + +**Table 1 - Performance/Speed Metrics** + ++---------+-------------------------------------------------------------------+ +| Category| Performance/Speed | +| | | ++---------+-------------------------------------------------------------------+ +| Compute | * Latency for random memory access | +| | * Latency for cache read/write operations | +| | * Processing speed (instructions per second) | +| | * Throughput for random memory access (bytes per second) | +| | | ++---------+-------------------------------------------------------------------+ +| Network | * Throughput per NFVI node (frames/byte per second) | +| | * Throughput provided to a VM (frames/byte per second) | +| | * Latency per traffic flow | +| | * Latency between VMs | +| | * Latency between NFVI nodes | +| | * Packet delay variation (jitter) between VMs | +| | * Packet delay variation (jitter) between NFVI nodes | +| | | ++---------+-------------------------------------------------------------------+ +| Storage | * Sequential read/write IOPS | +| | * Random read/write IOPS | +| | * Latency for storage read/write operations | +| | * Throughput for storage read/write operations | +| | | ++---------+-------------------------------------------------------------------+ + +.. _table2_2: + +**Table 2 - Capacity/Scale Metrics** + ++---------+-------------------------------------------------------------------+ +| Category| Capacity/Scale | +| | | ++---------+-------------------------------------------------------------------+ +| Compute | * Number of cores and threads- Available memory size | +| | * Cache size | +| | * Processor utilization (max, average, standard deviation) | +| | * Memory utilization (max, average, standard deviation) | +| | * Cache utilization (max, average, standard deviation) | +| | | ++---------+-------------------------------------------------------------------+ +| Network | * Number of connections | +| | * Number of frames sent/received | +| | * Maximum throughput between VMs (frames/byte per second) | +| | * Maximum throughput between NFVI nodes (frames/byte per second) | +| | * Network utilization (max, average, standard deviation) | +| | * Number of traffic flows | +| | | ++---------+-------------------------------------------------------------------+ +| Storage | * Storage/Disk size | +| | * Capacity allocation (block-based, object-based) | +| | * Block size | +| | * Maximum sequential read/write IOPS | +| | * Maximum random read/write IOPS | +| | * Disk utilization (max, average, standard deviation) | +| | | ++---------+-------------------------------------------------------------------+ + +.. _table2_3: + +**Table 3 - Availability/Reliability Metrics** + ++---------+-------------------------------------------------------------------+ +| Category| Availability/Reliability | +| | | ++---------+-------------------------------------------------------------------+ +| Compute | * Processor availability (Error free processing time) | +| | * Memory availability (Error free memory time) | +| | * Processor mean-time-to-failure | +| | * Memory mean-time-to-failure | +| | * Number of processing faults per second | +| | | ++---------+-------------------------------------------------------------------+ +| Network | * NIC availability (Error free connection time) | +| | * Link availability (Error free transmission time) | +| | * NIC mean-time-to-failure | +| | * Network timeout duration due to link failure | +| | * Frame loss rate | +| | | ++---------+-------------------------------------------------------------------+ +| Storage | * Disk availability (Error free disk access time) | +| | * Disk mean-time-to-failure | +| | * Number of failed storage read/write operations per second | +| | | ++---------+-------------------------------------------------------------------+ + +.. _table2_4: + +**Table 4 - Yardstick Generic Test Cases** + ++---------+-------------------+----------------+------------------------------+ +| Category| Performance/Speed | Capacity/Scale | Availability/Reliability | +| | | | | ++---------+-------------------+----------------+------------------------------+ +| Compute | TC003 | TC003 | TC013 [1]_ | +| | TC004 | TC004 | TC015 [1]_ | +| | TC014 | TC010 | | +| | TC024 | TC012 | | +| | | | | ++---------+-------------------+----------------+------------------------------+ +| Network | TC002 | TC001 | TC016 [1]_ | +| | TC011 | TC008 | TC018 [1]_ | +| | | TC009 | | +| | | | | ++---------+-------------------+----------------+------------------------------+ +| Storage | TC005 | TC005 | TC017 [1]_ | +| | | | | ++---------+-------------------+----------------+------------------------------+ + +.. note:: The description in this OPNFV document is intended as a reference for + users to understand the scope of the Yardstick Project and the + deliverables of the Yardstick framework. For complete description of + the methodology, refer to the ETSI document. + +.. rubric:: Footnotes +.. [1] To be included in future deliveries. diff --git a/docs/configguide/yardstick_testcases/03-list-of-tcs.rst b/docs/configguide/yardstick_testcases/03-list-of-tcs.rst new file mode 100644 index 000000000..9e951ffab --- /dev/null +++ b/docs/configguide/yardstick_testcases/03-list-of-tcs.rst @@ -0,0 +1,44 @@ +==================== +Yardstick Test Cases +==================== + +Abstract +======== + +This chapter lists available Yardstick test cases. +Yardstick test cases are divided in two main categories: + +* *Generic NFVI Test Cases* - Test Cases developed to realize the methodology +described in :doc:`02-methodology` + +* *OPNFV Feature Test Cases* - Test Cases developed to verify one or more +aspect of a feature delivered by an OPNFV Project. + +Generic NFVI Test Case Descriptions +=================================== + +.. toctree:: + :maxdepth: 1 + + opnfv_yardstick_tc001.rst + opnfv_yardstick_tc002.rst + opnfv_yardstick_tc008.rst + opnfv_yardstick_tc009.rst + opnfv_yardstick_tc010.rst + opnfv_yardstick_tc012.rst + opnfv_yardstick_tc037.rst + opnfv_yardstick_tc038.rst + +OPNFV Feature Test Cases +======================== + + + +Templates +========= + +.. toctree:: + :maxdepth: 1 + + testcase_description_v2_template + Yardstick_task_templates diff --git a/docs/configguide/yardstick_testcases/Yardstick_task_templates.rst b/docs/configguide/yardstick_testcases/Yardstick_task_templates.rst new file mode 100755 index 000000000..d2c2b7ec9 --- /dev/null +++ b/docs/configguide/yardstick_testcases/Yardstick_task_templates.rst @@ -0,0 +1,155 @@ +Task Template Syntax +==================== + +Basic template syntax +--------------------- +A nice feature of the input task format used in Yardstick is that it supports +the template syntax based on Jinja2. +This turns out to be extremely useful when, say, you have a fixed structure of +your task but you want to parameterize this task in some way. +For example, imagine your input task file (task.yaml) runs a set of Ping +scenarios: + +:: + + # Sample benchmark task config file + # measure network latency using ping + schema: "yardstick:task:0.1" + + scenarios: + - + type: Ping + options: + packetsize: 200 + host: athena.demo + target: ares.demo + + runner: + type: Duration + duration: 60 + interval: 1 + + sla: + max_rtt: 10 + action: monitor + + context: + ... + +Let's say you want to run the same set of scenarios with the same runner/ +context/sla, but you want to try another packetsize to compare the performance. +The most elegant solution is then to turn the packetsize name into a template +variable: + +:: + + # Sample benchmark task config file + # measure network latency using ping + + schema: "yardstick:task:0.1" + scenarios: + - + type: Ping + options: + packetsize: {{packetsize}} + host: athena.demo + target: ares.demo + + runner: + type: Duration + duration: 60 + interval: 1 + + sla: + max_rtt: 10 + action: monitor + + context: + ... + +and then pass the argument value for {{packetsize}} when starting a task with +this configuration file. +Yardstick provides you with different ways to do that: + +1.Pass the argument values directly in the command-line interface (with either +a JSON or YAML dictionary): + +:: + + yardstick task start samples/ping-template.yaml + --task-args'{"packetsize":"200"}' + +2.Refer to a file that specifies the argument values (JSON/YAML): + +:: + + yardstick task start samples/ping-template.yaml --task-args-file args.yaml + +Using the default values +------------------------ +Note that the Jinja2 template syntax allows you to set the default values for +your parameters. +With default values set, your task file will work even if you don't +parameterize it explicitly while starting a task. +The default values should be set using the {% set ... %} clause (task.yaml). +For example: + +:: + + # Sample benchmark task config file + # measure network latency using ping + schema: "yardstick:task:0.1" + {% set packetsize = packetsize or "100" %} + scenarios: + - + type: Ping + options: + packetsize: {{packetsize}} + host: athena.demo + target: ares.demo + + runner: + type: Duration + duration: 60 + interval: 1 + ... + +If you don't pass the value for {{packetsize}} while starting a task, the +default one will be used. + +Advanced templates +------------------ + +Yardstick makes it possible to use all the power of Jinja2 template syntax, +including the mechanism of built-in functions. +As an example, let us make up a task file that will do a block storage +performance test. +The input task file (fio-template.yaml) below uses the Jinja2 for-endfor +construct to accomplish that: + +:: + + #Test block sizes of 4KB, 8KB, 64KB, 1MB + #Test 5 workloads: read, write, randwrite, randread, rw + schema: "yardstick:task:0.1" + + scenarios: + {% for bs in ['4k', '8k', '64k', '1024k' ] %} + {% for rw in ['read', 'write', 'randwrite', 'randread', 'rw' ] %} + - + type: Fio + options: + filename: /home/ec2-user/data.raw + bs: {{bs}} + rw: {{rw}} + ramp_time: 10 + host: fio.demo + runner: + type: Duration + duration: 60 + interval: 60 + + {% endfor %} + {% endfor %} + context + ... diff --git a/docs/configguide/yardstick_testcases/glossary.rst b/docs/configguide/yardstick_testcases/glossary.rst new file mode 100644 index 000000000..67e126f58 --- /dev/null +++ b/docs/configguide/yardstick_testcases/glossary.rst @@ -0,0 +1,15 @@ +================== +Yardstick Glossary +================== + +.. glossary:: + :sorted: + + VNF + Virtual Network Function + + NFVI + Network Function Virtualization Infrastructure + + VTC + Virtual Traffic Classifier diff --git a/docs/configguide/yardstick_testcases/index.rst b/docs/configguide/yardstick_testcases/index.rst new file mode 100644 index 000000000..ccea55ccb --- /dev/null +++ b/docs/configguide/yardstick_testcases/index.rst @@ -0,0 +1,11 @@ +================== +Yardstick Overview +================== + +.. toctree:: + :maxdepth: 2 + + 01-introduction + 02-methodology + 03-list-of-tcs + glossary diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc001.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc001.rst new file mode 100644 index 000000000..810bad489 --- /dev/null +++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc001.rst @@ -0,0 +1,79 @@ +************************************* +Yardstick Test Case Description TC001 +************************************* + +.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt + ++-----------------------------------------------------------------------------+ +|Network Performance | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC001_NW PERF | +| | | ++--------------+--------------------------------------------------------------+ +|metric | Number of flows and throughput | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To evaluate the IaaS network performance with regards to | +| | flows and throughput, such as if and how different amounts | +| | of flows matter for the throughput between hosts on different| +| | compute blades. Typically e.g. the performance of a vSwitch | +| | depends on the number of flows running through it. Also | +| | performance of other equipment or entities can depend | +| | on the number of flows or the packet sizes used. | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs ans similar shall be stored for comparison reasons and| +| | product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc001.yaml | +| | | +| | Packet size: 60 bytes | +| | Number of ports: 10, 50, 100, 500 and 1000, where each | +| | runs for 20 seconds. The whole sequence is run | +| | twice. The client and server are distributed on different HW.| +| | For SLA max_ppm is set to 1000. The amount of configured | +| | ports map to between 110 up to 1001000 flows, respectively. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | pktgen | +| | | +| | (Pktgen is not always part of a Linux distribution, hence it | +| | needs to be installed. It is part of the Yardstick Docker | +| | image. | +| | As an example see the /yardstick/tools/ directory for how | +| | to generate a Linux image with pktgen included.) | +| | | ++--------------+--------------------------------------------------------------+ +|references | pktgen_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different packet sizes, amount | +| | of flows and test duration. Default values exist. | +| | | +| | SLA (optional): max_ppm: The number of packets per million | +| | packets sent that are acceptable to loose, not received. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with pktgen included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. pktgen is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc002.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc002.rst new file mode 100644 index 000000000..56350f5bb --- /dev/null +++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc002.rst @@ -0,0 +1,78 @@ +************************************* +Yardstick Test Case Description TC002 +************************************* + +.. _cirros-image: https://download.cirros-cloud.net + ++-----------------------------------------------------------------------------+ +|Network Latency | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC002_NW LATENCY | +| | | ++--------------+--------------------------------------------------------------+ +|metric | RTT, Round Trip Time | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To do a basic verification that network latency is within | +| | acceptable boundaries when packets travel between hosts | +| | located on same or different compute blades. | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs and similar shall be stored for comparison reasons and| +| | product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc002.yaml | +| | | +| | Packet size 100 bytes. Total test duration 600 seconds. | +| | One ping each 10 seconds. SLA RTT is set to maximum 10 ms. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | ping | +| | | +| | Ping is normally part of any Linux distribution, hence it | +| | doesn't need to be installed. It is also part of the | +| | Yardstick Docker image. | +| | (For example also a Cirros image can be downloaded from | +| | cirros-image_, it includes ping) | +| | | ++--------------+--------------------------------------------------------------+ +|references | Ping man page | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test case can be configured with different packet sizes, | +| | burst sizes, ping intervals and test duration. | +| | SLA is optional. The SLA in this test case serves as an | +| | example. Considerably lower RTT is expected, and | +| | also normal to achieve in balanced L2 environments. However, | +| | to cover most configurations, both bare metal and fully | +| | virtualized ones, this value should be possible to achieve | +| | and acceptable for black box testing. Many real time | +| | applications start to suffer badly if the RTT time is higher | +| | than this. Some may suffer bad also close to this RTT, while | +| | others may not suffer at all. It is a compromise that may | +| | have to be tuned for different configuration purposes. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with ping included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. Ping is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Test should not PASS if any RTT is above the optional SLA | +| | value, or if there is a test case execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc008.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc008.rst new file mode 100644 index 000000000..e176e633a --- /dev/null +++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc008.rst @@ -0,0 +1,85 @@ +************************************* +Yardstick Test Case Description TC008 +************************************* + +.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt + ++-----------------------------------------------------------------------------+ +|Packet Loss Extended Test | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC008_NW PERF, Packet loss Extended Test | +| | | ++--------------+--------------------------------------------------------------+ +|metric | Number of flows, packet size and throughput | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To evaluate the IaaS network performance with regards to | +| | flows and throughput, such as if and how different amounts | +| | of packet sizes and flows matter for the throughput between | +| | VMs on different compute blades. Typically e.g. the | +| | performance of a vSwitch | +| | depends on the number of flows running through it. Also | +| | performance of other equipment or entities can depend | +| | on the number of flows or the packet sizes used. | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs ans similar shall be stored for comparison reasons and| +| | product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc008.yaml | +| | | +| | Packet size: 64, 128, 256, 512, 1024, 1280 and 1518 bytes. | +| | | +| | Number of ports: 1, 10, 50, 100, 500 and 1000. The amount of | +| | configured ports map from 2 up to 1001000 flows, | +| | respectively. Each packet_size/port_amount combination is run| +| | ten times, for 20 seconds each. Then the next | +| | packet_size/port_amount combination is run, and so on. | +| | | +| | The client and server are distributed on different HW. | +| | | +| | For SLA max_ppm is set to 1000. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | pktgen | +| | | +| | (Pktgen is not always part of a Linux distribution, hence it | +| | needs to be installed. It is part of the Yardstick Docker | +| | image. | +| | As an example see the /yardstick/tools/ directory for how | +| | to generate a Linux image with pktgen included.) | +| | | ++--------------+--------------------------------------------------------------+ +|references | pktgen_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different packet sizes, amount | +| | of flows and test duration. Default values exist. | +| | | +| | SLA (optional): max_ppm: The number of packets per million | +| | packets sent that are acceptable to loose, not received. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with pktgen included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. pktgen is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc009.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc009.rst new file mode 100644 index 000000000..e4002a884 --- /dev/null +++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc009.rst @@ -0,0 +1,84 @@ +************************************* +Yardstick Test Case Description TC009 +************************************* + +.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt + ++-----------------------------------------------------------------------------+ +|Packet Loss | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC009_NW PERF, Packet loss | +| | | ++--------------+--------------------------------------------------------------+ +|metric | Number of flows, packets lost and throughput | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To evaluate the IaaS network performance with regards to | +| | flows and throughput, such as if and how different amounts | +| | of flows matter for the throughput between VMs on different | +| | compute blades. | +| | Typically e.g. the performance of a vSwitch | +| | depends on the number of flows running through it. Also | +| | performance of other equipment or entities can depend | +| | on the number of flows or the packet sizes used. | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs ans similar shall be stored for comparison reasons and| +| | product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc009.yaml | +| | | +| | Packet size: 64 bytes | +| | | +| | Number of ports: 1, 10, 50, 100, 500 and 1000. The amount of | +| | configured ports map from 2 up to 1001000 flows, | +| | respectively. Each port amount is run ten times, for 20 | +| | seconds each. Then the next port_amount is run, and so on. | +| | | +| | The client and server are distributed on different HW. | +| | | +| | For SLA max_ppm is set to 1000. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | pktgen | +| | | +| | (Pktgen is not always part of a Linux distribution, hence it | +| | needs to be installed. It is part of the Yardstick Docker | +| | image. | +| | As an example see the /yardstick/tools/ directory for how | +| | to generate a Linux image with pktgen included.) | +| | | ++--------------+--------------------------------------------------------------+ +|references | pktgen_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different packet sizes, amount | +| | of flows and test duration. Default values exist. | +| | | +| | SLA (optional): max_ppm: The number of packets per million | +| | packets sent that are acceptable to loose, not received. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with pktgen included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. pktgen is | +| | invoked and logs are produced and stored. | +| | | +| | Result: logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc010.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc010.rst new file mode 100644 index 000000000..ebb74ea30 --- /dev/null +++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc010.rst @@ -0,0 +1,77 @@ +************************************* +Yardstick Test Case Description TC010 +************************************* + +.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/lat_mem_rd.8.html + ++-----------------------------------------------------------------------------+ +|Memory Latency | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC010_Memory Latency | +| | | ++--------------+--------------------------------------------------------------+ +|metric | Latency in nanoseconds | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | Measure the memory read latency for varying memory sizes and | +| | strides. Whole memory hierarchy is measured including all | +| | levels of cache. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | File: opnfv_yardstick_tc010.yaml | +| | | +| | * SLA (max_latency): 30 nanoseconds | +| | * Stride - 128 bytes | +| | * Stop size - 64 megabytes | +| | * Iterations: 10 - test is run 10 times iteratively. | +| | * Interval: 1 - there is 1 second delay between each | +| | iteration. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | Lmbench | +| | | +| | Lmbench is a suite of operating system microbenchmarks. This | +| | test uses lat_mem_rd tool from that suite. | +| | Lmbench is not always part of a Linux distribution, hence it | +| | needs to be installed in the test image | +| | | ++--------------+--------------------------------------------------------------+ +|references | man-pages_ | +| | | +| | McVoy, Larry W.,and Carl Staelin. "lmbench: Portable Tools | +| | for Performance Analysis." USENIX annual technical | +| | conference 1996. | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different: | +| | | +| | * strides; | +| | * stop_size; | +| | * iterations and intervals. | +| | | +| | There are default values for each above-mentioned option. | +| | | +| | SLA (optional) : max_latency: The maximum memory latency | +| | that is accepted. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with Lmbench included in the image. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The host is installed as client. Lmbench's lat_mem_rd tool | +| | is invoked and logs are produced and stored. | +| | | +| | Result: logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Test fails if the measured memory latency is above the SLA | +| | value or if there is a test case execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc012.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc012.rst new file mode 100644 index 000000000..e7889c14e --- /dev/null +++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc012.rst @@ -0,0 +1,81 @@ +************************************* +Yardstick Test Case Description TC012 +************************************* + +.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/bw_mem.8.html + ++-----------------------------------------------------------------------------+ +|Memory Bandwidth | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC012_Memory Bandwidth | +| | | ++--------------+--------------------------------------------------------------+ +|metric | Megabyte per second (MBps) | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | Measure the rate at which data can be read from and written | +| | to the memory (this includes all levels of memory). | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | File: opnfv_yardstick_tc012.yaml | +| | | +| | * SLA (optional): 15000 (MBps) min_bw: The minimum amount of | +| | memory bandwidth that is accepted. | +| | * Size: 10 240 kB - test allocates twice that size (20 480kB)| +| | zeros it and then measures the time it takes to copy from | +| | one side to another. | +| | * Benchmark: rdwr - measures the time to read data into | +| | memory and then write data to the same location. | +| | * Warmup: 0 - the number of iterations to perform before | +| | taking actual measurements. | +| | * Iterations: 10 - test is run 10 times iteratively. | +| | * Interval: 1 - there is 1 second delay between each | +| | iteration. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | Lmbench | +| | | +| | Lmbench is a suite of operating system microbenchmarks. This | +| | test uses bw_mem tool from that suite. | +| | Lmbench is not always part of a Linux distribution, hence it | +| | needs to be installed in the test image. | +| | | ++--------------+--------------------------------------------------------------+ +|references | man-pages_ | +| | | +| | McVoy, Larry W., and Carl Staelin. "lmbench: Portable Tools | +| | for Performance Analysis." USENIX annual technical | +| | conference. 1996. | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different: | +| | | +| | * memory sizes; | +| | * memory operations (such as rd, wr, rdwr, cp, frd, fwr, | +| | fcp, bzero, bcopy); | +| | * number of warmup iterations; | +| | * iterations and intervals. | +| | | +| | There are default values for each above-mentioned option. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with Lmbench included in the image. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The host is installed as client. Lmbench's bw_mem tool is | +| | invoked and logs are produced and stored. | +| | | +| | Result: logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Test fails if the measured memory bandwidth is below the SLA | +| | value or if there is a test case execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc037.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc037.rst new file mode 100644 index 000000000..5c91f6bf1 --- /dev/null +++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc037.rst @@ -0,0 +1,99 @@ +************************************* +Yardstick Test Case Description TC037 +************************************* + +.. _cirros: https://download.cirros-cloud.net +.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt + ++-----------------------------------------------------------------------------+ +|Latency, CPU Load, Throughput, Packet Loss | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC037_Latency,CPU Load,Throughput,Packet Loss| +| | | ++--------------+--------------------------------------------------------------+ +|metric | Number of flows, latency, throughput, CPU load, packet loss | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To evaluate the IaaS network performance with regards to | +| | flows and throughput, such as if and how different amounts | +| | of flows matter for the throughput between hosts on different| +| | compute blades. Typically e.g. the performance of a vSwitch | +| | depends on the number of flows running through it. Also | +| | performance of other equipment or entities can depend | +| | on the number of flows or the packet sizes used. | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs ans similar shall be stored for comparison reasons and| +| | product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc037.yaml | +| | | +| | Packet size: 64 bytes | +| | Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. | +| | The amount configured ports map from 2 up to 1001000 flows, | +| | respectively. Each port amount is run two times, for 20 | +| | seconds each. Then the next port_amount is run, and so on. | +| | During the test CPU load on both client and server, and the | +| | network latency between the client and server are measured. | +| | The client and server are distributed on different HW. | +| | For SLA max_ppm is set to 1000. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | pktgen | +| | | +| | (Pktgen is not always part of a Linux distribution, hence it | +| | needs to be installed. It is part of the Yardstick Glance | +| | image. | +| | As an example see the /yardstick/tools/ directory for how | +| | to generate a Linux image with pktgen included.) | +| | | +| | ping | +| | | +| | Ping is normally part of any Linux distribution, hence it | +| | doesn't need to be installed. It is also part of the | +| | Yardstick Glance image. | +| | (For example also a cirros_ image can be downloaded, it | +| | includes ping) | +| | | +| | mpstat | +| | | +| | (Mpstat is not always part of a Linux distribution, hence it | +| | needs to be installed. It is part of the Yardstick Glance | +| | image. | +| | | ++--------------+--------------------------------------------------------------+ +|references | Ping and Mpstat man pages | +| | | +| | pktgen_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different packet sizes, amount | +| | of flows and test duration. Default values exist. | +| | | +| | SLA (optional): max_ppm: The number of packets per million | +| | packets sent that are acceptable to loose, not received. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with pktgen included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. pktgen is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc038.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc038.rst new file mode 100644 index 000000000..93c2cf3d8 --- /dev/null +++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc038.rst @@ -0,0 +1,99 @@ +************************************* +Yardstick Test Case Description TC038 +************************************* + +.. _cirros: https://download.cirros-cloud.net +.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt + ++-----------------------------------------------------------------------------+ +|Latency, CPU Load, Throughput, Packet Loss (Extended measurements) | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC038_Latency,CPU Load,Throughput,Packet Loss| +| | | ++--------------+--------------------------------------------------------------+ +|metric | Number of flows, latency, throughput, CPU load, packet loss | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To evaluate the IaaS network performance with regards to | +| | flows and throughput, such as if and how different amounts | +| | of flows matter for the throughput between hosts on different| +| | compute blades. Typically e.g. the performance of a vSwitch | +| | depends on the number of flows running through it. Also | +| | performance of other equipment or entities can depend | +| | on the number of flows or the packet sizes used. | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs ans similar shall be stored for comparison reasons and| +| | product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc038.yaml | +| | | +| | Packet size: 64 bytes | +| | Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. | +| | The amount configured ports map from 2 up to 1001000 flows, | +| | respectively. Each port amount is run ten times, for 20 | +| | seconds each. Then the next port_amount is run, and so on. | +| | During the test CPU load on both client and server, and the | +| | network latency between the client and server are measured. | +| | The client and server are distributed on different HW. | +| | For SLA max_ppm is set to 1000. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | pktgen | +| | | +| | (Pktgen is not always part of a Linux distribution, hence it | +| | needs to be installed. It is part of the Yardstick Glance | +| | image. | +| | As an example see the /yardstick/tools/ directory for how | +| | to generate a Linux image with pktgen included.) | +| | | +| | ping | +| | | +| | Ping is normally part of any Linux distribution, hence it | +| | doesn't need to be installed. It is also part of the | +| | Yardstick Glance image. | +| | (For example also a cirros_ image can be downloaded, it | +| | includes ping) | +| | | +| | mpstat | +| | | +| | (Mpstat is not always part of a Linux distribution, hence it | +| | needs to be installed. It is part of the Yardstick Glance | +| | image. | +| | | ++--------------+--------------------------------------------------------------+ +|references | Ping and Mpstat man pages | +| | | +| | pktgen_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different packet sizes, amount | +| | of flows and test duration. Default values exist. | +| | | +| | SLA (optional): max_ppm: The number of packets per million | +| | packets sent that are acceptable to loose, not received. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with pktgen included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. pktgen is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/testcase_description_v2_template.rst b/docs/configguide/yardstick_testcases/testcase_description_v2_template.rst new file mode 100644 index 000000000..1b8754b05 --- /dev/null +++ b/docs/configguide/yardstick_testcases/testcase_description_v2_template.rst @@ -0,0 +1,64 @@ +.. Template to be used for test case descriptions in Yardstick Project. + Write one .rst per test case. + Upload the .rst for the test case in /docs/source/yardstick directory. + Review in Gerrit. + +************************************* +Yardstick Test Case Description TCXXX +************************************* + ++-----------------------------------------------------------------------------+ +|test case slogan e.g. Network Latency | +| | ++--------------+--------------------------------------------------------------+ +|test case id | e.g. OPNFV_YARDSTICK_TC001_NW Latency | +| | | ++--------------+--------------------------------------------------------------+ +|metric | what will be measured, e.g. latency | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | describe what is the purpose of the test case | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | what .yaml file to use, state SLA if applicable, state | +| | test duration, list and describe the scenario options used in| +| | this TC and also list the options using default values. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | e.g. ping | +| | | ++--------------+--------------------------------------------------------------+ +|references | e.g. RFCxxx, ETSI-NFVyyy | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | describe variations of the test case which can be | +| | performend, e.g. run the test for different packet sizes | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | describe configuration in the tool(s) used to perform | +|conditions | the measurements (e.g. fio, pktgen), POD-specific | +| | configuration required to enable running the test | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | use this to describe tests that require sveveral steps e.g | +| | collect logs. | +| | | +| | Result: what happens in this step e.g. logs collected | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | remove interface | +| | | +| | Result: interface down. | +| | | ++--------------+--------------------------------------------------------------+ +|step N | what is done in step N | +| | | +| | Result: what happens | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | expected behavior, or SLA, pass/fail criteria | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/user_guides/framework/03-installation.rst b/docs/user_guides/framework/03-installation.rst deleted file mode 100644 index 31f8a922e..000000000 --- a/docs/user_guides/framework/03-installation.rst +++ /dev/null @@ -1,221 +0,0 @@ -.. - TODO As things will change, then this document has to be revised before the - next release. Steps: - 1. Verify that the instructions below are correct and have not been changed. - 2. Add everything that is currently missing and should be included in this document. - 3. Make sure each title has a paragraph or an introductory sentence under it. - 4. Make sure each sentence is grammatically correct and easily understandable. - 5. Remove this comment section. - -Installation -============== - -Yardstick currently supports installation on Ubuntu 14.04 or by using a Docker -image. Detailed steps about installing Yardstick using both of these options -can be found below. - -To use Yardstick you should have access to an OpenStack environment, -with at least Nova, Neutron, Glance, Keystone and Heat installed. - -The steps needed to run Yardstick are: - -1. Install Yardstick and create the test configuration .yaml file. -2. Build a guest image and load the image into the OpenStack environment. -3. Create a Neutron external network and load OpenStack environment variables. -4. Run the test case. - - -Installing Yardstick on Ubuntu 14.04 ------------------------------------- - -.. _install-framework: - -Installing Yardstick framework -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Install dependencies: -:: - - sudo apt-get install python-virtualenv python-dev - sudo apt-get install libffi-dev libssl-dev git - -Create a python virtual environment, source it and update setuptools: -:: - - virtualenv ~/yardstick_venv - source ~/yardstick_venv/bin/activate - easy_install -U setuptools - -Download source code and install python dependencies: -:: - - git clone https://gerrit.opnfv.org/gerrit/yardstick - cd yardstick - python setup.py install - -There is also a YouTube video, showing the above steps: - -.. image:: http://img.youtube.com/vi/4S4izNolmR0/0.jpg - :alt: http://www.youtube.com/watch?v=4S4izNolmR0 - :target: http://www.youtube.com/watch?v=4S4izNolmR0 - -Installing extra tools -^^^^^^^^^^^^^^^^^^^^^^ -yardstick-plot -"""""""""""""" -Yardstick has an internal plotting tool ``yardstick-plot``, which can be installed -using the following command: -:: - - python setup.py develop easy_install yardstick[plot] - -.. _guest-image: - -Building a guest image -^^^^^^^^^^^^^^^^^^^^^^ -Yardstick has a tool for building an Ubuntu Cloud Server image containing all -the required tools to run test cases supported by Yardstick. It is necessary to -have sudo rights to use this tool. - -This image can be built using the following command while in the directory where -Yardstick is installed (``~/yardstick`` if the framework is installed -by following the commands above): -:: - - sudo ./tools/yardstick-img-modify tools/ubuntu-server-cloudimg-modify.sh - -**Warning:** the script will create files by default in: -``/tmp/workspace/yardstick`` and the files will be owned by root! - -The created image can be added to OpenStack using the ``glance image-create`` or -via the OpenStack Dashboard. - -Example command: -:: - - glance --os-image-api-version 1 image-create \ - --name yardstick-trusty-server --is-public true \ - --disk-format qcow2 --container-format bare \ - --file /tmp/workspace/yardstick/yardstick-trusty-server.img - - -Installing Yardstick using Docker ---------------------------------- - -Yardstick has two Docker images, first one (**Yardstick-framework**) serves as a -replacement for installing the Yardstick framework in a virtual environment (for -example as done in :ref:`install-framework`), while the other image is mostly for -CI purposes (**Yardstick-CI**). - -Yardstick-framework image -^^^^^^^^^^^^^^^^^^^^^^^^^ -Download the source code: - -:: - - git clone https://gerrit.opnfv.org/gerrit/yardstick - -Build the Docker image and tag it as *yardstick-framework*: - -:: - - cd yardstick - docker build -t yardstick-framework . - -Run the Docker instance: - -:: - - docker run --name yardstick_instance -i -t yardstick-framework - -To build a guest image for Yardstick, see :ref:`guest-image`. - -Yardstick-CI image -^^^^^^^^^^^^^^^^^^ -Pull the Yardstick-CI Docker image from Docker hub: - -:: - - docker pull opnfv/yardstick-ci - -Run the Docker image: - -:: - - docker run \ - --privileged=true \ - --rm \ - -t \ - -e "INSTALLER_TYPE=${INSTALLER_TYPE}" \ - -e "INSTALLER_IP=${INSTALLER_IP}" \ - opnfv/yardstick-ci \ - run_benchmarks - -Where ``${INSTALLER_TYPE}`` can be fuel, foreman or compass and ``${INSTALLER_IP}`` -is the installer master node IP address (i.e. 10.20.0.2 is default for fuel). - -Basic steps performed by the **Yardstick-CI** container: - -1. clone yardstick and releng repos -2. setup OS credentials (releng scripts) -3. install yardstick and dependencies -4. build yardstick cloud image and upload it to glance -5. upload cirros-0.3.3 cloud image to glance -6. run yardstick test scenarios -7. cleanup - - -OpenStack parameters and credentials ------------------------------------- - -Yardstick-flavor -^^^^^^^^^^^^^^^^ -Most of the sample test cases in Yardstick are using an OpenStack flavor called -*yardstick-flavor* which deviates from the OpenStack standard m1.tiny flavor by the -disk size - instead of 1GB it has 3GB. Other parameters are the same as in m1.tiny. - -Environment variables -^^^^^^^^^^^^^^^^^^^^^ -Before running Yardstick it is necessary to export OpenStack environment variables -from the OpenStack *openrc* file (using the ``source`` command) and export the -external network name ``export EXTERNAL_NETWORK="external-network-name"``, -the default name for the external network is ``net04_ext``. - -Credential environment variables in the *openrc* file have to include at least: - -* OS_AUTH_URL -* OS_USERNAME -* OS_PASSWORD -* OS_TENANT_NAME - -Yardstick default key pair -^^^^^^^^^^^^^^^^^^^^^^^^^^ -Yardstick uses a SSH key pair to connect to the guest image. This key pair can -be found in the ``resources/files`` directory. To run the ``ping-hot.yaml`` test -sample, this key pair needs to be imported to the OpenStack environment. - - -Examples and verifying the install ----------------------------------- - -It is recommended to verify that Yardstick was installed successfully -by executing some simple commands and test samples. Below is an example invocation -of yardstick help command and ping.py test sample: -:: - - yardstick –h - yardstick task start samples/ping.yaml - -Each testing tool supported by Yardstick has a sample configuration file. -These configuration files can be found in the **samples** directory. - -Example invocation of ``yardstick-plot`` tool: -:: - - yardstick-plot -i /tmp/yardstick.out -o /tmp/plots/ - -Default location for the output is ``/tmp/yardstick.out``. - -More info about the tool can be found by executing: -:: - - yardstick-plot -h diff --git a/docs/user_guides/framework/index.rst b/docs/user_guides/framework/index.rst deleted file mode 100644 index f982c30ff..000000000 --- a/docs/user_guides/framework/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -================================= -Yardstick Framework Documentation -================================= - -.. toctree:: - :numbered: - :maxdepth: 2 - - 03-installation diff --git a/docs/userguide/yardstick_framework/03-installation.rst b/docs/userguide/yardstick_framework/03-installation.rst new file mode 100644 index 000000000..31f8a922e --- /dev/null +++ b/docs/userguide/yardstick_framework/03-installation.rst @@ -0,0 +1,221 @@ +.. + TODO As things will change, then this document has to be revised before the + next release. Steps: + 1. Verify that the instructions below are correct and have not been changed. + 2. Add everything that is currently missing and should be included in this document. + 3. Make sure each title has a paragraph or an introductory sentence under it. + 4. Make sure each sentence is grammatically correct and easily understandable. + 5. Remove this comment section. + +Installation +============== + +Yardstick currently supports installation on Ubuntu 14.04 or by using a Docker +image. Detailed steps about installing Yardstick using both of these options +can be found below. + +To use Yardstick you should have access to an OpenStack environment, +with at least Nova, Neutron, Glance, Keystone and Heat installed. + +The steps needed to run Yardstick are: + +1. Install Yardstick and create the test configuration .yaml file. +2. Build a guest image and load the image into the OpenStack environment. +3. Create a Neutron external network and load OpenStack environment variables. +4. Run the test case. + + +Installing Yardstick on Ubuntu 14.04 +------------------------------------ + +.. _install-framework: + +Installing Yardstick framework +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Install dependencies: +:: + + sudo apt-get install python-virtualenv python-dev + sudo apt-get install libffi-dev libssl-dev git + +Create a python virtual environment, source it and update setuptools: +:: + + virtualenv ~/yardstick_venv + source ~/yardstick_venv/bin/activate + easy_install -U setuptools + +Download source code and install python dependencies: +:: + + git clone https://gerrit.opnfv.org/gerrit/yardstick + cd yardstick + python setup.py install + +There is also a YouTube video, showing the above steps: + +.. image:: http://img.youtube.com/vi/4S4izNolmR0/0.jpg + :alt: http://www.youtube.com/watch?v=4S4izNolmR0 + :target: http://www.youtube.com/watch?v=4S4izNolmR0 + +Installing extra tools +^^^^^^^^^^^^^^^^^^^^^^ +yardstick-plot +"""""""""""""" +Yardstick has an internal plotting tool ``yardstick-plot``, which can be installed +using the following command: +:: + + python setup.py develop easy_install yardstick[plot] + +.. _guest-image: + +Building a guest image +^^^^^^^^^^^^^^^^^^^^^^ +Yardstick has a tool for building an Ubuntu Cloud Server image containing all +the required tools to run test cases supported by Yardstick. It is necessary to +have sudo rights to use this tool. + +This image can be built using the following command while in the directory where +Yardstick is installed (``~/yardstick`` if the framework is installed +by following the commands above): +:: + + sudo ./tools/yardstick-img-modify tools/ubuntu-server-cloudimg-modify.sh + +**Warning:** the script will create files by default in: +``/tmp/workspace/yardstick`` and the files will be owned by root! + +The created image can be added to OpenStack using the ``glance image-create`` or +via the OpenStack Dashboard. + +Example command: +:: + + glance --os-image-api-version 1 image-create \ + --name yardstick-trusty-server --is-public true \ + --disk-format qcow2 --container-format bare \ + --file /tmp/workspace/yardstick/yardstick-trusty-server.img + + +Installing Yardstick using Docker +--------------------------------- + +Yardstick has two Docker images, first one (**Yardstick-framework**) serves as a +replacement for installing the Yardstick framework in a virtual environment (for +example as done in :ref:`install-framework`), while the other image is mostly for +CI purposes (**Yardstick-CI**). + +Yardstick-framework image +^^^^^^^^^^^^^^^^^^^^^^^^^ +Download the source code: + +:: + + git clone https://gerrit.opnfv.org/gerrit/yardstick + +Build the Docker image and tag it as *yardstick-framework*: + +:: + + cd yardstick + docker build -t yardstick-framework . + +Run the Docker instance: + +:: + + docker run --name yardstick_instance -i -t yardstick-framework + +To build a guest image for Yardstick, see :ref:`guest-image`. + +Yardstick-CI image +^^^^^^^^^^^^^^^^^^ +Pull the Yardstick-CI Docker image from Docker hub: + +:: + + docker pull opnfv/yardstick-ci + +Run the Docker image: + +:: + + docker run \ + --privileged=true \ + --rm \ + -t \ + -e "INSTALLER_TYPE=${INSTALLER_TYPE}" \ + -e "INSTALLER_IP=${INSTALLER_IP}" \ + opnfv/yardstick-ci \ + run_benchmarks + +Where ``${INSTALLER_TYPE}`` can be fuel, foreman or compass and ``${INSTALLER_IP}`` +is the installer master node IP address (i.e. 10.20.0.2 is default for fuel). + +Basic steps performed by the **Yardstick-CI** container: + +1. clone yardstick and releng repos +2. setup OS credentials (releng scripts) +3. install yardstick and dependencies +4. build yardstick cloud image and upload it to glance +5. upload cirros-0.3.3 cloud image to glance +6. run yardstick test scenarios +7. cleanup + + +OpenStack parameters and credentials +------------------------------------ + +Yardstick-flavor +^^^^^^^^^^^^^^^^ +Most of the sample test cases in Yardstick are using an OpenStack flavor called +*yardstick-flavor* which deviates from the OpenStack standard m1.tiny flavor by the +disk size - instead of 1GB it has 3GB. Other parameters are the same as in m1.tiny. + +Environment variables +^^^^^^^^^^^^^^^^^^^^^ +Before running Yardstick it is necessary to export OpenStack environment variables +from the OpenStack *openrc* file (using the ``source`` command) and export the +external network name ``export EXTERNAL_NETWORK="external-network-name"``, +the default name for the external network is ``net04_ext``. + +Credential environment variables in the *openrc* file have to include at least: + +* OS_AUTH_URL +* OS_USERNAME +* OS_PASSWORD +* OS_TENANT_NAME + +Yardstick default key pair +^^^^^^^^^^^^^^^^^^^^^^^^^^ +Yardstick uses a SSH key pair to connect to the guest image. This key pair can +be found in the ``resources/files`` directory. To run the ``ping-hot.yaml`` test +sample, this key pair needs to be imported to the OpenStack environment. + + +Examples and verifying the install +---------------------------------- + +It is recommended to verify that Yardstick was installed successfully +by executing some simple commands and test samples. Below is an example invocation +of yardstick help command and ping.py test sample: +:: + + yardstick –h + yardstick task start samples/ping.yaml + +Each testing tool supported by Yardstick has a sample configuration file. +These configuration files can be found in the **samples** directory. + +Example invocation of ``yardstick-plot`` tool: +:: + + yardstick-plot -i /tmp/yardstick.out -o /tmp/plots/ + +Default location for the output is ``/tmp/yardstick.out``. + +More info about the tool can be found by executing: +:: + + yardstick-plot -h diff --git a/docs/userguide/yardstick_framework/index.rst b/docs/userguide/yardstick_framework/index.rst new file mode 100644 index 000000000..f982c30ff --- /dev/null +++ b/docs/userguide/yardstick_framework/index.rst @@ -0,0 +1,9 @@ +================================= +Yardstick Framework Documentation +================================= + +.. toctree:: + :numbered: + :maxdepth: 2 + + 03-installation diff --git a/docs/vTC/README.rst b/docs/vTC/README.rst deleted file mode 100644 index ae6fefa59..000000000 --- a/docs/vTC/README.rst +++ /dev/null @@ -1,87 +0,0 @@ -========================== -Virtual Traffic Classifier -========================== - -Overview -======== - -The virtual Traffic Classifier VNF [1], comprises in the current version of -1 VNFC [2]. The VNFC contains both the Traffic Inspection module, and the -Traffic forwarding module, needed to run the VNF. The exploitation of DPI -methods for traffic classification is built around two basic assumptions: - -(i) third parties unaffiliated with either source or recipient are able to -inspect each IP packet’s payload and -(ii) the classifier knows the relevant syntax of each application’s packet -payloads (protocol signatures, data patterns, etc.). - -The proposed DPI based approach will only use an indicative, small number of -the initial packets from each flow in order to identify the content and not -inspect each packet. - -In this respect it follows the Packet Based per Flow State (PBFS). -This method uses a table to track each session based on the 5-tuples -(src address,dest address,src port,dest port,transport protocol) -that is maintained for each flow. - -Concepts -======== - -Traffic Inspection: The process of packet analysis and application -identification of network traffic that passes through the vTC. - -Traffic Forwarding: The process of packet forwarding from an incoming -network interface to a pre-defined outgoing network interface. - -Traffic Rule Application: The process of packet tagging, based on a -predefined set of rules. Packet tagging may include e.g. ToS field -modification. - -Architecture -============ - -The Traffic Inspection module is the most computationally intensive component -of the VNF. It implements filtering and packet matching algorithms in order to -support the enhanced traffic forwarding capability of the VNF. The component -supports a flow table (exploiting hashing algorithms for fast indexing of -flows) and an inspection engine for traffic classification. - -The implementation used for these experiments exploits the nDPI library. -The packet capturing mechanism is implemented using libpcap. When the DPI -engine identifies a new flow, the flow register is updated with the -appropriate information and transmitted across the Traffic Forwarding module, -which then applies any required policy updates. - -The Traffic Forwarding moudle is responsible for routing and packet forwarding. -It accepts incoming network traffic, consults the flow table for classification -information for each incoming flow and then applies pre-defined policies -marking e.g. type of Service/Differentiated Services Code Point (TOS/DSCP) -multimedia traffic for QoS enablement on the forwarded traffic. -It is assumed that the traffic is forwarded using the default policy until it -is identified and new policies are enforced. - -The expected response delay is considered to be negligible,as only a small -number of packets are required to identify each flow. - -Graphical Overview -================== - -Install -======= - -run the build.sh with root privileges - -Run -=== - -sudo ./pfbridge -a eth1 -b eth2 - -Custom Image -============ - -TBD - -Development Environment -======================= - -Ubuntu 14.04 >= VM diff --git a/docs/vTC/abbreviations.rst b/docs/vTC/abbreviations.rst deleted file mode 100644 index a713ee66b..000000000 --- a/docs/vTC/abbreviations.rst +++ /dev/null @@ -1,5 +0,0 @@ -Abbreviations for the virtual Traffic Classifier -================================================ - -[1] VNF - Virtual Network Function -[2] VNFC - Virtual Network Function Component diff --git a/docs/yardstick/01-introduction.rst b/docs/yardstick/01-introduction.rst deleted file mode 100644 index 6cca2875e..000000000 --- a/docs/yardstick/01-introduction.rst +++ /dev/null @@ -1,38 +0,0 @@ -============ -Introduction -============ - -**Welcome to Yardstick's documentation !** - -.. _Yardstick: https://wiki.opnfv.org/yardstick - -Yardstick_ is an OPNFV Project. - -The project's goal is to verify infrastructure compliance, from the perspective -of a :term:`VNF`. - -The Project's scope is the development of a test framework, *Yardstick*, test -cases and test stimuli to enable :term:`NFVI` verification. -The Project also includes a sample :term:`VNF`, the :term:`VTC` and its -experimental framework, *ApexLake* ! - -The chapter :doc:`02-methodology` describes the methodology implemented by the -Yardstick Project for :term:`NFVI` verification. The chapter -:doc:`03-list-of-tcs` includes a list of available Yardstick test cases. - -Yardstick is used for verifying the OPNFV infrastructure and some of the OPNFV -features, listed in :doc:`03-list-of-tcs`. - -The *Yardstick* framework is deployed in several OPNFV community labs. It is -installer, infrastructure and application independent. - -.. _Pharos: https://wiki.opnfv.org/pharos - -.. seealso:: Pharos_ for information on OPNFV community labs. - -Contact Yardstick -================= - -Feedback? `Contact us`_ - -.. _Contact us: opnfv-users@lists.opnfv.org diff --git a/docs/yardstick/02-methodology.rst b/docs/yardstick/02-methodology.rst deleted file mode 100644 index 5097c566b..000000000 --- a/docs/yardstick/02-methodology.rst +++ /dev/null @@ -1,181 +0,0 @@ -=========== -Methodology -=========== - -Abstract -======== - -This chapter describes the methodology implemented by the Yardstick project for -verifying the NFV Infrastructure from the perspective of a VNF. - -ETSI-NFV -======== - -.. _NFV-TST001: https://docbox.etsi.org/ISG/NFV/Open/Drafts/TST001_-_Pre-deployment_Validation/ - -The document ETSI GS NFV-TST001_, "Pre-deployment Testing; Report on Validation -of NFV Environments and Services", recommends methods for pre-deployment -testing of the functional components of an NFV environment. - -The Yardstick project implements the methodology described in chapter 6, "Pre- -deployment validation of NFV infrastructure". - -The methodology consists in decomposing the typical VNF work-load performance -metrics into a number of characteristics/performance vectors, which each can be -represented by distinct test-cases. - -The methodology includes five steps: - -* *Step1:* Define Infrastruture - the HW, SW and corresponding configuration - target for validation; the OPNFV infrastructure, in OPNFV community labs. - -* *Step2:* Identify VNF type - the application for which the infrastructure is - to be validated, and its requirements on the underlying infrastructure. - -* *Step3:* Select test cases - depending on the workload that represents the - application for which the infrastruture is to be validated, the relevant - test cases amongst the list of available Yardstick test cases. - -* *Step4:* Execute tests - define the duration and number of iterations for the - selected test cases, tests runs are automated via OPNFV Jenkins Jobs. - -* *Step5:* Collect results - using the common API for result collection. - -Metrics -======= - -The metrics, as defined by ETSI GS NFV-TST001, are shown in -:ref:`Table1 `, :ref:`Table2 ` and -:ref:`Table3 `. - -In OPNFV Brahmaputra release, generic test cases covering aspects of the listed -metrics are available; further OPNFV releases will provide extended testing of -these metrics. -The view of available Yardstick test cases cross ETSI definitions in -:ref:`Table1 `, :ref:`Table2 ` and :ref:`Table3 ` -is shown in :ref:`Table4 `. -It shall be noticed that the Yardstick test cases are examples, the test -duration and number of iterations are configurable, as are the System Under -Test (SUT) and the attributes (or, in Yardstick nomemclature, the scenario -options). - -.. _table2_1: - -**Table 1 - Performance/Speed Metrics** - -+---------+-------------------------------------------------------------------+ -| Category| Performance/Speed | -| | | -+---------+-------------------------------------------------------------------+ -| Compute | * Latency for random memory access | -| | * Latency for cache read/write operations | -| | * Processing speed (instructions per second) | -| | * Throughput for random memory access (bytes per second) | -| | | -+---------+-------------------------------------------------------------------+ -| Network | * Throughput per NFVI node (frames/byte per second) | -| | * Throughput provided to a VM (frames/byte per second) | -| | * Latency per traffic flow | -| | * Latency between VMs | -| | * Latency between NFVI nodes | -| | * Packet delay variation (jitter) between VMs | -| | * Packet delay variation (jitter) between NFVI nodes | -| | | -+---------+-------------------------------------------------------------------+ -| Storage | * Sequential read/write IOPS | -| | * Random read/write IOPS | -| | * Latency for storage read/write operations | -| | * Throughput for storage read/write operations | -| | | -+---------+-------------------------------------------------------------------+ - -.. _table2_2: - -**Table 2 - Capacity/Scale Metrics** - -+---------+-------------------------------------------------------------------+ -| Category| Capacity/Scale | -| | | -+---------+-------------------------------------------------------------------+ -| Compute | * Number of cores and threads- Available memory size | -| | * Cache size | -| | * Processor utilization (max, average, standard deviation) | -| | * Memory utilization (max, average, standard deviation) | -| | * Cache utilization (max, average, standard deviation) | -| | | -+---------+-------------------------------------------------------------------+ -| Network | * Number of connections | -| | * Number of frames sent/received | -| | * Maximum throughput between VMs (frames/byte per second) | -| | * Maximum throughput between NFVI nodes (frames/byte per second) | -| | * Network utilization (max, average, standard deviation) | -| | * Number of traffic flows | -| | | -+---------+-------------------------------------------------------------------+ -| Storage | * Storage/Disk size | -| | * Capacity allocation (block-based, object-based) | -| | * Block size | -| | * Maximum sequential read/write IOPS | -| | * Maximum random read/write IOPS | -| | * Disk utilization (max, average, standard deviation) | -| | | -+---------+-------------------------------------------------------------------+ - -.. _table2_3: - -**Table 3 - Availability/Reliability Metrics** - -+---------+-------------------------------------------------------------------+ -| Category| Availability/Reliability | -| | | -+---------+-------------------------------------------------------------------+ -| Compute | * Processor availability (Error free processing time) | -| | * Memory availability (Error free memory time) | -| | * Processor mean-time-to-failure | -| | * Memory mean-time-to-failure | -| | * Number of processing faults per second | -| | | -+---------+-------------------------------------------------------------------+ -| Network | * NIC availability (Error free connection time) | -| | * Link availability (Error free transmission time) | -| | * NIC mean-time-to-failure | -| | * Network timeout duration due to link failure | -| | * Frame loss rate | -| | | -+---------+-------------------------------------------------------------------+ -| Storage | * Disk availability (Error free disk access time) | -| | * Disk mean-time-to-failure | -| | * Number of failed storage read/write operations per second | -| | | -+---------+-------------------------------------------------------------------+ - -.. _table2_4: - -**Table 4 - Yardstick Generic Test Cases** - -+---------+-------------------+----------------+------------------------------+ -| Category| Performance/Speed | Capacity/Scale | Availability/Reliability | -| | | | | -+---------+-------------------+----------------+------------------------------+ -| Compute | TC003 | TC003 | TC013 [1]_ | -| | TC004 | TC004 | TC015 [1]_ | -| | TC014 | TC010 | | -| | TC024 | TC012 | | -| | | | | -+---------+-------------------+----------------+------------------------------+ -| Network | TC002 | TC001 | TC016 [1]_ | -| | TC011 | TC008 | TC018 [1]_ | -| | | TC009 | | -| | | | | -+---------+-------------------+----------------+------------------------------+ -| Storage | TC005 | TC005 | TC017 [1]_ | -| | | | | -+---------+-------------------+----------------+------------------------------+ - -.. note:: The description in this OPNFV document is intended as a reference for - users to understand the scope of the Yardstick Project and the - deliverables of the Yardstick framework. For complete description of - the methodology, refer to the ETSI document. - -.. rubric:: Footnotes -.. [1] To be included in future deliveries. diff --git a/docs/yardstick/03-list-of-tcs.rst b/docs/yardstick/03-list-of-tcs.rst deleted file mode 100644 index 9e951ffab..000000000 --- a/docs/yardstick/03-list-of-tcs.rst +++ /dev/null @@ -1,44 +0,0 @@ -==================== -Yardstick Test Cases -==================== - -Abstract -======== - -This chapter lists available Yardstick test cases. -Yardstick test cases are divided in two main categories: - -* *Generic NFVI Test Cases* - Test Cases developed to realize the methodology -described in :doc:`02-methodology` - -* *OPNFV Feature Test Cases* - Test Cases developed to verify one or more -aspect of a feature delivered by an OPNFV Project. - -Generic NFVI Test Case Descriptions -=================================== - -.. toctree:: - :maxdepth: 1 - - opnfv_yardstick_tc001.rst - opnfv_yardstick_tc002.rst - opnfv_yardstick_tc008.rst - opnfv_yardstick_tc009.rst - opnfv_yardstick_tc010.rst - opnfv_yardstick_tc012.rst - opnfv_yardstick_tc037.rst - opnfv_yardstick_tc038.rst - -OPNFV Feature Test Cases -======================== - - - -Templates -========= - -.. toctree:: - :maxdepth: 1 - - testcase_description_v2_template - Yardstick_task_templates diff --git a/docs/yardstick/Yardstick_task_templates.rst b/docs/yardstick/Yardstick_task_templates.rst deleted file mode 100755 index d2c2b7ec9..000000000 --- a/docs/yardstick/Yardstick_task_templates.rst +++ /dev/null @@ -1,155 +0,0 @@ -Task Template Syntax -==================== - -Basic template syntax ---------------------- -A nice feature of the input task format used in Yardstick is that it supports -the template syntax based on Jinja2. -This turns out to be extremely useful when, say, you have a fixed structure of -your task but you want to parameterize this task in some way. -For example, imagine your input task file (task.yaml) runs a set of Ping -scenarios: - -:: - - # Sample benchmark task config file - # measure network latency using ping - schema: "yardstick:task:0.1" - - scenarios: - - - type: Ping - options: - packetsize: 200 - host: athena.demo - target: ares.demo - - runner: - type: Duration - duration: 60 - interval: 1 - - sla: - max_rtt: 10 - action: monitor - - context: - ... - -Let's say you want to run the same set of scenarios with the same runner/ -context/sla, but you want to try another packetsize to compare the performance. -The most elegant solution is then to turn the packetsize name into a template -variable: - -:: - - # Sample benchmark task config file - # measure network latency using ping - - schema: "yardstick:task:0.1" - scenarios: - - - type: Ping - options: - packetsize: {{packetsize}} - host: athena.demo - target: ares.demo - - runner: - type: Duration - duration: 60 - interval: 1 - - sla: - max_rtt: 10 - action: monitor - - context: - ... - -and then pass the argument value for {{packetsize}} when starting a task with -this configuration file. -Yardstick provides you with different ways to do that: - -1.Pass the argument values directly in the command-line interface (with either -a JSON or YAML dictionary): - -:: - - yardstick task start samples/ping-template.yaml - --task-args'{"packetsize":"200"}' - -2.Refer to a file that specifies the argument values (JSON/YAML): - -:: - - yardstick task start samples/ping-template.yaml --task-args-file args.yaml - -Using the default values ------------------------- -Note that the Jinja2 template syntax allows you to set the default values for -your parameters. -With default values set, your task file will work even if you don't -parameterize it explicitly while starting a task. -The default values should be set using the {% set ... %} clause (task.yaml). -For example: - -:: - - # Sample benchmark task config file - # measure network latency using ping - schema: "yardstick:task:0.1" - {% set packetsize = packetsize or "100" %} - scenarios: - - - type: Ping - options: - packetsize: {{packetsize}} - host: athena.demo - target: ares.demo - - runner: - type: Duration - duration: 60 - interval: 1 - ... - -If you don't pass the value for {{packetsize}} while starting a task, the -default one will be used. - -Advanced templates ------------------- - -Yardstick makes it possible to use all the power of Jinja2 template syntax, -including the mechanism of built-in functions. -As an example, let us make up a task file that will do a block storage -performance test. -The input task file (fio-template.yaml) below uses the Jinja2 for-endfor -construct to accomplish that: - -:: - - #Test block sizes of 4KB, 8KB, 64KB, 1MB - #Test 5 workloads: read, write, randwrite, randread, rw - schema: "yardstick:task:0.1" - - scenarios: - {% for bs in ['4k', '8k', '64k', '1024k' ] %} - {% for rw in ['read', 'write', 'randwrite', 'randread', 'rw' ] %} - - - type: Fio - options: - filename: /home/ec2-user/data.raw - bs: {{bs}} - rw: {{rw}} - ramp_time: 10 - host: fio.demo - runner: - type: Duration - duration: 60 - interval: 60 - - {% endfor %} - {% endfor %} - context - ... diff --git a/docs/yardstick/glossary.rst b/docs/yardstick/glossary.rst deleted file mode 100644 index 67e126f58..000000000 --- a/docs/yardstick/glossary.rst +++ /dev/null @@ -1,15 +0,0 @@ -================== -Yardstick Glossary -================== - -.. glossary:: - :sorted: - - VNF - Virtual Network Function - - NFVI - Network Function Virtualization Infrastructure - - VTC - Virtual Traffic Classifier diff --git a/docs/yardstick/index.rst b/docs/yardstick/index.rst deleted file mode 100644 index ccea55ccb..000000000 --- a/docs/yardstick/index.rst +++ /dev/null @@ -1,11 +0,0 @@ -================== -Yardstick Overview -================== - -.. toctree:: - :maxdepth: 2 - - 01-introduction - 02-methodology - 03-list-of-tcs - glossary diff --git a/docs/yardstick/opnfv_yardstick_tc001.rst b/docs/yardstick/opnfv_yardstick_tc001.rst deleted file mode 100644 index 810bad489..000000000 --- a/docs/yardstick/opnfv_yardstick_tc001.rst +++ /dev/null @@ -1,79 +0,0 @@ -************************************* -Yardstick Test Case Description TC001 -************************************* - -.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt - -+-----------------------------------------------------------------------------+ -|Network Performance | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC001_NW PERF | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Number of flows and throughput | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS network performance with regards to | -| | flows and throughput, such as if and how different amounts | -| | of flows matter for the throughput between hosts on different| -| | compute blades. Typically e.g. the performance of a vSwitch | -| | depends on the number of flows running through it. Also | -| | performance of other equipment or entities can depend | -| | on the number of flows or the packet sizes used. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc001.yaml | -| | | -| | Packet size: 60 bytes | -| | Number of ports: 10, 50, 100, 500 and 1000, where each | -| | runs for 20 seconds. The whole sequence is run | -| | twice. The client and server are distributed on different HW.| -| | For SLA max_ppm is set to 1000. The amount of configured | -| | ports map to between 110 up to 1001000 flows, respectively. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | pktgen | -| | | -| | (Pktgen is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Docker | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | pktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different packet sizes, amount | -| | of flows and test duration. Default values exist. | -| | | -| | SLA (optional): max_ppm: The number of packets per million | -| | packets sent that are acceptable to loose, not received. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with pktgen included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. pktgen is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/yardstick/opnfv_yardstick_tc002.rst b/docs/yardstick/opnfv_yardstick_tc002.rst deleted file mode 100644 index 56350f5bb..000000000 --- a/docs/yardstick/opnfv_yardstick_tc002.rst +++ /dev/null @@ -1,78 +0,0 @@ -************************************* -Yardstick Test Case Description TC002 -************************************* - -.. _cirros-image: https://download.cirros-cloud.net - -+-----------------------------------------------------------------------------+ -|Network Latency | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC002_NW LATENCY | -| | | -+--------------+--------------------------------------------------------------+ -|metric | RTT, Round Trip Time | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To do a basic verification that network latency is within | -| | acceptable boundaries when packets travel between hosts | -| | located on same or different compute blades. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs and similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc002.yaml | -| | | -| | Packet size 100 bytes. Total test duration 600 seconds. | -| | One ping each 10 seconds. SLA RTT is set to maximum 10 ms. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | ping | -| | | -| | Ping is normally part of any Linux distribution, hence it | -| | doesn't need to be installed. It is also part of the | -| | Yardstick Docker image. | -| | (For example also a Cirros image can be downloaded from | -| | cirros-image_, it includes ping) | -| | | -+--------------+--------------------------------------------------------------+ -|references | Ping man page | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test case can be configured with different packet sizes, | -| | burst sizes, ping intervals and test duration. | -| | SLA is optional. The SLA in this test case serves as an | -| | example. Considerably lower RTT is expected, and | -| | also normal to achieve in balanced L2 environments. However, | -| | to cover most configurations, both bare metal and fully | -| | virtualized ones, this value should be possible to achieve | -| | and acceptable for black box testing. Many real time | -| | applications start to suffer badly if the RTT time is higher | -| | than this. Some may suffer bad also close to this RTT, while | -| | others may not suffer at all. It is a compromise that may | -| | have to be tuned for different configuration purposes. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with ping included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. Ping is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Test should not PASS if any RTT is above the optional SLA | -| | value, or if there is a test case execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/yardstick/opnfv_yardstick_tc008.rst b/docs/yardstick/opnfv_yardstick_tc008.rst deleted file mode 100644 index e176e633a..000000000 --- a/docs/yardstick/opnfv_yardstick_tc008.rst +++ /dev/null @@ -1,85 +0,0 @@ -************************************* -Yardstick Test Case Description TC008 -************************************* - -.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt - -+-----------------------------------------------------------------------------+ -|Packet Loss Extended Test | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC008_NW PERF, Packet loss Extended Test | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Number of flows, packet size and throughput | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS network performance with regards to | -| | flows and throughput, such as if and how different amounts | -| | of packet sizes and flows matter for the throughput between | -| | VMs on different compute blades. Typically e.g. the | -| | performance of a vSwitch | -| | depends on the number of flows running through it. Also | -| | performance of other equipment or entities can depend | -| | on the number of flows or the packet sizes used. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc008.yaml | -| | | -| | Packet size: 64, 128, 256, 512, 1024, 1280 and 1518 bytes. | -| | | -| | Number of ports: 1, 10, 50, 100, 500 and 1000. The amount of | -| | configured ports map from 2 up to 1001000 flows, | -| | respectively. Each packet_size/port_amount combination is run| -| | ten times, for 20 seconds each. Then the next | -| | packet_size/port_amount combination is run, and so on. | -| | | -| | The client and server are distributed on different HW. | -| | | -| | For SLA max_ppm is set to 1000. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | pktgen | -| | | -| | (Pktgen is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Docker | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | pktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different packet sizes, amount | -| | of flows and test duration. Default values exist. | -| | | -| | SLA (optional): max_ppm: The number of packets per million | -| | packets sent that are acceptable to loose, not received. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with pktgen included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. pktgen is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/yardstick/opnfv_yardstick_tc009.rst b/docs/yardstick/opnfv_yardstick_tc009.rst deleted file mode 100644 index e4002a884..000000000 --- a/docs/yardstick/opnfv_yardstick_tc009.rst +++ /dev/null @@ -1,84 +0,0 @@ -************************************* -Yardstick Test Case Description TC009 -************************************* - -.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt - -+-----------------------------------------------------------------------------+ -|Packet Loss | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC009_NW PERF, Packet loss | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Number of flows, packets lost and throughput | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS network performance with regards to | -| | flows and throughput, such as if and how different amounts | -| | of flows matter for the throughput between VMs on different | -| | compute blades. | -| | Typically e.g. the performance of a vSwitch | -| | depends on the number of flows running through it. Also | -| | performance of other equipment or entities can depend | -| | on the number of flows or the packet sizes used. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc009.yaml | -| | | -| | Packet size: 64 bytes | -| | | -| | Number of ports: 1, 10, 50, 100, 500 and 1000. The amount of | -| | configured ports map from 2 up to 1001000 flows, | -| | respectively. Each port amount is run ten times, for 20 | -| | seconds each. Then the next port_amount is run, and so on. | -| | | -| | The client and server are distributed on different HW. | -| | | -| | For SLA max_ppm is set to 1000. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | pktgen | -| | | -| | (Pktgen is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Docker | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | pktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different packet sizes, amount | -| | of flows and test duration. Default values exist. | -| | | -| | SLA (optional): max_ppm: The number of packets per million | -| | packets sent that are acceptable to loose, not received. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with pktgen included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. pktgen is | -| | invoked and logs are produced and stored. | -| | | -| | Result: logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/yardstick/opnfv_yardstick_tc010.rst b/docs/yardstick/opnfv_yardstick_tc010.rst deleted file mode 100644 index ebb74ea30..000000000 --- a/docs/yardstick/opnfv_yardstick_tc010.rst +++ /dev/null @@ -1,77 +0,0 @@ -************************************* -Yardstick Test Case Description TC010 -************************************* - -.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/lat_mem_rd.8.html - -+-----------------------------------------------------------------------------+ -|Memory Latency | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC010_Memory Latency | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Latency in nanoseconds | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | Measure the memory read latency for varying memory sizes and | -| | strides. Whole memory hierarchy is measured including all | -| | levels of cache. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | File: opnfv_yardstick_tc010.yaml | -| | | -| | * SLA (max_latency): 30 nanoseconds | -| | * Stride - 128 bytes | -| | * Stop size - 64 megabytes | -| | * Iterations: 10 - test is run 10 times iteratively. | -| | * Interval: 1 - there is 1 second delay between each | -| | iteration. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | Lmbench | -| | | -| | Lmbench is a suite of operating system microbenchmarks. This | -| | test uses lat_mem_rd tool from that suite. | -| | Lmbench is not always part of a Linux distribution, hence it | -| | needs to be installed in the test image | -| | | -+--------------+--------------------------------------------------------------+ -|references | man-pages_ | -| | | -| | McVoy, Larry W.,and Carl Staelin. "lmbench: Portable Tools | -| | for Performance Analysis." USENIX annual technical | -| | conference 1996. | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different: | -| | | -| | * strides; | -| | * stop_size; | -| | * iterations and intervals. | -| | | -| | There are default values for each above-mentioned option. | -| | | -| | SLA (optional) : max_latency: The maximum memory latency | -| | that is accepted. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with Lmbench included in the image. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The host is installed as client. Lmbench's lat_mem_rd tool | -| | is invoked and logs are produced and stored. | -| | | -| | Result: logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Test fails if the measured memory latency is above the SLA | -| | value or if there is a test case execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/yardstick/opnfv_yardstick_tc012.rst b/docs/yardstick/opnfv_yardstick_tc012.rst deleted file mode 100644 index e7889c14e..000000000 --- a/docs/yardstick/opnfv_yardstick_tc012.rst +++ /dev/null @@ -1,81 +0,0 @@ -************************************* -Yardstick Test Case Description TC012 -************************************* - -.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/bw_mem.8.html - -+-----------------------------------------------------------------------------+ -|Memory Bandwidth | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC012_Memory Bandwidth | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Megabyte per second (MBps) | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | Measure the rate at which data can be read from and written | -| | to the memory (this includes all levels of memory). | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | File: opnfv_yardstick_tc012.yaml | -| | | -| | * SLA (optional): 15000 (MBps) min_bw: The minimum amount of | -| | memory bandwidth that is accepted. | -| | * Size: 10 240 kB - test allocates twice that size (20 480kB)| -| | zeros it and then measures the time it takes to copy from | -| | one side to another. | -| | * Benchmark: rdwr - measures the time to read data into | -| | memory and then write data to the same location. | -| | * Warmup: 0 - the number of iterations to perform before | -| | taking actual measurements. | -| | * Iterations: 10 - test is run 10 times iteratively. | -| | * Interval: 1 - there is 1 second delay between each | -| | iteration. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | Lmbench | -| | | -| | Lmbench is a suite of operating system microbenchmarks. This | -| | test uses bw_mem tool from that suite. | -| | Lmbench is not always part of a Linux distribution, hence it | -| | needs to be installed in the test image. | -| | | -+--------------+--------------------------------------------------------------+ -|references | man-pages_ | -| | | -| | McVoy, Larry W., and Carl Staelin. "lmbench: Portable Tools | -| | for Performance Analysis." USENIX annual technical | -| | conference. 1996. | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different: | -| | | -| | * memory sizes; | -| | * memory operations (such as rd, wr, rdwr, cp, frd, fwr, | -| | fcp, bzero, bcopy); | -| | * number of warmup iterations; | -| | * iterations and intervals. | -| | | -| | There are default values for each above-mentioned option. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with Lmbench included in the image. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The host is installed as client. Lmbench's bw_mem tool is | -| | invoked and logs are produced and stored. | -| | | -| | Result: logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Test fails if the measured memory bandwidth is below the SLA | -| | value or if there is a test case execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/yardstick/opnfv_yardstick_tc037.rst b/docs/yardstick/opnfv_yardstick_tc037.rst deleted file mode 100644 index 5c91f6bf1..000000000 --- a/docs/yardstick/opnfv_yardstick_tc037.rst +++ /dev/null @@ -1,99 +0,0 @@ -************************************* -Yardstick Test Case Description TC037 -************************************* - -.. _cirros: https://download.cirros-cloud.net -.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt - -+-----------------------------------------------------------------------------+ -|Latency, CPU Load, Throughput, Packet Loss | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC037_Latency,CPU Load,Throughput,Packet Loss| -| | | -+--------------+--------------------------------------------------------------+ -|metric | Number of flows, latency, throughput, CPU load, packet loss | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS network performance with regards to | -| | flows and throughput, such as if and how different amounts | -| | of flows matter for the throughput between hosts on different| -| | compute blades. Typically e.g. the performance of a vSwitch | -| | depends on the number of flows running through it. Also | -| | performance of other equipment or entities can depend | -| | on the number of flows or the packet sizes used. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc037.yaml | -| | | -| | Packet size: 64 bytes | -| | Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. | -| | The amount configured ports map from 2 up to 1001000 flows, | -| | respectively. Each port amount is run two times, for 20 | -| | seconds each. Then the next port_amount is run, and so on. | -| | During the test CPU load on both client and server, and the | -| | network latency between the client and server are measured. | -| | The client and server are distributed on different HW. | -| | For SLA max_ppm is set to 1000. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | pktgen | -| | | -| | (Pktgen is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Glance | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -| | ping | -| | | -| | Ping is normally part of any Linux distribution, hence it | -| | doesn't need to be installed. It is also part of the | -| | Yardstick Glance image. | -| | (For example also a cirros_ image can be downloaded, it | -| | includes ping) | -| | | -| | mpstat | -| | | -| | (Mpstat is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Glance | -| | image. | -| | | -+--------------+--------------------------------------------------------------+ -|references | Ping and Mpstat man pages | -| | | -| | pktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different packet sizes, amount | -| | of flows and test duration. Default values exist. | -| | | -| | SLA (optional): max_ppm: The number of packets per million | -| | packets sent that are acceptable to loose, not received. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with pktgen included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. pktgen is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/yardstick/opnfv_yardstick_tc038.rst b/docs/yardstick/opnfv_yardstick_tc038.rst deleted file mode 100644 index 93c2cf3d8..000000000 --- a/docs/yardstick/opnfv_yardstick_tc038.rst +++ /dev/null @@ -1,99 +0,0 @@ -************************************* -Yardstick Test Case Description TC038 -************************************* - -.. _cirros: https://download.cirros-cloud.net -.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt - -+-----------------------------------------------------------------------------+ -|Latency, CPU Load, Throughput, Packet Loss (Extended measurements) | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC038_Latency,CPU Load,Throughput,Packet Loss| -| | | -+--------------+--------------------------------------------------------------+ -|metric | Number of flows, latency, throughput, CPU load, packet loss | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS network performance with regards to | -| | flows and throughput, such as if and how different amounts | -| | of flows matter for the throughput between hosts on different| -| | compute blades. Typically e.g. the performance of a vSwitch | -| | depends on the number of flows running through it. Also | -| | performance of other equipment or entities can depend | -| | on the number of flows or the packet sizes used. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc038.yaml | -| | | -| | Packet size: 64 bytes | -| | Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. | -| | The amount configured ports map from 2 up to 1001000 flows, | -| | respectively. Each port amount is run ten times, for 20 | -| | seconds each. Then the next port_amount is run, and so on. | -| | During the test CPU load on both client and server, and the | -| | network latency between the client and server are measured. | -| | The client and server are distributed on different HW. | -| | For SLA max_ppm is set to 1000. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | pktgen | -| | | -| | (Pktgen is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Glance | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -| | ping | -| | | -| | Ping is normally part of any Linux distribution, hence it | -| | doesn't need to be installed. It is also part of the | -| | Yardstick Glance image. | -| | (For example also a cirros_ image can be downloaded, it | -| | includes ping) | -| | | -| | mpstat | -| | | -| | (Mpstat is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Glance | -| | image. | -| | | -+--------------+--------------------------------------------------------------+ -|references | Ping and Mpstat man pages | -| | | -| | pktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different packet sizes, amount | -| | of flows and test duration. Default values exist. | -| | | -| | SLA (optional): max_ppm: The number of packets per million | -| | packets sent that are acceptable to loose, not received. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with pktgen included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. pktgen is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/yardstick/testcase_description_v2_template.rst b/docs/yardstick/testcase_description_v2_template.rst deleted file mode 100644 index 1b8754b05..000000000 --- a/docs/yardstick/testcase_description_v2_template.rst +++ /dev/null @@ -1,64 +0,0 @@ -.. Template to be used for test case descriptions in Yardstick Project. - Write one .rst per test case. - Upload the .rst for the test case in /docs/source/yardstick directory. - Review in Gerrit. - -************************************* -Yardstick Test Case Description TCXXX -************************************* - -+-----------------------------------------------------------------------------+ -|test case slogan e.g. Network Latency | -| | -+--------------+--------------------------------------------------------------+ -|test case id | e.g. OPNFV_YARDSTICK_TC001_NW Latency | -| | | -+--------------+--------------------------------------------------------------+ -|metric | what will be measured, e.g. latency | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | describe what is the purpose of the test case | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | what .yaml file to use, state SLA if applicable, state | -| | test duration, list and describe the scenario options used in| -| | this TC and also list the options using default values. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | e.g. ping | -| | | -+--------------+--------------------------------------------------------------+ -|references | e.g. RFCxxx, ETSI-NFVyyy | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | describe variations of the test case which can be | -| | performend, e.g. run the test for different packet sizes | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | describe configuration in the tool(s) used to perform | -|conditions | the measurements (e.g. fio, pktgen), POD-specific | -| | configuration required to enable running the test | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | use this to describe tests that require sveveral steps e.g | -| | collect logs. | -| | | -| | Result: what happens in this step e.g. logs collected | -| | | -+--------------+--------------------------------------------------------------+ -|step 2 | remove interface | -| | | -| | Result: interface down. | -| | | -+--------------+--------------------------------------------------------------+ -|step N | what is done in step N | -| | | -| | Result: what happens | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | expected behavior, or SLA, pass/fail criteria | -| | | -+--------------+--------------------------------------------------------------+ -- cgit 1.2.3-korg