diff options
| -rw-r--r-- | docs/templates/testcase_description_v2_template.rst | 43 | ||||
| -rw-r--r-- | docs/yardstick/01-introduction.rst | 38 | ||||
| -rw-r--r-- | docs/yardstick/02-methodology.rst | 181 | ||||
| -rw-r--r-- | docs/yardstick/03-list-of-tcs.rst | 42 | ||||
| -rwxr-xr-x | docs/yardstick/Yardstick_task_templates.rst | 155 | ||||
| -rw-r--r-- | docs/yardstick/glossary.rst | 15 | ||||
| -rw-r--r-- | docs/yardstick/index.rst | 26 | ||||
| -rw-r--r-- | docs/yardstick/opnfv_yardstick_tc001.rst | 42 | ||||
| -rw-r--r-- | docs/yardstick/opnfv_yardstick_tc002.rst | 35 | ||||
| -rw-r--r-- | docs/yardstick/opnfv_yardstick_tc008.rst | 44 | ||||
| -rw-r--r-- | docs/yardstick/opnfv_yardstick_tc009.rst | 46 | ||||
| -rw-r--r-- | docs/yardstick/opnfv_yardstick_tc010.rst | 58 | ||||
| -rw-r--r-- | docs/yardstick/opnfv_yardstick_tc012.rst | 61 | ||||
| -rw-r--r-- | docs/yardstick/testcase_description_v2_template.rst | 64 |
14 files changed, 717 insertions, 133 deletions
diff --git a/docs/templates/testcase_description_v2_template.rst b/docs/templates/testcase_description_v2_template.rst index da90f561e..1b8754b05 100644 --- a/docs/templates/testcase_description_v2_template.rst +++ b/docs/templates/testcase_description_v2_template.rst @@ -9,37 +9,56 @@ Yardstick Test Case Description TCXXX +-----------------------------------------------------------------------------+ |test case slogan e.g. Network Latency | -+==============+==============================================================+ +| | ++--------------+--------------------------------------------------------------+ |test case id | e.g. OPNFV_YARDSTICK_TC001_NW Latency | +| | | +--------------+--------------------------------------------------------------+ |metric | what will be measured, e.g. latency | +| | | +--------------+--------------------------------------------------------------+ |test purpose | describe what is the purpose of the test case | +| | | +--------------+--------------------------------------------------------------+ |configuration | what .yaml file to use, state SLA if applicable, state | | | test duration, list and describe the scenario options used in| | | this TC and also list the options using default values. | +| | | +--------------+--------------------------------------------------------------+ |test tool | e.g. ping | +| | | +--------------+--------------------------------------------------------------+ |references | e.g. RFCxxx, ETSI-NFVyyy | +| | | +--------------+--------------------------------------------------------------+ |applicability | describe variations of the test case which can be | | | performend, e.g. run the test for different packet sizes | +| | | +--------------+--------------------------------------------------------------+ |pre-test | describe configuration in the tool(s) used to perform | |conditions | the measurements (e.g. fio, pktgen), POD-specific | | | configuration required to enable running the test | -+--------------+------+----------------------------------+--------------------+ -|test sequence | step | description | result | -| +------+----------------------------------+--------------------+ -| | 1 | use this to describe tests that | what happens in | -| | | require several steps e.g. | this step | -| | | step 1 collect logs | e.g. logs collected| -| +------+----------------------------------+--------------------+ -| | 2 | remove interface | interface down | -| +------+----------------------------------+--------------------+ -| | N | what is done in step N | what happens | -+--------------+------+----------------------------------+--------------------+ +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | use this to describe tests that require sveveral steps e.g | +| | collect logs. | +| | | +| | Result: what happens in this step e.g. logs collected | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | remove interface | +| | | +| | Result: interface down. | +| | | ++--------------+--------------------------------------------------------------+ +|step N | what is done in step N | +| | | +| | Result: what happens | +| | | ++--------------+--------------------------------------------------------------+ |test verdict | expected behavior, or SLA, pass/fail criteria | +| | | +--------------+--------------------------------------------------------------+ diff --git a/docs/yardstick/01-introduction.rst b/docs/yardstick/01-introduction.rst new file mode 100644 index 000000000..6cca2875e --- /dev/null +++ b/docs/yardstick/01-introduction.rst @@ -0,0 +1,38 @@ +============ +Introduction +============ + +**Welcome to Yardstick's documentation !** + +.. _Yardstick: https://wiki.opnfv.org/yardstick + +Yardstick_ is an OPNFV Project. + +The project's goal is to verify infrastructure compliance, from the perspective +of a :term:`VNF`. + +The Project's scope is the development of a test framework, *Yardstick*, test +cases and test stimuli to enable :term:`NFVI` verification. +The Project also includes a sample :term:`VNF`, the :term:`VTC` and its +experimental framework, *ApexLake* ! + +The chapter :doc:`02-methodology` describes the methodology implemented by the +Yardstick Project for :term:`NFVI` verification. The chapter +:doc:`03-list-of-tcs` includes a list of available Yardstick test cases. + +Yardstick is used for verifying the OPNFV infrastructure and some of the OPNFV +features, listed in :doc:`03-list-of-tcs`. + +The *Yardstick* framework is deployed in several OPNFV community labs. It is +installer, infrastructure and application independent. + +.. _Pharos: https://wiki.opnfv.org/pharos + +.. seealso:: Pharos_ for information on OPNFV community labs. + +Contact Yardstick +================= + +Feedback? `Contact us`_ + +.. _Contact us: opnfv-users@lists.opnfv.org diff --git a/docs/yardstick/02-methodology.rst b/docs/yardstick/02-methodology.rst new file mode 100644 index 000000000..5097c566b --- /dev/null +++ b/docs/yardstick/02-methodology.rst @@ -0,0 +1,181 @@ +=========== +Methodology +=========== + +Abstract +======== + +This chapter describes the methodology implemented by the Yardstick project for +verifying the NFV Infrastructure from the perspective of a VNF. + +ETSI-NFV +======== + +.. _NFV-TST001: https://docbox.etsi.org/ISG/NFV/Open/Drafts/TST001_-_Pre-deployment_Validation/ + +The document ETSI GS NFV-TST001_, "Pre-deployment Testing; Report on Validation +of NFV Environments and Services", recommends methods for pre-deployment +testing of the functional components of an NFV environment. + +The Yardstick project implements the methodology described in chapter 6, "Pre- +deployment validation of NFV infrastructure". + +The methodology consists in decomposing the typical VNF work-load performance +metrics into a number of characteristics/performance vectors, which each can be +represented by distinct test-cases. + +The methodology includes five steps: + +* *Step1:* Define Infrastruture - the HW, SW and corresponding configuration + target for validation; the OPNFV infrastructure, in OPNFV community labs. + +* *Step2:* Identify VNF type - the application for which the infrastructure is + to be validated, and its requirements on the underlying infrastructure. + +* *Step3:* Select test cases - depending on the workload that represents the + application for which the infrastruture is to be validated, the relevant + test cases amongst the list of available Yardstick test cases. + +* *Step4:* Execute tests - define the duration and number of iterations for the + selected test cases, tests runs are automated via OPNFV Jenkins Jobs. + +* *Step5:* Collect results - using the common API for result collection. + +Metrics +======= + +The metrics, as defined by ETSI GS NFV-TST001, are shown in +:ref:`Table1 <table2_1>`, :ref:`Table2 <table2_2>` and +:ref:`Table3 <table2_3>`. + +In OPNFV Brahmaputra release, generic test cases covering aspects of the listed +metrics are available; further OPNFV releases will provide extended testing of +these metrics. +The view of available Yardstick test cases cross ETSI definitions in +:ref:`Table1 <table2_1>`, :ref:`Table2 <table2_2>` and :ref:`Table3 <table2_3>` +is shown in :ref:`Table4 <table2_4>`. +It shall be noticed that the Yardstick test cases are examples, the test +duration and number of iterations are configurable, as are the System Under +Test (SUT) and the attributes (or, in Yardstick nomemclature, the scenario +options). + +.. _table2_1: + +**Table 1 - Performance/Speed Metrics** + ++---------+-------------------------------------------------------------------+ +| Category| Performance/Speed | +| | | ++---------+-------------------------------------------------------------------+ +| Compute | * Latency for random memory access | +| | * Latency for cache read/write operations | +| | * Processing speed (instructions per second) | +| | * Throughput for random memory access (bytes per second) | +| | | ++---------+-------------------------------------------------------------------+ +| Network | * Throughput per NFVI node (frames/byte per second) | +| | * Throughput provided to a VM (frames/byte per second) | +| | * Latency per traffic flow | +| | * Latency between VMs | +| | * Latency between NFVI nodes | +| | * Packet delay variation (jitter) between VMs | +| | * Packet delay variation (jitter) between NFVI nodes | +| | | ++---------+-------------------------------------------------------------------+ +| Storage | * Sequential read/write IOPS | +| | * Random read/write IOPS | +| | * Latency for storage read/write operations | +| | * Throughput for storage read/write operations | +| | | ++---------+-------------------------------------------------------------------+ + +.. _table2_2: + +**Table 2 - Capacity/Scale Metrics** + ++---------+-------------------------------------------------------------------+ +| Category| Capacity/Scale | +| | | ++---------+-------------------------------------------------------------------+ +| Compute | * Number of cores and threads- Available memory size | +| | * Cache size | +| | * Processor utilization (max, average, standard deviation) | +| | * Memory utilization (max, average, standard deviation) | +| | * Cache utilization (max, average, standard deviation) | +| | | ++---------+-------------------------------------------------------------------+ +| Network | * Number of connections | +| | * Number of frames sent/received | +| | * Maximum throughput between VMs (frames/byte per second) | +| | * Maximum throughput between NFVI nodes (frames/byte per second) | +| | * Network utilization (max, average, standard deviation) | +| | * Number of traffic flows | +| | | ++---------+-------------------------------------------------------------------+ +| Storage | * Storage/Disk size | +| | * Capacity allocation (block-based, object-based) | +| | * Block size | +| | * Maximum sequential read/write IOPS | +| | * Maximum random read/write IOPS | +| | * Disk utilization (max, average, standard deviation) | +| | | ++---------+-------------------------------------------------------------------+ + +.. _table2_3: + +**Table 3 - Availability/Reliability Metrics** + ++---------+-------------------------------------------------------------------+ +| Category| Availability/Reliability | +| | | ++---------+-------------------------------------------------------------------+ +| Compute | * Processor availability (Error free processing time) | +| | * Memory availability (Error free memory time) | +| | * Processor mean-time-to-failure | +| | * Memory mean-time-to-failure | +| | * Number of processing faults per second | +| | | ++---------+-------------------------------------------------------------------+ +| Network | * NIC availability (Error free connection time) | +| | * Link availability (Error free transmission time) | +| | * NIC mean-time-to-failure | +| | * Network timeout duration due to link failure | +| | * Frame loss rate | +| | | ++---------+-------------------------------------------------------------------+ +| Storage | * Disk availability (Error free disk access time) | +| | * Disk mean-time-to-failure | +| | * Number of failed storage read/write operations per second | +| | | ++---------+-------------------------------------------------------------------+ + +.. _table2_4: + +**Table 4 - Yardstick Generic Test Cases** + ++---------+-------------------+----------------+------------------------------+ +| Category| Performance/Speed | Capacity/Scale | Availability/Reliability | +| | | | | ++---------+-------------------+----------------+------------------------------+ +| Compute | TC003 | TC003 | TC013 [1]_ | +| | TC004 | TC004 | TC015 [1]_ | +| | TC014 | TC010 | | +| | TC024 | TC012 | | +| | | | | ++---------+-------------------+----------------+------------------------------+ +| Network | TC002 | TC001 | TC016 [1]_ | +| | TC011 | TC008 | TC018 [1]_ | +| | | TC009 | | +| | | | | ++---------+-------------------+----------------+------------------------------+ +| Storage | TC005 | TC005 | TC017 [1]_ | +| | | | | ++---------+-------------------+----------------+------------------------------+ + +.. note:: The description in this OPNFV document is intended as a reference for + users to understand the scope of the Yardstick Project and the + deliverables of the Yardstick framework. For complete description of + the methodology, refer to the ETSI document. + +.. rubric:: Footnotes +.. [1] To be included in future deliveries. diff --git a/docs/yardstick/03-list-of-tcs.rst b/docs/yardstick/03-list-of-tcs.rst new file mode 100644 index 000000000..d2abf4e25 --- /dev/null +++ b/docs/yardstick/03-list-of-tcs.rst @@ -0,0 +1,42 @@ +==================== +Yardstick Test Cases +==================== + +Abstract +======== + +This chapter lists available Yardstick test cases. +Yardstick test cases are divided in two main categories: + +* *Generic NFVI Test Cases* - Test Cases developed to realize the methodology +described in :doc:`02-methodology` + +* *OPNFV Feature Test Cases* - Test Cases developed to verify one or more +aspect of a feature delivered by an OPNFV Project. + +Generic NFVI Test Case Descriptions +=================================== + +.. toctree:: + :maxdepth: 1 + + opnfv_yardstick_tc001.rst + opnfv_yardstick_tc002.rst + opnfv_yardstick_tc008.rst + opnfv_yardstick_tc009.rst + opnfv_yardstick_tc010.rst + opnfv_yardstick_tc012.rst + +OPNFV Feature Test Cases +======================== + + + +Templates +========= + +.. toctree:: + :maxdepth: 1 + + testcase_description_v2_template + Yardstick_task_templates diff --git a/docs/yardstick/Yardstick_task_templates.rst b/docs/yardstick/Yardstick_task_templates.rst new file mode 100755 index 000000000..d2c2b7ec9 --- /dev/null +++ b/docs/yardstick/Yardstick_task_templates.rst @@ -0,0 +1,155 @@ +Task Template Syntax +==================== + +Basic template syntax +--------------------- +A nice feature of the input task format used in Yardstick is that it supports +the template syntax based on Jinja2. +This turns out to be extremely useful when, say, you have a fixed structure of +your task but you want to parameterize this task in some way. +For example, imagine your input task file (task.yaml) runs a set of Ping +scenarios: + +:: + + # Sample benchmark task config file + # measure network latency using ping + schema: "yardstick:task:0.1" + + scenarios: + - + type: Ping + options: + packetsize: 200 + host: athena.demo + target: ares.demo + + runner: + type: Duration + duration: 60 + interval: 1 + + sla: + max_rtt: 10 + action: monitor + + context: + ... + +Let's say you want to run the same set of scenarios with the same runner/ +context/sla, but you want to try another packetsize to compare the performance. +The most elegant solution is then to turn the packetsize name into a template +variable: + +:: + + # Sample benchmark task config file + # measure network latency using ping + + schema: "yardstick:task:0.1" + scenarios: + - + type: Ping + options: + packetsize: {{packetsize}} + host: athena.demo + target: ares.demo + + runner: + type: Duration + duration: 60 + interval: 1 + + sla: + max_rtt: 10 + action: monitor + + context: + ... + +and then pass the argument value for {{packetsize}} when starting a task with +this configuration file. +Yardstick provides you with different ways to do that: + +1.Pass the argument values directly in the command-line interface (with either +a JSON or YAML dictionary): + +:: + + yardstick task start samples/ping-template.yaml + --task-args'{"packetsize":"200"}' + +2.Refer to a file that specifies the argument values (JSON/YAML): + +:: + + yardstick task start samples/ping-template.yaml --task-args-file args.yaml + +Using the default values +------------------------ +Note that the Jinja2 template syntax allows you to set the default values for +your parameters. +With default values set, your task file will work even if you don't +parameterize it explicitly while starting a task. +The default values should be set using the {% set ... %} clause (task.yaml). +For example: + +:: + + # Sample benchmark task config file + # measure network latency using ping + schema: "yardstick:task:0.1" + {% set packetsize = packetsize or "100" %} + scenarios: + - + type: Ping + options: + packetsize: {{packetsize}} + host: athena.demo + target: ares.demo + + runner: + type: Duration + duration: 60 + interval: 1 + ... + +If you don't pass the value for {{packetsize}} while starting a task, the +default one will be used. + +Advanced templates +------------------ + +Yardstick makes it possible to use all the power of Jinja2 template syntax, +including the mechanism of built-in functions. +As an example, let us make up a task file that will do a block storage +performance test. +The input task file (fio-template.yaml) below uses the Jinja2 for-endfor +construct to accomplish that: + +:: + + #Test block sizes of 4KB, 8KB, 64KB, 1MB + #Test 5 workloads: read, write, randwrite, randread, rw + schema: "yardstick:task:0.1" + + scenarios: + {% for bs in ['4k', '8k', '64k', '1024k' ] %} + {% for rw in ['read', 'write', 'randwrite', 'randread', 'rw' ] %} + - + type: Fio + options: + filename: /home/ec2-user/data.raw + bs: {{bs}} + rw: {{rw}} + ramp_time: 10 + host: fio.demo + runner: + type: Duration + duration: 60 + interval: 60 + + {% endfor %} + {% endfor %} + context + ... diff --git a/docs/yardstick/glossary.rst b/docs/yardstick/glossary.rst new file mode 100644 index 000000000..67e126f58 --- /dev/null +++ b/docs/yardstick/glossary.rst @@ -0,0 +1,15 @@ +================== +Yardstick Glossary +================== + +.. glossary:: + :sorted: + + VNF + Virtual Network Function + + NFVI + Network Function Virtualization Infrastructure + + VTC + Virtual Traffic Classifier diff --git a/docs/yardstick/index.rst b/docs/yardstick/index.rst index b14670bdd..ccea55ccb 100644 --- a/docs/yardstick/index.rst +++ b/docs/yardstick/index.rst @@ -1,21 +1,11 @@ -====================== -Yardstick Config Guide -====================== - -Test Case Descriptions -====================== - -.. toctree:: - :maxdepth: 1 - - opnfv_yardstick_tc001.rst - opnfv_yardstick_tc002.rst - -Templates -========= +================== +Yardstick Overview +================== .. toctree:: - :maxdepth: 1 + :maxdepth: 2 - ../templates/Yardstick_task_templates - ../templates/testcase_description_v2_template + 01-introduction + 02-methodology + 03-list-of-tcs + glossary diff --git a/docs/yardstick/opnfv_yardstick_tc001.rst b/docs/yardstick/opnfv_yardstick_tc001.rst index 16c9d2c60..810bad489 100644 --- a/docs/yardstick/opnfv_yardstick_tc001.rst +++ b/docs/yardstick/opnfv_yardstick_tc001.rst @@ -1,12 +1,18 @@ ************************************* Yardstick Test Case Description TC001 ************************************* + +.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt + +-----------------------------------------------------------------------------+ |Network Performance | -+==============+==============================================================+ +| | ++--------------+--------------------------------------------------------------+ |test case id | OPNFV_YARDSTICK_TC001_NW PERF | +| | | +--------------+--------------------------------------------------------------+ |metric | Number of flows and throughput | +| | | +--------------+--------------------------------------------------------------+ |test purpose | To evaluate the IaaS network performance with regards to | | | flows and throughput, such as if and how different amounts | @@ -19,6 +25,7 @@ Yardstick Test Case Description TC001 | | graphs ans similar shall be stored for comparison reasons and| | | product evolution understanding between different OPNFV | | | versions and/or configurations. | +| | | +--------------+--------------------------------------------------------------+ |configuration | file: opnfv_yardstick_tc001.yaml | | | | @@ -28,6 +35,7 @@ Yardstick Test Case Description TC001 | | twice. The client and server are distributed on different HW.| | | For SLA max_ppm is set to 1000. The amount of configured | | | ports map to between 110 up to 1001000 flows, respectively. | +| | | +--------------+--------------------------------------------------------------+ |test tool | pktgen | | | | @@ -36,30 +44,36 @@ Yardstick Test Case Description TC001 | | image. | | | As an example see the /yardstick/tools/ directory for how | | | to generate a Linux image with pktgen included.) | +| | | +--------------+--------------------------------------------------------------+ -|references |https://www.kernel.org/doc/Documentation/networking/pktgen.txt| +|references | pktgen_ | +| | | +| | ETSI-NFV-TST001 | | | | -| |ETSI-NFV-TST001 | +--------------+--------------------------------------------------------------+ |applicability | Test can be configured with different packet sizes, amount | | | of flows and test duration. Default values exist. | | | | -| |SLA (optional): | -| | max_ppm: The number of packets per million packets sent | -| | that are acceptable to lose, i.e. not received. | +| | SLA (optional): max_ppm: The number of packets per million | +| | packets sent that are acceptable to loose, not received. | +| | | +--------------+--------------------------------------------------------------+ |pre-test | The test case image needs to be installed into Glance | |conditions | with pktgen included in it. | | | | | | No POD specific requirements have been identified. | -+--------------+------+----------------------------------+--------------------+ -|test sequence | step | description | result | -| +------+----------------------------------+--------------------+ -| | 1 | The hosts are installed, as | Logs are stored | -| | | server and client. pktgen is | | -| | | invoked and logs are produced | | -| | | and stored. | | -+--------------+------+----------------------------------+--------------------+ +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. pktgen is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ |test verdict | Fails only if SLA is not passed, or if there is a test case | | | execution problem. | +| | | +--------------+--------------------------------------------------------------+ diff --git a/docs/yardstick/opnfv_yardstick_tc002.rst b/docs/yardstick/opnfv_yardstick_tc002.rst index bc795bf38..56350f5bb 100644 --- a/docs/yardstick/opnfv_yardstick_tc002.rst +++ b/docs/yardstick/opnfv_yardstick_tc002.rst @@ -2,12 +2,17 @@ Yardstick Test Case Description TC002 ************************************* +.. _cirros-image: https://download.cirros-cloud.net + +-----------------------------------------------------------------------------+ |Network Latency | -+==============+==============================================================+ +| | ++--------------+--------------------------------------------------------------+ |test case id | OPNFV_YARDSTICK_TC002_NW LATENCY | +| | | +--------------+--------------------------------------------------------------+ |metric | RTT, Round Trip Time | +| | | +--------------+--------------------------------------------------------------+ |test purpose | To do a basic verification that network latency is within | | | acceptable boundaries when packets travel between hosts | @@ -16,11 +21,13 @@ Yardstick Test Case Description TC002 | | graphs and similar shall be stored for comparison reasons and| | | product evolution understanding between different OPNFV | | | versions and/or configurations. | +| | | +--------------+--------------------------------------------------------------+ |configuration | file: opnfv_yardstick_tc002.yaml | | | | | | Packet size 100 bytes. Total test duration 600 seconds. | | | One ping each 10 seconds. SLA RTT is set to maximum 10 ms. | +| | | +--------------+--------------------------------------------------------------+ |test tool | ping | | | | @@ -28,11 +35,13 @@ Yardstick Test Case Description TC002 | | doesn't need to be installed. It is also part of the | | | Yardstick Docker image. | | | (For example also a Cirros image can be downloaded from | -| | https://download.cirros-cloud.net, it includes ping) | +| | cirros-image_, it includes ping) | +| | | +--------------+--------------------------------------------------------------+ |references | Ping man page | | | | | | ETSI-NFV-TST001 | +| | | +--------------+--------------------------------------------------------------+ |applicability | Test case can be configured with different packet sizes, | | | burst sizes, ping intervals and test duration. | @@ -46,20 +55,24 @@ Yardstick Test Case Description TC002 | | than this. Some may suffer bad also close to this RTT, while | | | others may not suffer at all. It is a compromise that may | | | have to be tuned for different configuration purposes. | +| | | +--------------+--------------------------------------------------------------+ |pre-test | The test case image needs to be installed into Glance | |conditions | with ping included in it. | | | | | | No POD specific requirements have been identified. | -+--------------+------+----------------------------------+--------------------+ -|test sequence | step | description | result | -| +------+----------------------------------+--------------------+ -| | 1 | The hosts are installed, as | Logs are stored | -| | | server and client. Ping is | | -| | | invoked and logs are produced | | -| | | and stored. | | -+--------------+------+----------------------------------+--------------------+ +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. Ping is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ |test verdict | Test should not PASS if any RTT is above the optional SLA | | | value, or if there is a test case execution problem. | +| | | +--------------+--------------------------------------------------------------+ - diff --git a/docs/yardstick/opnfv_yardstick_tc008.rst b/docs/yardstick/opnfv_yardstick_tc008.rst index f4971fbad..e176e633a 100644 --- a/docs/yardstick/opnfv_yardstick_tc008.rst +++ b/docs/yardstick/opnfv_yardstick_tc008.rst @@ -1,12 +1,18 @@ ************************************* Yardstick Test Case Description TC008 ************************************* + +.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt + +-----------------------------------------------------------------------------+ -|Network Performance | -+==============+==============================================================+ +|Packet Loss Extended Test | +| | ++--------------+--------------------------------------------------------------+ |test case id | OPNFV_YARDSTICK_TC008_NW PERF, Packet loss Extended Test | +| | | +--------------+--------------------------------------------------------------+ |metric | Number of flows, packet size and throughput | +| | | +--------------+--------------------------------------------------------------+ |test purpose | To evaluate the IaaS network performance with regards to | | | flows and throughput, such as if and how different amounts | @@ -20,6 +26,7 @@ Yardstick Test Case Description TC008 | | graphs ans similar shall be stored for comparison reasons and| | | product evolution understanding between different OPNFV | | | versions and/or configurations. | +| | | +--------------+--------------------------------------------------------------+ |configuration | file: opnfv_yardstick_tc008.yaml | | | | @@ -34,6 +41,7 @@ Yardstick Test Case Description TC008 | | The client and server are distributed on different HW. | | | | | | For SLA max_ppm is set to 1000. | +| | | +--------------+--------------------------------------------------------------+ |test tool | pktgen | | | | @@ -42,30 +50,36 @@ Yardstick Test Case Description TC008 | | image. | | | As an example see the /yardstick/tools/ directory for how | | | to generate a Linux image with pktgen included.) | +| | | +--------------+--------------------------------------------------------------+ -|references |https://www.kernel.org/doc/Documentation/networking/pktgen.txt| +|references | pktgen_ | +| | | +| | ETSI-NFV-TST001 | | | | -| |ETSI-NFV-TST001 | +--------------+--------------------------------------------------------------+ |applicability | Test can be configured with different packet sizes, amount | | | of flows and test duration. Default values exist. | | | | -| |SLA (optional): | -| | max_ppm: The number of packets per million packets sent | -| | that are acceptable to lose, i.e. not received. | +| | SLA (optional): max_ppm: The number of packets per million | +| | packets sent that are acceptable to loose, not received. | +| | | +--------------+--------------------------------------------------------------+ |pre-test | The test case image needs to be installed into Glance | |conditions | with pktgen included in it. | | | | | | No POD specific requirements have been identified. | -+--------------+------+----------------------------------+--------------------+ -|test sequence | step | description | result | -| +------+----------------------------------+--------------------+ -| | 1 | The hosts are installed, as | Logs are stored | -| | | server and client. pktgen is | | -| | | invoked and logs are produced | | -| | | and stored. | | -+--------------+------+----------------------------------+--------------------+ +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. pktgen is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ |test verdict | Fails only if SLA is not passed, or if there is a test case | | | execution problem. | +| | | +--------------+--------------------------------------------------------------+ diff --git a/docs/yardstick/opnfv_yardstick_tc009.rst b/docs/yardstick/opnfv_yardstick_tc009.rst index 07d7fbfea..e4002a884 100644 --- a/docs/yardstick/opnfv_yardstick_tc009.rst +++ b/docs/yardstick/opnfv_yardstick_tc009.rst @@ -1,12 +1,18 @@ ************************************* Yardstick Test Case Description TC009 ************************************* + +.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt + +-----------------------------------------------------------------------------+ -|Network Performance | -+==============+==============================================================+ +|Packet Loss | +| | ++--------------+--------------------------------------------------------------+ |test case id | OPNFV_YARDSTICK_TC009_NW PERF, Packet loss | +| | | +--------------+--------------------------------------------------------------+ -|metric | Number of flows and throughput | +|metric | Number of flows, packets lost and throughput | +| | | +--------------+--------------------------------------------------------------+ |test purpose | To evaluate the IaaS network performance with regards to | | | flows and throughput, such as if and how different amounts | @@ -20,6 +26,7 @@ Yardstick Test Case Description TC009 | | graphs ans similar shall be stored for comparison reasons and| | | product evolution understanding between different OPNFV | | | versions and/or configurations. | +| | | +--------------+--------------------------------------------------------------+ |configuration | file: opnfv_yardstick_tc009.yaml | | | | @@ -33,6 +40,7 @@ Yardstick Test Case Description TC009 | | The client and server are distributed on different HW. | | | | | | For SLA max_ppm is set to 1000. | +| | | +--------------+--------------------------------------------------------------+ |test tool | pktgen | | | | @@ -41,30 +49,36 @@ Yardstick Test Case Description TC009 | | image. | | | As an example see the /yardstick/tools/ directory for how | | | to generate a Linux image with pktgen included.) | +| | | +--------------+--------------------------------------------------------------+ -|references |https://www.kernel.org/doc/Documentation/networking/pktgen.txt| +|references | pktgen_ | +| | | +| | ETSI-NFV-TST001 | | | | -| |ETSI-NFV-TST001 | +--------------+--------------------------------------------------------------+ |applicability | Test can be configured with different packet sizes, amount | | | of flows and test duration. Default values exist. | | | | -| |SLA (optional): | -| | max_ppm: The number of packets per million packets sent | -| | that are acceptable to lose, i.e. not received. | +| | SLA (optional): max_ppm: The number of packets per million | +| | packets sent that are acceptable to loose, not received. | +| | | +--------------+--------------------------------------------------------------+ |pre-test | The test case image needs to be installed into Glance | |conditions | with pktgen included in it. | | | | | | No POD specific requirements have been identified. | -+--------------+------+----------------------------------+--------------------+ -|test sequence | step | description | result | -| +------+----------------------------------+--------------------+ -| | 1 | The hosts are installed, as | Logs are stored | -| | | server and client. pktgen is | | -| | | invoked and logs are produced | | -| | | and stored. | | -+--------------+------+----------------------------------+--------------------+ +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. pktgen is | +| | invoked and logs are produced and stored. | +| | | +| | Result: logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ |test verdict | Fails only if SLA is not passed, or if there is a test case | | | execution problem. | +| | | +--------------+--------------------------------------------------------------+ diff --git a/docs/yardstick/opnfv_yardstick_tc010.rst b/docs/yardstick/opnfv_yardstick_tc010.rst index c2f8eb4e2..ebb74ea30 100644 --- a/docs/yardstick/opnfv_yardstick_tc010.rst +++ b/docs/yardstick/opnfv_yardstick_tc010.rst @@ -1,17 +1,23 @@ - ************************************* Yardstick Test Case Description TC010 ************************************* + +.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/lat_mem_rd.8.html + +-----------------------------------------------------------------------------+ |Memory Latency | -+==============+==============================================================+ +| | ++--------------+--------------------------------------------------------------+ |test case id | OPNFV_YARDSTICK_TC010_Memory Latency | +| | | +--------------+--------------------------------------------------------------+ |metric | Latency in nanoseconds | +| | | +--------------+--------------------------------------------------------------+ |test purpose | Measure the memory read latency for varying memory sizes and | | | strides. Whole memory hierarchy is measured including all | | | levels of cache. | +| | | +--------------+--------------------------------------------------------------+ |configuration | File: opnfv_yardstick_tc010.yaml | | | | @@ -21,6 +27,7 @@ Yardstick Test Case Description TC010 | | * Iterations: 10 - test is run 10 times iteratively. | | | * Interval: 1 - there is 1 second delay between each | | | iteration. | +| | | +--------------+--------------------------------------------------------------+ |test tool | Lmbench | | | | @@ -28,38 +35,43 @@ Yardstick Test Case Description TC010 | | test uses lat_mem_rd tool from that suite. | | | Lmbench is not always part of a Linux distribution, hence it | | | needs to be installed in the test image | -| | (See :ref:`guest-image` for how to generate a Linux image | -| | for Glance with Lmbench included). | +| | | +--------------+--------------------------------------------------------------+ -|references |* http://manpages.ubuntu.com/manpages/trusty/lat_mem_rd.8.html| +|references | man-pages_ | +| | | +| | McVoy, Larry W.,and Carl Staelin. "lmbench: Portable Tools | +| | for Performance Analysis." USENIX annual technical | +| | conference 1996. | | | | -| |* McVoy, Larry W., and Carl Staelin. "lmbench: Portable Tools | -| | for Performance Analysis." *USENIX annual technical | -| | conference*. 1996. | +--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different | -| | * strides; | -| | * stop_size; | -| | * iterations and intervals; | +|applicability | Test can be configured with different: | +| | | +| | * strides; | +| | * stop_size; | +| | * iterations and intervals. | | | | | | There are default values for each above-mentioned option. | | | | -| | * SLA (optional). | -| | max_latency: The maximum memory latency that is | -| | accepted. | +| | SLA (optional) : max_latency: The maximum memory latency | +| | that is accepted. | +| | | +--------------+--------------------------------------------------------------+ |pre-test | The test case image needs to be installed into Glance | |conditions | with Lmbench included in the image. | | | | | | No POD specific requirements have been identified. | -+--------------+------+----------------------------------+--------------------+ -|test sequence | step | description | result | -| +------+----------------------------------+--------------------+ -| | 1 | The host is installed as client. | Logs are stored | -| | | Lmbench's lat_mem_rd tool is | | -| | | invoked and logs are produced and| | -| | | stored. | | -+--------------+------+----------------------------------+--------------------+ +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The host is installed as client. Lmbench's lat_mem_rd tool | +| | is invoked and logs are produced and stored. | +| | | +| | Result: logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ |test verdict | Test fails if the measured memory latency is above the SLA | | | value or if there is a test case execution problem. | +| | | +--------------+--------------------------------------------------------------+ diff --git a/docs/yardstick/opnfv_yardstick_tc012.rst b/docs/yardstick/opnfv_yardstick_tc012.rst index b5768c0c5..e7889c14e 100644 --- a/docs/yardstick/opnfv_yardstick_tc012.rst +++ b/docs/yardstick/opnfv_yardstick_tc012.rst @@ -2,21 +2,26 @@ Yardstick Test Case Description TC012 ************************************* +.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/bw_mem.8.html + +-----------------------------------------------------------------------------+ |Memory Bandwidth | -+==============+==============================================================+ +| | ++--------------+--------------------------------------------------------------+ |test case id | OPNFV_YARDSTICK_TC012_Memory Bandwidth | +| | | +--------------+--------------------------------------------------------------+ |metric | Megabyte per second (MBps) | +| | | +--------------+--------------------------------------------------------------+ |test purpose | Measure the rate at which data can be read from and written | | | to the memory (this includes all levels of memory). | +| | | +--------------+--------------------------------------------------------------+ |configuration | File: opnfv_yardstick_tc012.yaml | | | | -| | * SLA (optional): 15000 (MBps) | -| | min_bw: The minimum amount of memory bandwidth that is | -| | accepted. | +| | * SLA (optional): 15000 (MBps) min_bw: The minimum amount of | +| | memory bandwidth that is accepted. | | | * Size: 10 240 kB - test allocates twice that size (20 480kB)| | | zeros it and then measures the time it takes to copy from | | | one side to another. | @@ -27,42 +32,50 @@ Yardstick Test Case Description TC012 | | * Iterations: 10 - test is run 10 times iteratively. | | | * Interval: 1 - there is 1 second delay between each | | | iteration. | +| | | +--------------+--------------------------------------------------------------+ |test tool | Lmbench | | | | | | Lmbench is a suite of operating system microbenchmarks. This | | | test uses bw_mem tool from that suite. | | | Lmbench is not always part of a Linux distribution, hence it | -| | needs to be installed in the test image | -| | (See :ref:`guest-image` for how to generate a Linux image | -| | for Glance with Lmbench included). | +| | needs to be installed in the test image. | +| | | +--------------+--------------------------------------------------------------+ -|references | * http://manpages.ubuntu.com/manpages/trusty/bw_mem.8.html | +|references | man-pages_ | +| | | +| | McVoy, Larry W., and Carl Staelin. "lmbench: Portable Tools | +| | for Performance Analysis." USENIX annual technical | +| | conference. 1996. | | | | -| | * McVoy, Larry W., and Carl Staelin. "lmbench: Portable Tools| -| | for Performance Analysis." | -| | * USENIX annual technical conference. 1996. | +--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different | -| | * memory sizes; | -| | * memory operations (such as rd, wr, rdwr, cp, frd, fwr, | -| | fcp, bzero, bcopy); | -| | * number of warmup iterations; | -| | * iterations and intervals. | +|applicability | Test can be configured with different: | +| | | +| | * memory sizes; | +| | * memory operations (such as rd, wr, rdwr, cp, frd, fwr, | +| | fcp, bzero, bcopy); | +| | * number of warmup iterations; | +| | * iterations and intervals. | | | | | | There are default values for each above-mentioned option. | +| | | +--------------+--------------------------------------------------------------+ |pre-test | The test case image needs to be installed into Glance | |conditions | with Lmbench included in the image. | | | | | | No POD specific requirements have been identified. | -+--------------+------+----------------------------------+--------------------+ -|test sequence | step | description | result | -| +------+----------------------------------+--------------------+ -| | 1 | The host is installed as client. | Logs are stored | -| | | Lmbench's bw_mem tool is invoked | | -| | | and logs are produced and stored.| | -+--------------+------+----------------------------------+--------------------+ +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The host is installed as client. Lmbench's bw_mem tool is | +| | invoked and logs are produced and stored. | +| | | +| | Result: logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ |test verdict | Test fails if the measured memory bandwidth is below the SLA | | | value or if there is a test case execution problem. | +| | | +--------------+--------------------------------------------------------------+ diff --git a/docs/yardstick/testcase_description_v2_template.rst b/docs/yardstick/testcase_description_v2_template.rst new file mode 100644 index 000000000..1b8754b05 --- /dev/null +++ b/docs/yardstick/testcase_description_v2_template.rst @@ -0,0 +1,64 @@ +.. Template to be used for test case descriptions in Yardstick Project. + Write one .rst per test case. + Upload the .rst for the test case in /docs/source/yardstick directory. + Review in Gerrit. + +************************************* +Yardstick Test Case Description TCXXX +************************************* + ++-----------------------------------------------------------------------------+ +|test case slogan e.g. Network Latency | +| | ++--------------+--------------------------------------------------------------+ +|test case id | e.g. OPNFV_YARDSTICK_TC001_NW Latency | +| | | ++--------------+--------------------------------------------------------------+ +|metric | what will be measured, e.g. latency | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | describe what is the purpose of the test case | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | what .yaml file to use, state SLA if applicable, state | +| | test duration, list and describe the scenario options used in| +| | this TC and also list the options using default values. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | e.g. ping | +| | | ++--------------+--------------------------------------------------------------+ +|references | e.g. RFCxxx, ETSI-NFVyyy | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | describe variations of the test case which can be | +| | performend, e.g. run the test for different packet sizes | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | describe configuration in the tool(s) used to perform | +|conditions | the measurements (e.g. fio, pktgen), POD-specific | +| | configuration required to enable running the test | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | use this to describe tests that require sveveral steps e.g | +| | collect logs. | +| | | +| | Result: what happens in this step e.g. logs collected | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | remove interface | +| | | +| | Result: interface down. | +| | | ++--------------+--------------------------------------------------------------+ +|step N | what is done in step N | +| | | +| | Result: what happens | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | expected behavior, or SLA, pass/fail criteria | +| | | ++--------------+--------------------------------------------------------------+ |