diff options
| author |   Ana C <ana.cunha@ericsson.com> | 2016-02-10 19:07:13 +0100 |
|---|---|---|
| committer | Ana C <ana.cunha@ericsson.com> | 2016-02-11 09:39:59 +0100 |
| commit | 702eebbf78b29fb9046436dd71575b4f210f4731 (patch) | |
| tree | e7ef6278850111d67e414a44561066ec00061331 /docs/configguide/yardstick_testcases | |
| parent | fd3dd37a734a741a823962dbc646480ece70b3bb (diff) | |
Add license info and update structure
This change adds license to all .rst files under userguide.
It also combines all configguide files into user guide.
New reference.rst with list of links.
Updated glossary, removed separate directories for apexlake and yardstick
framework.
Change-Id: I6532ed073905b0fa85a17e759ea7dc3c24acb91f
Signed-off-by: Ana C <ana.cunha@ericsson.com>
Diffstat (limited to 'docs/configguide/yardstick_testcases')
29 files changed, 0 insertions, 2681 deletions
diff --git a/docs/configguide/yardstick_testcases/01-introduction.rst b/docs/configguide/yardstick_testcases/01-introduction.rst deleted file mode 100644 index 6cca2875e..000000000 --- a/docs/configguide/yardstick_testcases/01-introduction.rst +++ /dev/null @@ -1,38 +0,0 @@ -============ -Introduction -============ - -**Welcome to Yardstick's documentation !** - -.. _Yardstick: https://wiki.opnfv.org/yardstick - -Yardstick_ is an OPNFV Project. - -The project's goal is to verify infrastructure compliance, from the perspective -of a :term:`VNF`. - -The Project's scope is the development of a test framework, *Yardstick*, test -cases and test stimuli to enable :term:`NFVI` verification. -The Project also includes a sample :term:`VNF`, the :term:`VTC` and its -experimental framework, *ApexLake* ! - -The chapter :doc:`02-methodology` describes the methodology implemented by the -Yardstick Project for :term:`NFVI` verification. The chapter -:doc:`03-list-of-tcs` includes a list of available Yardstick test cases. - -Yardstick is used for verifying the OPNFV infrastructure and some of the OPNFV -features, listed in :doc:`03-list-of-tcs`. - -The *Yardstick* framework is deployed in several OPNFV community labs. It is -installer, infrastructure and application independent. - -.. _Pharos: https://wiki.opnfv.org/pharos - -.. seealso:: Pharos_ for information on OPNFV community labs. - -Contact Yardstick -================= - -Feedback? `Contact us`_ - -.. _Contact us: opnfv-users@lists.opnfv.org diff --git a/docs/configguide/yardstick_testcases/02-methodology.rst b/docs/configguide/yardstick_testcases/02-methodology.rst deleted file mode 100644 index 5097c566b..000000000 --- a/docs/configguide/yardstick_testcases/02-methodology.rst +++ /dev/null @@ -1,181 +0,0 @@ -=========== -Methodology -=========== - -Abstract -======== - -This chapter describes the methodology implemented by the Yardstick project for -verifying the NFV Infrastructure from the perspective of a VNF. - -ETSI-NFV -======== - -.. _NFV-TST001: https://docbox.etsi.org/ISG/NFV/Open/Drafts/TST001_-_Pre-deployment_Validation/ - -The document ETSI GS NFV-TST001_, "Pre-deployment Testing; Report on Validation -of NFV Environments and Services", recommends methods for pre-deployment -testing of the functional components of an NFV environment. - -The Yardstick project implements the methodology described in chapter 6, "Pre- -deployment validation of NFV infrastructure". - -The methodology consists in decomposing the typical VNF work-load performance -metrics into a number of characteristics/performance vectors, which each can be -represented by distinct test-cases. - -The methodology includes five steps: - -* *Step1:* Define Infrastruture - the HW, SW and corresponding configuration - target for validation; the OPNFV infrastructure, in OPNFV community labs. - -* *Step2:* Identify VNF type - the application for which the infrastructure is - to be validated, and its requirements on the underlying infrastructure. - -* *Step3:* Select test cases - depending on the workload that represents the - application for which the infrastruture is to be validated, the relevant - test cases amongst the list of available Yardstick test cases. - -* *Step4:* Execute tests - define the duration and number of iterations for the - selected test cases, tests runs are automated via OPNFV Jenkins Jobs. - -* *Step5:* Collect results - using the common API for result collection. - -Metrics -======= - -The metrics, as defined by ETSI GS NFV-TST001, are shown in -:ref:`Table1 <table2_1>`, :ref:`Table2 <table2_2>` and -:ref:`Table3 <table2_3>`. - -In OPNFV Brahmaputra release, generic test cases covering aspects of the listed -metrics are available; further OPNFV releases will provide extended testing of -these metrics. -The view of available Yardstick test cases cross ETSI definitions in -:ref:`Table1 <table2_1>`, :ref:`Table2 <table2_2>` and :ref:`Table3 <table2_3>` -is shown in :ref:`Table4 <table2_4>`. -It shall be noticed that the Yardstick test cases are examples, the test -duration and number of iterations are configurable, as are the System Under -Test (SUT) and the attributes (or, in Yardstick nomemclature, the scenario -options). - -.. _table2_1: - -**Table 1 - Performance/Speed Metrics** - -+---------+-------------------------------------------------------------------+ -| Category| Performance/Speed | -| | | -+---------+-------------------------------------------------------------------+ -| Compute | * Latency for random memory access | -| | * Latency for cache read/write operations | -| | * Processing speed (instructions per second) | -| | * Throughput for random memory access (bytes per second) | -| | | -+---------+-------------------------------------------------------------------+ -| Network | * Throughput per NFVI node (frames/byte per second) | -| | * Throughput provided to a VM (frames/byte per second) | -| | * Latency per traffic flow | -| | * Latency between VMs | -| | * Latency between NFVI nodes | -| | * Packet delay variation (jitter) between VMs | -| | * Packet delay variation (jitter) between NFVI nodes | -| | | -+---------+-------------------------------------------------------------------+ -| Storage | * Sequential read/write IOPS | -| | * Random read/write IOPS | -| | * Latency for storage read/write operations | -| | * Throughput for storage read/write operations | -| | | -+---------+-------------------------------------------------------------------+ - -.. _table2_2: - -**Table 2 - Capacity/Scale Metrics** - -+---------+-------------------------------------------------------------------+ -| Category| Capacity/Scale | -| | | -+---------+-------------------------------------------------------------------+ -| Compute | * Number of cores and threads- Available memory size | -| | * Cache size | -| | * Processor utilization (max, average, standard deviation) | -| | * Memory utilization (max, average, standard deviation) | -| | * Cache utilization (max, average, standard deviation) | -| | | -+---------+-------------------------------------------------------------------+ -| Network | * Number of connections | -| | * Number of frames sent/received | -| | * Maximum throughput between VMs (frames/byte per second) | -| | * Maximum throughput between NFVI nodes (frames/byte per second) | -| | * Network utilization (max, average, standard deviation) | -| | * Number of traffic flows | -| | | -+---------+-------------------------------------------------------------------+ -| Storage | * Storage/Disk size | -| | * Capacity allocation (block-based, object-based) | -| | * Block size | -| | * Maximum sequential read/write IOPS | -| | * Maximum random read/write IOPS | -| | * Disk utilization (max, average, standard deviation) | -| | | -+---------+-------------------------------------------------------------------+ - -.. _table2_3: - -**Table 3 - Availability/Reliability Metrics** - -+---------+-------------------------------------------------------------------+ -| Category| Availability/Reliability | -| | | -+---------+-------------------------------------------------------------------+ -| Compute | * Processor availability (Error free processing time) | -| | * Memory availability (Error free memory time) | -| | * Processor mean-time-to-failure | -| | * Memory mean-time-to-failure | -| | * Number of processing faults per second | -| | | -+---------+-------------------------------------------------------------------+ -| Network | * NIC availability (Error free connection time) | -| | * Link availability (Error free transmission time) | -| | * NIC mean-time-to-failure | -| | * Network timeout duration due to link failure | -| | * Frame loss rate | -| | | -+---------+-------------------------------------------------------------------+ -| Storage | * Disk availability (Error free disk access time) | -| | * Disk mean-time-to-failure | -| | * Number of failed storage read/write operations per second | -| | | -+---------+-------------------------------------------------------------------+ - -.. _table2_4: - -**Table 4 - Yardstick Generic Test Cases** - -+---------+-------------------+----------------+------------------------------+ -| Category| Performance/Speed | Capacity/Scale | Availability/Reliability | -| | | | | -+---------+-------------------+----------------+------------------------------+ -| Compute | TC003 | TC003 | TC013 [1]_ | -| | TC004 | TC004 | TC015 [1]_ | -| | TC014 | TC010 | | -| | TC024 | TC012 | | -| | | | | -+---------+-------------------+----------------+------------------------------+ -| Network | TC002 | TC001 | TC016 [1]_ | -| | TC011 | TC008 | TC018 [1]_ | -| | | TC009 | | -| | | | | -+---------+-------------------+----------------+------------------------------+ -| Storage | TC005 | TC005 | TC017 [1]_ | -| | | | | -+---------+-------------------+----------------+------------------------------+ - -.. note:: The description in this OPNFV document is intended as a reference for - users to understand the scope of the Yardstick Project and the - deliverables of the Yardstick framework. For complete description of - the methodology, refer to the ETSI document. - -.. rubric:: Footnotes -.. [1] To be included in future deliveries. diff --git a/docs/configguide/yardstick_testcases/03-list-of-tcs.rst b/docs/configguide/yardstick_testcases/03-list-of-tcs.rst deleted file mode 100644 index bb3fbbac5..000000000 --- a/docs/configguide/yardstick_testcases/03-list-of-tcs.rst +++ /dev/null @@ -1,91 +0,0 @@ -==================== -Yardstick Test Cases -==================== - -Abstract -======== - -This chapter lists available Yardstick test cases. -Yardstick test cases are divided in two main categories: - -* *Generic NFVI Test Cases* - Test Cases developed to realize the methodology -described in :doc:`02-methodology` - -* *OPNFV Feature Test Cases* - Test Cases developed to verify one or more -aspect of a feature delivered by an OPNFV Project, including the test cases -developed for the :term:`VTC`. - -Generic NFVI Test Case Descriptions -=================================== - -.. toctree:: - :maxdepth: 1 - - opnfv_yardstick_tc001.rst - opnfv_yardstick_tc002.rst - opnfv_yardstick_tc005.rst - opnfv_yardstick_tc008.rst - opnfv_yardstick_tc009.rst - opnfv_yardstick_tc010.rst - opnfv_yardstick_tc011.rst - opnfv_yardstick_tc012.rst - opnfv_yardstick_tc014.rst - opnfv_yardstick_tc024.rst - opnfv_yardstick_tc037.rst - opnfv_yardstick_tc038.rst - -OPNFV Feature Test Cases -======================== - -H A ---- - -.. toctree:: - :maxdepth: 1 - - opnfv_yardstick_tc019.rst - opnfv_yardstick_tc025.rst - -IPv6 ----- - -.. toctree:: - :maxdepth: 1 - - opnfv_yardstick_tc027.rst - -KVM ---- - -.. toctree:: - :maxdepth: 1 - - opnfv_yardstick_tc028.rst - -Parser ------- - -.. toctree:: - :maxdepth: 1 - - opnfv_yardstick_tc040.rst - -virtual Traffic Classifier --------------------------- - -.. toctree:: - :maxdepth: 1 - - opnfv_yardstick_tc006.rst - opnfv_yardstick_tc007.rst - opnfv_yardstick_tc020.rst - opnfv_yardstick_tc021.rst - -Templates -========= - -.. toctree:: - :maxdepth: 1 - - testcase_description_v2_template - Yardstick_task_templates diff --git a/docs/configguide/yardstick_testcases/04-vtc-overview.rst b/docs/configguide/yardstick_testcases/04-vtc-overview.rst deleted file mode 100644 index 95159a9bc..000000000 --- a/docs/configguide/yardstick_testcases/04-vtc-overview.rst +++ /dev/null @@ -1,114 +0,0 @@ -========================== -Virtual Traffic Classifier -========================== - -Abstract -======== - -.. _TNOVA: http://www.t-nova.eu/ -.. _TNOVAresults: http://www.t-nova.eu/results/ -.. _Yardstick: https://wiki.opnfv.org/yardstick - -This chapter provides an overview of the virtual Traffic Classifier, a -contribution to OPNFV Yardstick_ from the EU Project TNOVA_. -Additional documentation is available in TNOVAresults_. - -Overview -======== - -The virtual Traffic Classifier :term:`VNF`, the :term:`VTC`, comprises of a -:term:`VNFC`. The :term:`VNFC` contains both the Traffic Inspection module, and -the Traffic forwarding module, needed to run the VNF. The exploitation of -:term:`DPI` methods for traffic classification is built around two basic -assumptions: - -* third parties unaffiliated with either source or recipient are able to -inspect each IP packet’s payload - -* the classifier knows the relevant syntax of each application’s packet -payloads (protocol signatures, data patterns, etc.). - -The proposed :term:`DPI` based approach will only use an indicative, small -number of the initial packets from each flow in order to identify the content -and not inspect each packet. - -In this respect it follows the :term:`PBFS`. This method uses a table to track -each session based on the 5-tuples (src address, dest address, src port,dest -port, transport protocol) that is maintained for each flow. - -Concepts -======== - -* *Traffic Inspection*: The process of packet analysis and application -identification of network traffic that passes through the :term:`VTC`. - -* *Traffic Forwarding*: The process of packet forwarding from an incoming -network interface to a pre-defined outgoing network interface. - -* *Traffic Rule Application*: The process of packet tagging, based on a -predefined set of rules. Packet tagging may include e.g. :term:`ToS` field -modification. - -Architecture -============ - -The Traffic Inspection module is the most computationally intensive component -of the :term:`VNF`. It implements filtering and packet matching algorithms in -order to support the enhanced traffic forwarding capability of the :term:`VNF`. -The component supports a flow table (exploiting hashing algorithms for fast -indexing of flows) and an inspection engine for traffic classification. - -The implementation used for these experiments exploits the nDPI library. -The packet capturing mechanism is implemented using libpcap. When the -:term:`DPI` engine identifies a new flow, the flow register is updated with the -appropriate information and transmitted across the Traffic Forwarding module, -which then applies any required policy updates. - -The Traffic Forwarding moudle is responsible for routing and packet forwarding. -It accepts incoming network traffic, consults the flow table for classification -information for each incoming flow and then applies pre-defined policies -marking e.g. :term:`ToS`/:term:`DSCP` multimedia traffic for :term:`QoS` -enablement on the forwarded traffic. -It is assumed that the traffic is forwarded using the default policy until it -is identified and new policies are enforced. - -The expected response delay is considered to be negligible, as only a small -number of packets are required to identify each flow. - -Graphical Overview -================== - -.. code-block:: console - - +----------------------------+ - | | - | Virtual Traffic Classifier | - | | - | Analysing/Forwarding | - | ------------> | - | ethA ethB | - | | - +----------------------------+ - | ^ - | | - v | - +----------------------------+ - | | - | Virtual Switch | - | | - +----------------------------+ - -Install -======= - -run the build.sh with root privileges - -Run -=== - -sudo ./pfbridge -a eth1 -b eth2 - -Development Environment -======================= - -Ubuntu 14.04 diff --git a/docs/configguide/yardstick_testcases/Yardstick_task_templates.rst b/docs/configguide/yardstick_testcases/Yardstick_task_templates.rst deleted file mode 100755 index 8185062b2..000000000 --- a/docs/configguide/yardstick_testcases/Yardstick_task_templates.rst +++ /dev/null @@ -1,155 +0,0 @@ -Task Template Syntax -==================== - -Basic template syntax ---------------------- -A nice feature of the input task format used in Yardstick is that it supports -the template syntax based on Jinja2. -This turns out to be extremely useful when, say, you have a fixed structure of -your task but you want to parameterize this task in some way. -For example, imagine your input task file (task.yaml) runs a set of Ping -scenarios: - -:: - - # Sample benchmark task config file - # measure network latency using ping - schema: "yardstick:task:0.1" - - scenarios: - - - type: Ping - options: - packetsize: 200 - host: athena.demo - target: ares.demo - - runner: - type: Duration - duration: 60 - interval: 1 - - sla: - max_rtt: 10 - action: monitor - - context: - ... - -Let's say you want to run the same set of scenarios with the same runner/ -context/sla, but you want to try another packetsize to compare the performance. -The most elegant solution is then to turn the packetsize name into a template -variable: - -:: - - # Sample benchmark task config file - # measure network latency using ping - - schema: "yardstick:task:0.1" - scenarios: - - - type: Ping - options: - packetsize: {{packetsize}} - host: athena.demo - target: ares.demo - - runner: - type: Duration - duration: 60 - interval: 1 - - sla: - max_rtt: 10 - action: monitor - - context: - ... - -and then pass the argument value for {{packetsize}} when starting a task with -this configuration file. -Yardstick provides you with different ways to do that: - -1.Pass the argument values directly in the command-line interface (with either -a JSON or YAML dictionary): - -:: - - yardstick task start samples/ping-template.yaml - --task-args'{"packetsize":"200"}' - -2.Refer to a file that specifies the argument values (JSON/YAML): - -:: - - yardstick task start samples/ping-template.yaml --task-args-file args.yaml - -Using the default values ------------------------- -Note that the Jinja2 template syntax allows you to set the default values for -your parameters. -With default values set, your task file will work even if you don't -parameterize it explicitly while starting a task. -The default values should be set using the {% set ... %} clause (task.yaml). -For example: - -:: - - # Sample benchmark task config file - # measure network latency using ping - schema: "yardstick:task:0.1" - {% set packetsize = packetsize or "100" %} - scenarios: - - - type: Ping - options: - packetsize: {{packetsize}} - host: athena.demo - target: ares.demo - - runner: - type: Duration - duration: 60 - interval: 1 - ... - -If you don't pass the value for {{packetsize}} while starting a task, the -default one will be used. - -Advanced templates ------------------- - -Yardstick makes it possible to use all the power of Jinja2 template syntax, -including the mechanism of built-in functions. -As an example, let us make up a task file that will do a block storage -performance test. -The input task file (fio-template.yaml) below uses the Jinja2 for-endfor -construct to accomplish that: - -:: - - #Test block sizes of 4KB, 8KB, 64KB, 1MB - #Test 5 workloads: read, write, randwrite, randread, rw - schema: "yardstick:task:0.1" - - scenarios: - {% for bs in ['4k', '8k', '64k', '1024k' ] %} - {% for rw in ['read', 'write', 'randwrite', 'randread', 'rw' ] %} - - - type: Fio - options: - filename: /home/ubuntu/data.raw - bs: {{bs}} - rw: {{rw}} - ramp_time: 10 - host: fio.demo - runner: - type: Duration - duration: 60 - interval: 60 - - {% endfor %} - {% endfor %} - context - ... diff --git a/docs/configguide/yardstick_testcases/glossary.rst b/docs/configguide/yardstick_testcases/glossary.rst deleted file mode 100644 index 8ce9a6ba0..000000000 --- a/docs/configguide/yardstick_testcases/glossary.rst +++ /dev/null @@ -1,33 +0,0 @@ -================== -Yardstick Glossary -================== - -.. glossary:: - :sorted: - - DPI - Deep Packet Inspection - - DSCP - Differentiated Services Code Point - - PBFS - Packet Based per Flow State - - QoS - Quality of Service - - VNF - Virtual Network Function - - VNFC - Virtual Network Function Component - - NFVI - Network Function Virtualization Infrastructure - - ToS - Type of Service - - VTC - Virtual Traffic Classifier diff --git a/docs/configguide/yardstick_testcases/index.rst b/docs/configguide/yardstick_testcases/index.rst deleted file mode 100644 index 55d4ea3e1..000000000 --- a/docs/configguide/yardstick_testcases/index.rst +++ /dev/null @@ -1,12 +0,0 @@ -================== -Yardstick Overview -================== - -.. toctree:: - :maxdepth: 2 - - 01-introduction - 02-methodology - 04-vtc-overview - 03-list-of-tcs - glossary diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc001.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc001.rst deleted file mode 100644 index 810bad489..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc001.rst +++ /dev/null @@ -1,79 +0,0 @@ -************************************* -Yardstick Test Case Description TC001 -************************************* - -.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt - -+-----------------------------------------------------------------------------+ -|Network Performance | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC001_NW PERF | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Number of flows and throughput | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS network performance with regards to | -| | flows and throughput, such as if and how different amounts | -| | of flows matter for the throughput between hosts on different| -| | compute blades. Typically e.g. the performance of a vSwitch | -| | depends on the number of flows running through it. Also | -| | performance of other equipment or entities can depend | -| | on the number of flows or the packet sizes used. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc001.yaml | -| | | -| | Packet size: 60 bytes | -| | Number of ports: 10, 50, 100, 500 and 1000, where each | -| | runs for 20 seconds. The whole sequence is run | -| | twice. The client and server are distributed on different HW.| -| | For SLA max_ppm is set to 1000. The amount of configured | -| | ports map to between 110 up to 1001000 flows, respectively. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | pktgen | -| | | -| | (Pktgen is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Docker | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | pktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different packet sizes, amount | -| | of flows and test duration. Default values exist. | -| | | -| | SLA (optional): max_ppm: The number of packets per million | -| | packets sent that are acceptable to loose, not received. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with pktgen included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. pktgen is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc002.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc002.rst deleted file mode 100644 index 56350f5bb..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc002.rst +++ /dev/null @@ -1,78 +0,0 @@ -************************************* -Yardstick Test Case Description TC002 -************************************* - -.. _cirros-image: https://download.cirros-cloud.net - -+-----------------------------------------------------------------------------+ -|Network Latency | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC002_NW LATENCY | -| | | -+--------------+--------------------------------------------------------------+ -|metric | RTT, Round Trip Time | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To do a basic verification that network latency is within | -| | acceptable boundaries when packets travel between hosts | -| | located on same or different compute blades. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs and similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc002.yaml | -| | | -| | Packet size 100 bytes. Total test duration 600 seconds. | -| | One ping each 10 seconds. SLA RTT is set to maximum 10 ms. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | ping | -| | | -| | Ping is normally part of any Linux distribution, hence it | -| | doesn't need to be installed. It is also part of the | -| | Yardstick Docker image. | -| | (For example also a Cirros image can be downloaded from | -| | cirros-image_, it includes ping) | -| | | -+--------------+--------------------------------------------------------------+ -|references | Ping man page | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test case can be configured with different packet sizes, | -| | burst sizes, ping intervals and test duration. | -| | SLA is optional. The SLA in this test case serves as an | -| | example. Considerably lower RTT is expected, and | -| | also normal to achieve in balanced L2 environments. However, | -| | to cover most configurations, both bare metal and fully | -| | virtualized ones, this value should be possible to achieve | -| | and acceptable for black box testing. Many real time | -| | applications start to suffer badly if the RTT time is higher | -| | than this. Some may suffer bad also close to this RTT, while | -| | others may not suffer at all. It is a compromise that may | -| | have to be tuned for different configuration purposes. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with ping included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. Ping is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Test should not PASS if any RTT is above the optional SLA | -| | value, or if there is a test case execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc005.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc005.rst deleted file mode 100644 index 83b6aedd6..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc005.rst +++ /dev/null @@ -1,72 +0,0 @@ -************************************* -Yardstick Test Case Description TC005 -************************************* - -.. _fio: http://www.bluestop.org/fio/HOWTO.txt - -+-----------------------------------------------------------------------------+ -|Storage Performance | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC005_Storage Performance | -| | | -+--------------+--------------------------------------------------------------+ -|metric | IOPS, throughput and latency | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS storage performance with regards to | -| | IOPS, throughput and latency. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs and similar shall be stored for comparison reasons | -| | and product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc005.yaml | -| | | -| | IO types: read, write, randwrite, randread, rw | -| | IO block size: 4KB, 64KB, 1024KB, where each | -| | runs for 30 seconds(10 for ramp time, 20 for runtime). | -| | | -| | For SLA minimum read/write iops is set to 100, minimum | -| | read/write throughput is set to 400 KB/s, and maximum | -| | read/write latency is set to 20000 usec. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | fio | -| | | -| | (fio is not always part of a Linux distribution, hence it | -| | needs to be installed. As an example see the | -| | /yardstick/tools/ directory for how to generate a Linux | -| | image with fio included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | fio_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different read/write types, IO | -| | block size, IO depth, ramp time (runtime required for stable | -| | results) and test duration. Default values exist. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with fio included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The host is installed and fio is invoked and logs are | -| | produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc006.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc006.rst deleted file mode 100644 index b68315078..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc006.rst +++ /dev/null @@ -1,139 +0,0 @@ -************************************* -Yardstick Test Case Description TC006 -************************************* - -.. _DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/ -.. _rfc2544: https://www.ietf.org/rfc/rfc2544.txt - -+-----------------------------------------------------------------------------+ -|Network Performance | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC006_Virtual Traffic Classifier Data Plane | -| | Throughput Benchmarking Test. | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Throughput | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To measure the throughput supported by the virtual Traffic | -| | Classifier according to the RFC2544 methodology for a | -| | user-defined set of vTC deployment configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: file: opnfv_yardstick_tc006.yaml | -| | | -| | packet_size: size of the packets to be used during the | -| | throughput calculation. | -| | Allowe values: [64, 128, 256, 512, 1024, 1280, 1518] | -| | | -| | vnic_type: type of VNIC to be used. | -| | Allowed values are: | -| | - normal: for default OvS port configuration | -| | - direct: for SR-IOV port configuration | -| | Default value: None | -| | | -| | vtc_flavor: OpenStack flavor to be used for the vTC | -| | Default available values are: m1.small, m1.medium, | -| | and m1.large, but the user can create his/her own | -| | flavor and give it as input | -| | Default value: None | -| | | -| | vlan_sender: vlan tag of the network on which the vTC will | -| | receive traffic (VLAN Network 1). | -| | Allowed values: range (1, 4096) | -| | | -| | vlan_receiver: vlan tag of the network on which the vTC | -| | will send traffic back to the packet generator | -| | (VLAN Network 2). | -| | Allowed values: range (1, 4096) | -| | | -| | default_net_name: neutron name of the defaul network that | -| | is used for access to the internet from the vTC | -| | (vNIC 1). | -| | | -| | default_subnet_name: subnet name for vNIC1 | -| | (information available through Neutron). | -| | | -| | vlan_net_1_name: Neutron Name for VLAN Network 1 | -| | (information available through Neutron). | -| | | -| | vlan_subnet_1_name: Subnet Neutron name for VLAN Network 1 | -| | (information available through Neutron). | -| | | -| | vlan_net_2_name: Neutron Name for VLAN Network 2 | -| | (information available through Neutron). | -| | | -| | vlan_subnet_2_name: Subnet Neutron name for VLAN Network 2 | -| | (information available through Neutron). | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | DPDK pktgen | -| | | -| | DPDK Pktgen is not part of a Linux distribution, | -| | hence it needs to be installed by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|references | DPDK Pktgen: DPDKpktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -| | RFC 2544: rfc2544_ | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different flavors, vNIC type | -| | and packet sizes. Default values exist as specified above. | -| | The vNIC type and flavor MUST be specified by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The vTC has been successfully instantiated and configured. | -| | The user has correctly assigned the values to the deployment | -| | configuration parameters. | -| | | -| | - Multicast traffic MUST be enabled on the network. | -| | The Data network switches need to be configured in | -| | order to manage multicast traffic. | -| | - In the case of SR-IOV vNICs use, SR-IOV compatible NICs | -| | must be used on the compute node. | -| | - Yarsdtick needs to be installed on a host connected to the | -| | data network and the host must have 2 DPDK-compatible | -| | NICs. Proper configuration of DPDK and DPDK pktgen is | -| | required before to run the test case. | -| | (For further instructions please refer to the ApexLake | -| | documentation). | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | Description and expected results | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The vTC is deployed, according to the user-defined | -| | configuration | -| | | -+--------------+--------------------------------------------------------------+ -|step 2 | The vTC is correctly deployed and configured as necessary | -| | The initialization script has been correctly executed and | -| | vTC is ready to receive and process the traffic. | -| | | -+--------------+--------------------------------------------------------------+ -|step 3 | Test case is executed with the selected parameters: | -| | - vTC flavor | -| | - vNIC type | -| | - packet size | -| | The traffic is sent to the vTC using the maximum available | -| | traffic rate for 60 seconds. | -| | | -+--------------+--------------------------------------------------------------+ -|step 4 | The vTC instance forwards all the packets back to the packet | -| | generator for 60 seconds, as specified by RFC 2544. | -| | | -| | Steps 3 and 4 are executed different times, with different | -| | rates in order to find the maximum supported traffic rate | -| | according to the current definition of throughput in RFC | -| | 2544. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | The result of the test is a number between 0 and 100 which | -| | represents the throughput in terms of percentage of the | -| | available pktgen NIC bandwidth. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc007.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc007.rst deleted file mode 100644 index a7a4776d5..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc007.rst +++ /dev/null @@ -1,157 +0,0 @@ -************************************* -Yardstick Test Case Description TC007 -************************************* - -.. _DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/ -.. _rfc2544: https://www.ietf.org/rfc/rfc2544.txt - -+-----------------------------------------------------------------------------+ -|Network Performance | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC007_Virtual Traffic Classifier Data Plane | -| | Throughput Benchmarking Test in Presence of Noisy | -| | neighbours | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Throughput | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To measure the throughput supported by the virtual Traffic | -| | Classifier according to the RFC2544 methodology for a | -| | user-defined set of vTC deployment configurations in the | -| | presence of noisy neighbours. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc007.yaml | -| | | -| | packet_size: size of the packets to be used during the | -| | throughput calculation. | -| | Allowe values: [64, 128, 256, 512, 1024, 1280, 1518] | -| | | -| | vnic_type: type of VNIC to be used. | -| | Allowed values are: | -| | - normal: for default OvS port configuration | -| | - direct: for SR-IOV port configuration | -| | | -| | vtc_flavor: OpenStack flavor to be used for the vTC | -| | Default available values are: m1.small, m1.medium, | -| | and m1.large, but the user can create his/her own | -| | flavor and give it as input | -| | | -| | num_of_neighbours: Number of noisy neighbours (VMs) to be | -| | instantiated during the experiment. | -| | Allowed values: range (1, 10) | -| | | -| | amount_of_ram: RAM to be used by each neighbor. | -| | Allowed values: ['250M', '1G', '2G', '3G', '4G', '5G', | -| | '6G', '7G', '8G', '9G', '10G'] | -| | Deault value: 256M | -| | | -| | number_of_cores: Number of noisy neighbours (VMs) to be | -| | instantiated during the experiment. | -| | Allowed values: range (1, 10) | -| | Default value: 1 | -| | | -| | vlan_sender: vlan tag of the network on which the vTC will | -| | receive traffic (VLAN Network 1). | -| | Allowed values: range (1, 4096) | -| | | -| | vlan_receiver: vlan tag of the network on which the vTC | -| | will send traffic back to the packet generator | -| | (VLAN Network 2). | -| | Allowed values: range (1, 4096) | -| | | -| | default_net_name: neutron name of the defaul network that | -| | is used for access to the internet from the vTC | -| | (vNIC 1). | -| | | -| | default_subnet_name: subnet name for vNIC1 | -| | (information available through Neutron). | -| | | -| | vlan_net_1_name: Neutron Name for VLAN Network 1 | -| | (information available through Neutron). | -| | | -| | vlan_subnet_1_name: Subnet Neutron name for VLAN Network 1 | -| | (information available through Neutron). | -| | | -| | vlan_net_2_name: Neutron Name for VLAN Network 2 | -| | (information available through Neutron). | -| | | -| | vlan_subnet_2_name: Subnet Neutron name for VLAN Network 2 | -| | (information available through Neutron). | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | DPDK pktgen | -| | | -| | DPDK Pktgen is not part of a Linux distribution, | -| | hence it needs to be installed by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|references | DPDKpktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -| | rfc2544_ | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different flavors, vNIC type | -| | and packet sizes. Default values exist as specified above. | -| | The vNIC type and flavor MUST be specified by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The vTC has been successfully instantiated and configured. | -| | The user has correctly assigned the values to the deployment | -| | configuration parameters. | -| | | -| | - Multicast traffic MUST be enabled on the network. | -| | The Data network switches need to be configured in | -| | order to manage multicast traffic. | -| | - In the case of SR-IOV vNICs use, SR-IOV compatible NICs | -| | must be used on the compute node. | -| | - Yarsdtick needs to be installed on a host connected to the | -| | data network and the host must have 2 DPDK-compatible | -| | NICs. Proper configuration of DPDK and DPDK pktgen is | -| | required before to run the test case. | -| | (For further instructions please refer to the ApexLake | -| | documentation). | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | Description and expected results | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The noisy neighbours are deployed as required by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|step 2 | The vTC is deployed, according to the configuration required | -| | by the user | -| | | -+--------------+--------------------------------------------------------------+ -|step 3 | The vTC is correctly deployed and configured as necessary. | -| | The initialization script has been correctly executed and | -| | the vTC is ready to receive and process the traffic. | -| | | -+--------------+--------------------------------------------------------------+ -|step 4 | Test case is executed with the parameters specified by the | -| | user: | -| | - vTC flavor | -| | - vNIC type | -| | - packet size | -| | The traffic is sent to the vTC using the maximum available | -| | traffic rate | -| | | -+--------------+--------------------------------------------------------------+ -|step 5 | The vTC instance forwards all the packets back to the | -| | packet generator for 60 seconds, as specified by RFC 2544. | -| | | -| | Steps 4 and 5 are executed different times with different | -| | with different traffic rates, in order to find the maximum | -| | supported traffic rate, accoring to the current definition | -| | of throughput in RFC 2544. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | The result of the test is a number between 0 and 100 which | -| | represents the throughput in terms of percentage of the | -| | available pktgen NIC bandwidth. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc008.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc008.rst deleted file mode 100644 index e176e633a..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc008.rst +++ /dev/null @@ -1,85 +0,0 @@ -************************************* -Yardstick Test Case Description TC008 -************************************* - -.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt - -+-----------------------------------------------------------------------------+ -|Packet Loss Extended Test | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC008_NW PERF, Packet loss Extended Test | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Number of flows, packet size and throughput | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS network performance with regards to | -| | flows and throughput, such as if and how different amounts | -| | of packet sizes and flows matter for the throughput between | -| | VMs on different compute blades. Typically e.g. the | -| | performance of a vSwitch | -| | depends on the number of flows running through it. Also | -| | performance of other equipment or entities can depend | -| | on the number of flows or the packet sizes used. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc008.yaml | -| | | -| | Packet size: 64, 128, 256, 512, 1024, 1280 and 1518 bytes. | -| | | -| | Number of ports: 1, 10, 50, 100, 500 and 1000. The amount of | -| | configured ports map from 2 up to 1001000 flows, | -| | respectively. Each packet_size/port_amount combination is run| -| | ten times, for 20 seconds each. Then the next | -| | packet_size/port_amount combination is run, and so on. | -| | | -| | The client and server are distributed on different HW. | -| | | -| | For SLA max_ppm is set to 1000. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | pktgen | -| | | -| | (Pktgen is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Docker | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | pktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different packet sizes, amount | -| | of flows and test duration. Default values exist. | -| | | -| | SLA (optional): max_ppm: The number of packets per million | -| | packets sent that are acceptable to loose, not received. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with pktgen included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. pktgen is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc009.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc009.rst deleted file mode 100644 index e4002a884..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc009.rst +++ /dev/null @@ -1,84 +0,0 @@ -************************************* -Yardstick Test Case Description TC009 -************************************* - -.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt - -+-----------------------------------------------------------------------------+ -|Packet Loss | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC009_NW PERF, Packet loss | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Number of flows, packets lost and throughput | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS network performance with regards to | -| | flows and throughput, such as if and how different amounts | -| | of flows matter for the throughput between VMs on different | -| | compute blades. | -| | Typically e.g. the performance of a vSwitch | -| | depends on the number of flows running through it. Also | -| | performance of other equipment or entities can depend | -| | on the number of flows or the packet sizes used. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc009.yaml | -| | | -| | Packet size: 64 bytes | -| | | -| | Number of ports: 1, 10, 50, 100, 500 and 1000. The amount of | -| | configured ports map from 2 up to 1001000 flows, | -| | respectively. Each port amount is run ten times, for 20 | -| | seconds each. Then the next port_amount is run, and so on. | -| | | -| | The client and server are distributed on different HW. | -| | | -| | For SLA max_ppm is set to 1000. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | pktgen | -| | | -| | (Pktgen is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Docker | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | pktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different packet sizes, amount | -| | of flows and test duration. Default values exist. | -| | | -| | SLA (optional): max_ppm: The number of packets per million | -| | packets sent that are acceptable to loose, not received. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with pktgen included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. pktgen is | -| | invoked and logs are produced and stored. | -| | | -| | Result: logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc010.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc010.rst deleted file mode 100644 index ebb74ea30..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc010.rst +++ /dev/null @@ -1,77 +0,0 @@ -************************************* -Yardstick Test Case Description TC010 -************************************* - -.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/lat_mem_rd.8.html - -+-----------------------------------------------------------------------------+ -|Memory Latency | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC010_Memory Latency | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Latency in nanoseconds | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | Measure the memory read latency for varying memory sizes and | -| | strides. Whole memory hierarchy is measured including all | -| | levels of cache. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | File: opnfv_yardstick_tc010.yaml | -| | | -| | * SLA (max_latency): 30 nanoseconds | -| | * Stride - 128 bytes | -| | * Stop size - 64 megabytes | -| | * Iterations: 10 - test is run 10 times iteratively. | -| | * Interval: 1 - there is 1 second delay between each | -| | iteration. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | Lmbench | -| | | -| | Lmbench is a suite of operating system microbenchmarks. This | -| | test uses lat_mem_rd tool from that suite. | -| | Lmbench is not always part of a Linux distribution, hence it | -| | needs to be installed in the test image | -| | | -+--------------+--------------------------------------------------------------+ -|references | man-pages_ | -| | | -| | McVoy, Larry W.,and Carl Staelin. "lmbench: Portable Tools | -| | for Performance Analysis." USENIX annual technical | -| | conference 1996. | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different: | -| | | -| | * strides; | -| | * stop_size; | -| | * iterations and intervals. | -| | | -| | There are default values for each above-mentioned option. | -| | | -| | SLA (optional) : max_latency: The maximum memory latency | -| | that is accepted. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with Lmbench included in the image. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The host is installed as client. Lmbench's lat_mem_rd tool | -| | is invoked and logs are produced and stored. | -| | | -| | Result: logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Test fails if the measured memory latency is above the SLA | -| | value or if there is a test case execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc011.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc011.rst deleted file mode 100644 index 6760ce067..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc011.rst +++ /dev/null @@ -1,83 +0,0 @@ -************************************* -Yardstick Test Case Description TC011 -************************************* - -.. _iperf3: https://iperf.fr/ - -+-----------------------------------------------------------------------------+ -|Packet delay variation between VMs | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC011_Packet delay variation between VMs | -| | | -+--------------+--------------------------------------------------------------+ -|metric | jitter: packet delay variation (ms) | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | Measure the packet delay variation sending the packets from | -| | one VM to the other. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | File: opnfv_yardstick_tc011.yaml | -| | | -| | * options: | -| | protocol: udp # The protocol used by iperf3 tools | -| | bandwidth: 20m # It will send the given number of packets | -| | without pausing | -| | * runner: | -| | duration: 30 # Total test duration 30 seconds. | -| | | -| | * SLA (optional): | -| | jitter: 10 (ms) # The maximum amount of jitter that is | -| | accepted. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | iperf3 | -| | | -| | iPerf3 is a tool for active measurements of the maximum | -| | achievable bandwidth on IP networks. It supports tuning of | -| | various parameters related to timing, buffers and protocols. | -| | The UDP protocols can be used to measure jitter delay. | -| | | -| | (iperf3 is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Docker | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | iperf3_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different | -| | | -| | * bandwidth: Test case can be configured with different | -| | bandwidth | -| | | -| | * duration: The test duration can be configured | -| | | -| | * jitter: SLA is optional. The SLA in this test case | -| | serves as an example. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with iperf3 included in the image. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. iperf3 is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Test should not PASS if any jitter is above the optional SLA | -| | value, or if there is a test case execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc012.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc012.rst deleted file mode 100644 index e7889c14e..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc012.rst +++ /dev/null @@ -1,81 +0,0 @@ -************************************* -Yardstick Test Case Description TC012 -************************************* - -.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/bw_mem.8.html - -+-----------------------------------------------------------------------------+ -|Memory Bandwidth | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC012_Memory Bandwidth | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Megabyte per second (MBps) | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | Measure the rate at which data can be read from and written | -| | to the memory (this includes all levels of memory). | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | File: opnfv_yardstick_tc012.yaml | -| | | -| | * SLA (optional): 15000 (MBps) min_bw: The minimum amount of | -| | memory bandwidth that is accepted. | -| | * Size: 10 240 kB - test allocates twice that size (20 480kB)| -| | zeros it and then measures the time it takes to copy from | -| | one side to another. | -| | * Benchmark: rdwr - measures the time to read data into | -| | memory and then write data to the same location. | -| | * Warmup: 0 - the number of iterations to perform before | -| | taking actual measurements. | -| | * Iterations: 10 - test is run 10 times iteratively. | -| | * Interval: 1 - there is 1 second delay between each | -| | iteration. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | Lmbench | -| | | -| | Lmbench is a suite of operating system microbenchmarks. This | -| | test uses bw_mem tool from that suite. | -| | Lmbench is not always part of a Linux distribution, hence it | -| | needs to be installed in the test image. | -| | | -+--------------+--------------------------------------------------------------+ -|references | man-pages_ | -| | | -| | McVoy, Larry W., and Carl Staelin. "lmbench: Portable Tools | -| | for Performance Analysis." USENIX annual technical | -| | conference. 1996. | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different: | -| | | -| | * memory sizes; | -| | * memory operations (such as rd, wr, rdwr, cp, frd, fwr, | -| | fcp, bzero, bcopy); | -| | * number of warmup iterations; | -| | * iterations and intervals. | -| | | -| | There are default values for each above-mentioned option. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with Lmbench included in the image. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The host is installed as client. Lmbench's bw_mem tool is | -| | invoked and logs are produced and stored. | -| | | -| | Result: logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Test fails if the measured memory bandwidth is below the SLA | -| | value or if there is a test case execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc014.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc014.rst deleted file mode 100644 index 68d36ecd2..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc014.rst +++ /dev/null @@ -1,69 +0,0 @@ -************************************* -Yardstick Test Case Description TC014 -************************************* - -.. _unixbench: https://github.com/kdlucas/byte-unixbench/blob/master/UnixBench - -+-----------------------------------------------------------------------------+ -|Processing speed | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC014_Processing speed | -| | | -+--------------+--------------------------------------------------------------+ -|metric | score of single cpu running, score of parallel running | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS processing speed with regards to score | -| | of single cpu running and parallel running | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs and similar shall be stored for comparison reasons | -| | and product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc014.yaml | -| | | -| | run_mode: Run unixbench in quiet mode or verbose mode | -| | test_type: dhry2reg, whetstone and so on | -| | | -| | For SLA with single_score and parallel_score, both can be | -| | set by user, default is NA | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | unixbench | -| | | -| | (unixbench is not always part of a Linux distribution, hence | -| | it needs to be installed. As an example see the | -| | /yardstick/tools/ directory for how to generate a Linux | -| | image with unixbench included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | unixbench_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different test types, dhry2reg, | -| | whetstone and so on. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with unixbench included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as a client. unixbench is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc019.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc019.rst deleted file mode 100644 index 482260b48..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc019.rst +++ /dev/null @@ -1,129 +0,0 @@ -************************************* -Yardstick Test Case Description TC019 -************************************* - -+-----------------------------------------------------------------------------+ -|Control Node Openstack Service High Availability | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC019_HA: Control node Openstack service down| -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | This test case will verify the high availability of the | -| | service provided by OpenStack (like nova-api, neutro-server) | -| | on control node. | -| | | -+--------------+--------------------------------------------------------------+ -|test method | This test case kills the processes of a specific Openstack | -| | service on a selected control node, then checks whether the | -| | request of the related Openstack command is OK and the killed| -| | processes are recovered. | -| | | -+--------------+--------------------------------------------------------------+ -|attackers | In this test case, an attacker called "kill-process" is | -| | needed. This attacker includes three parameters: | -| | 1) fault_type: which is used for finding the attacker's | -| | scripts. It should be always set to "kill-process" in this | -| | test case. | -| | 2) process_name: which is the process name of the specified | -| | OpenStack service. If there are multiple processes use the | -| | same name on the host, all of them are killed by this | -| | attacker. | -| | 3) host: which is the name of a control node being attacked. | -| | | -| | e.g. | -| | -fault_type: "kill-process" | -| | -process_name: "nova-api" | -| | -host: node1 | -| | | -+--------------+--------------------------------------------------------------+ -|monitors | In this test case, two kinds of monitor are needed: | -| | 1. the "openstack-cmd" monitor constantly request a specific | -| | Openstack command, which needs two parameters: | -| | 1) monitor_type: which is used for finding the monitor class | -| | and related scritps. It should be always set to | -| | "openstack-cmd" for this monitor. | -| | 2) command_name: which is the command name used for request | -| | | -| | 2. the "process" monitor check whether a process is running | -| | on a specific node, which needs three parameters: | -| | 1) monitor_type: which used for finding the monitor class and| -| | related scritps. It should be always set to "process" | -| | for this monitor. | -| | 2) process_name: which is the process name for monitor | -| | 3) host: which is the name of the node runing the process | -| | | -| | e.g. | -| | monitor1: | -| | -monitor_type: "openstack-cmd" | -| | -command_name: "nova image-list" | -| | monitor2: | -| | -monitor_type: "process" | -| | -process_name: "nova-api" | -| | -host: node1 | -| | | -+--------------+--------------------------------------------------------------+ -|metrics | In this test case, there are two metrics: | -| | 1)service_outage_time: which indicates the maximum outage | -| | time (seconds) of the specified Openstack command request. | -| | 2)process_recover_time: which indicates the maximun time | -| | (seconds) from the process being killed to recovered | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | Developed by the project. Please see folder: | -| | "yardstick/benchmark/scenarios/availability/ha_tools" | -| | | -+--------------+--------------------------------------------------------------+ -|references | ETSI NFV REL001 | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | This test case needs two configuration files: | -| | 1) test case file: opnfv_yardstick_tc019.yaml | -| | -Attackers: see above "attackers" discription | -| | -waiting_time: which is the time (seconds) from the process | -| | being killed to stoping monitors the monitors | -| | -Monitors: see above "monitors" discription | -| | -SLA: see above "metrics" discription | -| | | -| | 2)POD file: pod.yaml | -| | The POD configuration should record on pod.yaml first. | -| | the "host" item in this test case will use the node name in | -| | the pod.yaml. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | start monitors: | -| | each monitor will run with independently process | -| | | -| | Result: The monitor info will be collected. | -| | | -+--------------+--------------------------------------------------------------+ -|step 2 | do attacker: connect the host through SSH, and then execute | -| | the kill process script with param value specified by | -| | "process_name" | -| | | -| | Result: Process will be killed. | -| | | -+--------------+--------------------------------------------------------------+ -|step 3 | stop monitors after a period of time specified by | -| | "waiting_time" | -| | | -| | Result: The monitor info will be aggregated. | -| | | -+--------------+--------------------------------------------------------------+ -|step 4 | verify the SLA | -| | | -| | Result: The test case is passed or not. | -| | | -+--------------+--------------------------------------------------------------+ -|post-action | It is the action when the test cases exist. It will check the| -| | status of the specified process on the host, and restart the | -| | process if it is not running for next test cases | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc020.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc020.rst deleted file mode 100644 index 9a5130f71..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc020.rst +++ /dev/null @@ -1,136 +0,0 @@ -************************************* -Yardstick Test Case Description TC020 -************************************* - -.. _DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/ -.. _rfc2544: https://www.ietf.org/rfc/rfc2544.txt - -+-----------------------------------------------------------------------------+ -|Network Performance | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC0020_Virtual Traffic Classifier | -| | Instantiation Test | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Failure | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To verify that a newly instantiated vTC is 'alive' and | -| | functional and its instantiation is correctly supported by | -| | the infrastructure. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc020.yaml | -| | | -| | vnic_type: type of VNIC to be used. | -| | Allowed values are: | -| | - normal: for default OvS port configuration | -| | - direct: for SR-IOV port configuration | -| | Default value: None | -| | | -| | vtc_flavor: OpenStack flavor to be used for the vTC | -| | Default available values are: m1.small, m1.medium, | -| | and m1.large, but the user can create his/her own | -| | flavor and give it as input | -| | Default value: None | -| | | -| | vlan_sender: vlan tag of the network on which the vTC will | -| | receive traffic (VLAN Network 1). | -| | Allowed values: range (1, 4096) | -| | | -| | vlan_receiver: vlan tag of the network on which the vTC | -| | will send traffic back to the packet generator | -| | (VLAN Network 2). | -| | Allowed values: range (1, 4096) | -| | | -| | default_net_name: neutron name of the defaul network that | -| | is used for access to the internet from the vTC | -| | (vNIC 1). | -| | | -| | default_subnet_name: subnet name for vNIC1 | -| | (information available through Neutron). | -| | | -| | vlan_net_1_name: Neutron Name for VLAN Network 1 | -| | (information available through Neutron). | -| | | -| | vlan_subnet_1_name: Subnet Neutron name for VLAN Network 1 | -| | (information available through Neutron). | -| | | -| | vlan_net_2_name: Neutron Name for VLAN Network 2 | -| | (information available through Neutron). | -| | | -| | vlan_subnet_2_name: Subnet Neutron name for VLAN Network 2 | -| | (information available through Neutron). | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | DPDK pktgen | -| | | -| | DPDK Pktgen is not part of a Linux distribution, | -| | hence it needs to be installed by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|references | DPDKpktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -| | rfc2544_ | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different flavors, vNIC type | -| | and packet sizes. Default values exist as specified above. | -| | The vNIC type and flavor MUST be specified by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The vTC has been successfully instantiated and configured. | -| | The user has correctly assigned the values to the deployment | -| | configuration parameters. | -| | | -| | - Multicast traffic MUST be enabled on the network. | -| | The Data network switches need to be configured in | -| | order to manage multicast traffic. | -| | Installation and configuration of smcroute is required | -| | before to run the test case. | -| | (For further instructions please refer to the ApexLake | -| | documentation). | -| | - In the case of SR-IOV vNICs use, SR-IOV compatible NICs | -| | must be used on the compute node. | -| | - Yarsdtick needs to be installed on a host connected to the | -| | data network and the host must have 2 DPDK-compatible | -| | NICs. Proper configuration of DPDK and DPDK pktgen is | -| | required before to run the test case. | -| | (For further instructions please refer to the ApexLake | -| | documentation). | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | Description and expected results | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The vTC is deployed, according to the configuration provided | -| | by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|step 2 | The vTC is correctly deployed and configured as necessary. | -| | The initialization script has been correctly executed and | -| | the vTC is ready to receive and process the traffic. | -| | | -+--------------+--------------------------------------------------------------+ -|step 3 | Test case is executed with the parameters specified by the | -| | the user: | -| | - vTC flavor | -| | - vNIC type | -| | A constant rate traffic is sent to the vTC for 10 seconds. | -| | | -+--------------+--------------------------------------------------------------+ -|step 4 | The vTC instance tags all the packets and sends them back to | -| | the packet generator for 10 seconds. | -| | | -| | The framework checks that the packet generator receives | -| | back all the packets with the correct tag from the vTC. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | The vTC is deemed to be successfully instantiated if all | -| | packets are sent back with the right tag as requested, | -| | else it is deemed DoA (Dead on arrival) | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc021.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc021.rst deleted file mode 100644 index a493ddfc0..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc021.rst +++ /dev/null @@ -1,152 +0,0 @@ -************************************* -Yardstick Test Case Description TC021 -************************************* - -.. _DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/ -.. _rfc2544: https://www.ietf.org/rfc/rfc2544.txt - -+-----------------------------------------------------------------------------+ -|Network Performance | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC0021_Virtual Traffic Classifier | -| | Instantiation Test in Presence of Noisy Neighbours | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Failure | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To verify that a newly instantiated vTC is 'alive' and | -| | functional and its instantiation is correctly supported by | -| | the infrastructure in the presence of noisy neighbours. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc021.yaml | -| | | -| | vnic_type: type of VNIC to be used. | -| | Allowed values are: | -| | - normal: for default OvS port configuration | -| | - direct: for SR-IOV port configuration | -| | Default value: None | -| | | -| | vtc_flavor: OpenStack flavor to be used for the vTC | -| | Default available values are: m1.small, m1.medium, | -| | and m1.large, but the user can create his/her own | -| | flavor and give it as input | -| | Default value: None | -| | | -| | num_of_neighbours: Number of noisy neighbours (VMs) to be | -| | instantiated during the experiment. | -| | Allowed values: range (1, 10) | -| | | -| | amount_of_ram: RAM to be used by each neighbor. | -| | Allowed values: ['250M', '1G', '2G', '3G', '4G', '5G', | -| | '6G', '7G', '8G', '9G', '10G'] | -| | Deault value: 256M | -| | | -| | number_of_cores: Number of noisy neighbours (VMs) to be | -| | instantiated during the experiment. | -| | Allowed values: range (1, 10) | -| | Default value: 1 | -| | | -| | vlan_sender: vlan tag of the network on which the vTC will | -| | receive traffic (VLAN Network 1). | -| | Allowed values: range (1, 4096) | -| | | -| | vlan_receiver: vlan tag of the network on which the vTC | -| | will send traffic back to the packet generator | -| | (VLAN Network 2). | -| | Allowed values: range (1, 4096) | -| | | -| | default_net_name: neutron name of the defaul network that | -| | is used for access to the internet from the vTC | -| | (vNIC 1). | -| | | -| | default_subnet_name: subnet name for vNIC1 | -| | (information available through Neutron). | -| | | -| | vlan_net_1_name: Neutron Name for VLAN Network 1 | -| | (information available through Neutron). | -| | | -| | vlan_subnet_1_name: Subnet Neutron name for VLAN Network 1 | -| | (information available through Neutron). | -| | | -| | vlan_net_2_name: Neutron Name for VLAN Network 2 | -| | (information available through Neutron). | -| | | -| | vlan_subnet_2_name: Subnet Neutron name for VLAN Network 2 | -| | (information available through Neutron). | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | DPDK pktgen | -| | | -| | DPDK Pktgen is not part of a Linux distribution, | -| | hence it needs to be installed by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|references | DPDK Pktgen: DPDK Pktgen: DPDKpktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -| | RFC 2544: rfc2544_ | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different flavors, vNIC type | -| | and packet sizes. Default values exist as specified above. | -| | The vNIC type and flavor MUST be specified by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The vTC has been successfully instantiated and configured. | -| | The user has correctly assigned the values to the deployment | -| | configuration parameters. | -| | | -| | - Multicast traffic MUST be enabled on the network. | -| | The Data network switches need to be configured in | -| | order to manage multicast traffic. | -| | Installation and configuration of smcroute is required | -| | before to run the test case. | -| | (For further instructions please refer to the ApexLake | -| | documentation). | -| | - In the case of SR-IOV vNICs use, SR-IOV compatible NICs | -| | must be used on the compute node. | -| | - Yarsdtick needs to be installed on a host connected to the | -| | data network and the host must have 2 DPDK-compatible | -| | NICs. Proper configuration of DPDK and DPDK pktgen is | -| | required before to run the test case. | -| | (For further instructions please refer to the ApexLake | -| | documentation). | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | Description and expected results | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The noisy neighbours are deployed as required by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|step 2 | The vTC is deployed, according to the configuration provided | -| | by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|step 3 | The vTC is correctly deployed and configured as necessary. | -| | The initialization script has been correctly executed and | -| | the vTC is ready to receive and process the traffic. | -| | | -+--------------+--------------------------------------------------------------+ -|step 4 | Test case is executed with the selected parameters: | -| | - vTC flavor | -| | - vNIC type | -| | A constant rate traffic is sent to the vTC for 10 seconds. | -| | | -+--------------+--------------------------------------------------------------+ -|step 5 | The vTC instance tags all the packets and sends them back to | -| | the packet generator for 10 seconds. | -| | | -| | The framework checks if the packet generator receives back | -| | all the packets with the correct tag from the vTC. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | The vTC is deemed to be successfully instantiated if all | -| | packets are sent back with the right tag as requested, | -| | else it is deemed DoA (Dead on arrival) | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc024.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc024.rst deleted file mode 100644 index 1e8d4e6f3..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc024.rst +++ /dev/null @@ -1,64 +0,0 @@ -************************************* -Yardstick Test Case Description TC024 -************************************* - -.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/man1/mpstat.1.html - -+-----------------------------------------------------------------------------+ -| CPU Load | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC024_CPU Load | -| | | -+--------------+--------------------------------------------------------------+ -|metric | CPU load | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the CPU load performance of the IaaS. This test | -| | case should be run in parallel to other Yardstick test cases | -| | and not run as a stand-alone test case. | -| | | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: cpuload.yaml (in the 'samples' directory) | -| | | -| | There is are no additional configurations to be set for this | -| | TC. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | mpstat | -| | | -| | (mpstat is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Glance | -| | image. However, if mpstat is not present the TC instead uses | -| | /proc/stats as source to produce "mpstat" output. | -| | | -+--------------+--------------------------------------------------------------+ -|references | man-pages_ | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Run in background with other test cases. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with mpstat included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The host is installed. The related TC, or TCs, is | -| | invoked and mpstat logs are produced and stored. | -| | | -| | Result: Stored logs | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | None. CPU load results are fetched and stored. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc025.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc025.rst deleted file mode 100644 index 0bc0b78ab..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc025.rst +++ /dev/null @@ -1,118 +0,0 @@ -************************************* -Yardstick Test Case Description TC025 -************************************* - -+-----------------------------------------------------------------------------+ -|OpenStack Controller Node abnormally shutdown High Availability | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC025_HA: OpenStack Controller Node | -| | abnormally shutdown | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | This test case will verify the high availability of | -| | controller node. When one of the controller node abnormally | -| | shutdown, the service provided by it should be OK. | -| | | -+--------------+--------------------------------------------------------------+ -|test method | This test case shutdowns a specified controller node with | -| | some fault injection tools, then checks whether all services | -| | provided by the controller node are OK with some monitor | -| | tools. | -| | | -+--------------+--------------------------------------------------------------+ -|attackers | In this test case, an attacker called "host-shutdown" is | -| | needed. This attacker includes two parameters: | -| | 1) fault_type: which is used for finding the attacker's | -| | scripts. It should be always set to "host-shutdown" in | -| | this test case. | -| | 2) host: the name of a controller node being attacked. | -| | | -| | e.g. | -| | -fault_type: "host-shutdown" | -| | -host: node1 | -| | | -+--------------+--------------------------------------------------------------+ -|monitors | In this test case, one kind of monitor are needed: | -| | 1. the "openstack-cmd" monitor constantly request a specific | -| | Openstack command, which needs two parameters | -| | 1) monitor_type: which is used for finding the monitor class | -| | and related scritps. It should be always set to | -| | "openstack-cmd" for this monitor. | -| | 2) command_name: which is the command name used for request | -| | | -| | There are four instance of the "openstack-cmd" monitor: | -| | monitor1: | -| | -monitor_type: "openstack-cmd" | -| | -api_name: "nova image-list" | -| | monitor2: | -| | -monitor_type: "openstack-cmd" | -| | -api_name: "neutron router-list" | -| | monitor3: | -| | -monitor_type: "openstack-cmd" | -| | -api_name: "heat stack-list" | -| | monitor4: | -| | -monitor_type: "openstack-cmd" | -| | -api_name: "cinder list" | -| | | -+--------------+--------------------------------------------------------------+ -|metrics | In this test case, there is one metric: | -| | 1)service_outage_time: which indicates the maximum outage | -| | time (seconds) of the specified Openstack command request. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | Developed by the project. Please see folder: | -| | "yardstick/benchmark/scenarios/availability/ha_tools" | -| | | -+--------------+--------------------------------------------------------------+ -|references | ETSI NFV REL001 | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | This test case needs two configuration files: | -| | 1) test case file: opnfv_yardstick_tc019.yaml | -| | -Attackers: see above "attackers" discription | -| | -waiting_time: which is the time (seconds) from the process | -| | being killed to stoping monitors the monitors | -| | -Monitors: see above "monitors" discription | -| | -SLA: see above "metrics" discription | -| | | -| | 2)POD file: pod.yaml | -| | The POD configuration should record on pod.yaml first. | -| | the "host" item in this test case will use the node name in | -| | the pod.yaml. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | start monitors: | -| | each monitor will run with independently process | -| | | -| | Result: The monitor info will be collected. | -| | | -+--------------+--------------------------------------------------------------+ -|step 2 | do attacker: connect the host through SSH, and then execute | -| | shutdown script on the host | -| | | -| | Result: The host will be shutdown. | -| | | -+--------------+--------------------------------------------------------------+ -|step 3 | stop monitors after a period of time specified by | -| | "waiting_time" | -| | | -| | Result: All monitor result will be aggregated. | -| | | -+--------------+--------------------------------------------------------------+ -|step 4 | verify the SLA | -| | | -| | Result: The test case is passed or not. | -| | | -+--------------+--------------------------------------------------------------+ -|post-action | It is the action when the test cases exist. It restarts the | -| | specified controller node if it is not restarted. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc027.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc027.rst deleted file mode 100644 index 56c8227df..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc027.rst +++ /dev/null @@ -1,67 +0,0 @@ -************************************* -Yardstick Test Case Description TC027 -************************************* - -.. _ipv6: https://wiki.opnfv.org/ipv6_opnfv_project - -+-----------------------------------------------------------------------------+ -|IPv6 connectivity between nodes on the tenant network | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC002_IPv6 connectivity | -| | | -+--------------+--------------------------------------------------------------+ -|metric | RTT, Round Trip Time | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To do a basic verification that IPv6 connectivity is within | -| | acceptable boundaries when ipv6 packets travel between hosts | -| | located on same or different compute blades. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs and similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc027.yaml | -| | | -| | Packet size 56 bytes. | -| | SLA RTT is set to maximum 10 ms. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | ping6 | -| | | -| | Ping6 is normally part of Linux distribution, hence it | -| | doesn't need to be installed. | -| | | -+--------------+--------------------------------------------------------------+ -|references | ipv6_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test case can be configured with different run step | -| | you can run setup, run benchmakr, teardown independently | -| | SLA is optional. The SLA in this test case serves as an | -| | example. Considerably lower RTT is expected. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with ping6 included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. Ping is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Test should not PASS if any RTT is above the optional SLA | -| | value, or if there is a test case execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc028.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc028.rst deleted file mode 100644 index 8ac5f49f9..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc028.rst +++ /dev/null @@ -1,65 +0,0 @@ -************************************* -Yardstick Test Case Description TC028 -************************************* - -.. _Cyclictest: https://rt.wiki.kernel.org/index.php/Cyclictest - -+-----------------------------------------------------------------------------+ -|KVM Latency measurements | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC028_KVM Latency measurements | -| | | -+--------------+--------------------------------------------------------------+ -|metric | min, avg and max latency | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS KVM virtualization capability with | -| | regards to min, avg and max latency. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs and similar shall be stored for comparison reasons | -| | and product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: samples/cyclictest-node-context.yaml | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | Cyclictest | -| | | -| | (Cyclictest is not always part of a Linux distribution, | -| | hence it needs to be installed. As an example see the | -| | /yardstick/tools/ directory for how to generate a Linux | -| | image with cyclictest included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | Cyclictest_ | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | This test case is mainly for kvm4nfv project CI verify. | -| | Upgrade host linux kernel, boot a gust vm update it's linux | -| | kernel, and then run the cyclictest to test the new kernel | -| | is work well. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test kernel rpm, test sequence scripts and test guest | -|conditions | image need put the right folders as specified in the test | -| | case yaml file. | -| | The test guest image needs with cyclictest included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The host and guest os kernel is upgraded. Cyclictest is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc037.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc037.rst deleted file mode 100644 index 5c91f6bf1..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc037.rst +++ /dev/null @@ -1,99 +0,0 @@ -************************************* -Yardstick Test Case Description TC037 -************************************* - -.. _cirros: https://download.cirros-cloud.net -.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt - -+-----------------------------------------------------------------------------+ -|Latency, CPU Load, Throughput, Packet Loss | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC037_Latency,CPU Load,Throughput,Packet Loss| -| | | -+--------------+--------------------------------------------------------------+ -|metric | Number of flows, latency, throughput, CPU load, packet loss | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS network performance with regards to | -| | flows and throughput, such as if and how different amounts | -| | of flows matter for the throughput between hosts on different| -| | compute blades. Typically e.g. the performance of a vSwitch | -| | depends on the number of flows running through it. Also | -| | performance of other equipment or entities can depend | -| | on the number of flows or the packet sizes used. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc037.yaml | -| | | -| | Packet size: 64 bytes | -| | Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. | -| | The amount configured ports map from 2 up to 1001000 flows, | -| | respectively. Each port amount is run two times, for 20 | -| | seconds each. Then the next port_amount is run, and so on. | -| | During the test CPU load on both client and server, and the | -| | network latency between the client and server are measured. | -| | The client and server are distributed on different HW. | -| | For SLA max_ppm is set to 1000. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | pktgen | -| | | -| | (Pktgen is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Glance | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -| | ping | -| | | -| | Ping is normally part of any Linux distribution, hence it | -| | doesn't need to be installed. It is also part of the | -| | Yardstick Glance image. | -| | (For example also a cirros_ image can be downloaded, it | -| | includes ping) | -| | | -| | mpstat | -| | | -| | (Mpstat is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Glance | -| | image. | -| | | -+--------------+--------------------------------------------------------------+ -|references | Ping and Mpstat man pages | -| | | -| | pktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different packet sizes, amount | -| | of flows and test duration. Default values exist. | -| | | -| | SLA (optional): max_ppm: The number of packets per million | -| | packets sent that are acceptable to loose, not received. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with pktgen included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. pktgen is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc038.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc038.rst deleted file mode 100644 index 93c2cf3d8..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc038.rst +++ /dev/null @@ -1,99 +0,0 @@ -************************************* -Yardstick Test Case Description TC038 -************************************* - -.. _cirros: https://download.cirros-cloud.net -.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt - -+-----------------------------------------------------------------------------+ -|Latency, CPU Load, Throughput, Packet Loss (Extended measurements) | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC038_Latency,CPU Load,Throughput,Packet Loss| -| | | -+--------------+--------------------------------------------------------------+ -|metric | Number of flows, latency, throughput, CPU load, packet loss | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS network performance with regards to | -| | flows and throughput, such as if and how different amounts | -| | of flows matter for the throughput between hosts on different| -| | compute blades. Typically e.g. the performance of a vSwitch | -| | depends on the number of flows running through it. Also | -| | performance of other equipment or entities can depend | -| | on the number of flows or the packet sizes used. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc038.yaml | -| | | -| | Packet size: 64 bytes | -| | Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. | -| | The amount configured ports map from 2 up to 1001000 flows, | -| | respectively. Each port amount is run ten times, for 20 | -| | seconds each. Then the next port_amount is run, and so on. | -| | During the test CPU load on both client and server, and the | -| | network latency between the client and server are measured. | -| | The client and server are distributed on different HW. | -| | For SLA max_ppm is set to 1000. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | pktgen | -| | | -| | (Pktgen is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Glance | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -| | ping | -| | | -| | Ping is normally part of any Linux distribution, hence it | -| | doesn't need to be installed. It is also part of the | -| | Yardstick Glance image. | -| | (For example also a cirros_ image can be downloaded, it | -| | includes ping) | -| | | -| | mpstat | -| | | -| | (Mpstat is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Glance | -| | image. | -| | | -+--------------+--------------------------------------------------------------+ -|references | Ping and Mpstat man pages | -| | | -| | pktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different packet sizes, amount | -| | of flows and test duration. Default values exist. | -| | | -| | SLA (optional): max_ppm: The number of packets per million | -| | packets sent that are acceptable to loose, not received. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with pktgen included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. pktgen is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc040.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc040.rst deleted file mode 100644 index 044ccf193..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc040.rst +++ /dev/null @@ -1,60 +0,0 @@ -************************************* -Yardstick Test Case Description TC040 -************************************* - -.. _Parser: https://wiki.opnfv.org/parser - -+-----------------------------------------------------------------------------+ -|Verify Parser Yang-to-Tosca | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC040 Verify Parser Yang-to-Tosca | -| | | -+--------------+--------------------------------------------------------------+ -|metric | 1. tosca file which is converted from yang file by Parser | -| | 2. result whether the output is same with expected outcome | -+--------------+--------------------------------------------------------------+ -|test purpose | To verify the function of Yang-to-Tosca in Parser. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc040.yaml | -| | | -| | yangfile: the path of the yangfile which you want to convert | -| | toscafile: the path of the toscafile which is your expected | -| | outcome. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | Parser | -| | | -| | (Parser is not part of a Linux distribution, hence it | -| | needs to be installed. As an example see the | -| | /yardstick/benchmark/scenarios/parser/parser_setup.sh for | -| | how to install it manual. Of course, it will be installed | -| | and uninstalled automatically when you run this test case | -| | by yardstick) | -+--------------+--------------------------------------------------------------+ -|references | Parser_ | -| | | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different path of yangfile and | -| | toscafile to fit your real environment to verify Parser | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | No POD specific requirements have been identified. | -|conditions | it can be run without VM | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | parser is installed without VM, running Yang-to-Tosca module | -| | to convert yang file to tosca file, validating output against| -| | expected outcome. | -| | | -| | Result: Logs are stored. | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if output is different with expected outcome | -| | or if there is a test case execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/testcase_description_v2_template.rst b/docs/configguide/yardstick_testcases/testcase_description_v2_template.rst deleted file mode 100644 index 1b8754b05..000000000 --- a/docs/configguide/yardstick_testcases/testcase_description_v2_template.rst +++ /dev/null @@ -1,64 +0,0 @@ -.. Template to be used for test case descriptions in Yardstick Project. - Write one .rst per test case. - Upload the .rst for the test case in /docs/source/yardstick directory. - Review in Gerrit. - -************************************* -Yardstick Test Case Description TCXXX -************************************* - -+-----------------------------------------------------------------------------+ -|test case slogan e.g. Network Latency | -| | -+--------------+--------------------------------------------------------------+ -|test case id | e.g. OPNFV_YARDSTICK_TC001_NW Latency | -| | | -+--------------+--------------------------------------------------------------+ -|metric | what will be measured, e.g. latency | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | describe what is the purpose of the test case | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | what .yaml file to use, state SLA if applicable, state | -| | test duration, list and describe the scenario options used in| -| | this TC and also list the options using default values. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | e.g. ping | -| | | -+--------------+--------------------------------------------------------------+ -|references | e.g. RFCxxx, ETSI-NFVyyy | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | describe variations of the test case which can be | -| | performend, e.g. run the test for different packet sizes | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | describe configuration in the tool(s) used to perform | -|conditions | the measurements (e.g. fio, pktgen), POD-specific | -| | configuration required to enable running the test | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | use this to describe tests that require sveveral steps e.g | -| | collect logs. | -| | | -| | Result: what happens in this step e.g. logs collected | -| | | -+--------------+--------------------------------------------------------------+ -|step 2 | remove interface | -| | | -| | Result: interface down. | -| | | -+--------------+--------------------------------------------------------------+ -|step N | what is done in step N | -| | | -| | Result: what happens | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | expected behavior, or SLA, pass/fail criteria | -| | | -+--------------+--------------------------------------------------------------+ |