diff options
Diffstat (limited to 'docs/configguide/yardstick_testcases')
29 files changed, 0 insertions, 2681 deletions
diff --git a/docs/configguide/yardstick_testcases/01-introduction.rst b/docs/configguide/yardstick_testcases/01-introduction.rst deleted file mode 100644 index 6cca2875e..000000000 --- a/docs/configguide/yardstick_testcases/01-introduction.rst +++ /dev/null @@ -1,38 +0,0 @@ -============ -Introduction -============ - -**Welcome to Yardstick's documentation !** - -.. _Yardstick: https://wiki.opnfv.org/yardstick - -Yardstick_ is an OPNFV Project. - -The project's goal is to verify infrastructure compliance, from the perspective -of a :term:`VNF`. - -The Project's scope is the development of a test framework, *Yardstick*, test -cases and test stimuli to enable :term:`NFVI` verification. -The Project also includes a sample :term:`VNF`, the :term:`VTC` and its -experimental framework, *ApexLake* ! - -The chapter :doc:`02-methodology` describes the methodology implemented by the -Yardstick Project for :term:`NFVI` verification. The chapter -:doc:`03-list-of-tcs` includes a list of available Yardstick test cases. - -Yardstick is used for verifying the OPNFV infrastructure and some of the OPNFV -features, listed in :doc:`03-list-of-tcs`. - -The *Yardstick* framework is deployed in several OPNFV community labs. It is -installer, infrastructure and application independent. - -.. _Pharos: https://wiki.opnfv.org/pharos - -.. seealso:: Pharos_ for information on OPNFV community labs. - -Contact Yardstick -================= - -Feedback? `Contact us`_ - -.. _Contact us: opnfv-users@lists.opnfv.org diff --git a/docs/configguide/yardstick_testcases/02-methodology.rst b/docs/configguide/yardstick_testcases/02-methodology.rst deleted file mode 100644 index 5097c566b..000000000 --- a/docs/configguide/yardstick_testcases/02-methodology.rst +++ /dev/null @@ -1,181 +0,0 @@ -=========== -Methodology -=========== - -Abstract -======== - -This chapter describes the methodology implemented by the Yardstick project for -verifying the NFV Infrastructure from the perspective of a VNF. - -ETSI-NFV -======== - -.. _NFV-TST001: https://docbox.etsi.org/ISG/NFV/Open/Drafts/TST001_-_Pre-deployment_Validation/ - -The document ETSI GS NFV-TST001_, "Pre-deployment Testing; Report on Validation -of NFV Environments and Services", recommends methods for pre-deployment -testing of the functional components of an NFV environment. - -The Yardstick project implements the methodology described in chapter 6, "Pre- -deployment validation of NFV infrastructure". - -The methodology consists in decomposing the typical VNF work-load performance -metrics into a number of characteristics/performance vectors, which each can be -represented by distinct test-cases. - -The methodology includes five steps: - -* *Step1:* Define Infrastruture - the HW, SW and corresponding configuration - target for validation; the OPNFV infrastructure, in OPNFV community labs. - -* *Step2:* Identify VNF type - the application for which the infrastructure is - to be validated, and its requirements on the underlying infrastructure. - -* *Step3:* Select test cases - depending on the workload that represents the - application for which the infrastruture is to be validated, the relevant - test cases amongst the list of available Yardstick test cases. - -* *Step4:* Execute tests - define the duration and number of iterations for the - selected test cases, tests runs are automated via OPNFV Jenkins Jobs. - -* *Step5:* Collect results - using the common API for result collection. - -Metrics -======= - -The metrics, as defined by ETSI GS NFV-TST001, are shown in -:ref:`Table1 <table2_1>`, :ref:`Table2 <table2_2>` and -:ref:`Table3 <table2_3>`. - -In OPNFV Brahmaputra release, generic test cases covering aspects of the listed -metrics are available; further OPNFV releases will provide extended testing of -these metrics. -The view of available Yardstick test cases cross ETSI definitions in -:ref:`Table1 <table2_1>`, :ref:`Table2 <table2_2>` and :ref:`Table3 <table2_3>` -is shown in :ref:`Table4 <table2_4>`. -It shall be noticed that the Yardstick test cases are examples, the test -duration and number of iterations are configurable, as are the System Under -Test (SUT) and the attributes (or, in Yardstick nomemclature, the scenario -options). - -.. _table2_1: - -**Table 1 - Performance/Speed Metrics** - -+---------+-------------------------------------------------------------------+ -| Category| Performance/Speed | -| | | -+---------+-------------------------------------------------------------------+ -| Compute | * Latency for random memory access | -| | * Latency for cache read/write operations | -| | * Processing speed (instructions per second) | -| | * Throughput for random memory access (bytes per second) | -| | | -+---------+-------------------------------------------------------------------+ -| Network | * Throughput per NFVI node (frames/byte per second) | -| | * Throughput provided to a VM (frames/byte per second) | -| | * Latency per traffic flow | -| | * Latency between VMs | -| | * Latency between NFVI nodes | -| | * Packet delay variation (jitter) between VMs | -| | * Packet delay variation (jitter) between NFVI nodes | -| | | -+---------+-------------------------------------------------------------------+ -| Storage | * Sequential read/write IOPS | -| | * Random read/write IOPS | -| | * Latency for storage read/write operations | -| | * Throughput for storage read/write operations | -| | | -+---------+-------------------------------------------------------------------+ - -.. _table2_2: - -**Table 2 - Capacity/Scale Metrics** - -+---------+-------------------------------------------------------------------+ -| Category| Capacity/Scale | -| | | -+---------+-------------------------------------------------------------------+ -| Compute | * Number of cores and threads- Available memory size | -| | * Cache size | -| | * Processor utilization (max, average, standard deviation) | -| | * Memory utilization (max, average, standard deviation) | -| | * Cache utilization (max, average, standard deviation) | -| | | -+---------+-------------------------------------------------------------------+ -| Network | * Number of connections | -| | * Number of frames sent/received | -| | * Maximum throughput between VMs (frames/byte per second) | -| | * Maximum throughput between NFVI nodes (frames/byte per second) | -| | * Network utilization (max, average, standard deviation) | -| | * Number of traffic flows | -| | | -+---------+-------------------------------------------------------------------+ -| Storage | * Storage/Disk size | -| | * Capacity allocation (block-based, object-based) | -| | * Block size | -| | * Maximum sequential read/write IOPS | -| | * Maximum random read/write IOPS | -| | * Disk utilization (max, average, standard deviation) | -| | | -+---------+-------------------------------------------------------------------+ - -.. _table2_3: - -**Table 3 - Availability/Reliability Metrics** - -+---------+-------------------------------------------------------------------+ -| Category| Availability/Reliability | -| | | -+---------+-------------------------------------------------------------------+ -| Compute | * Processor availability (Error free processing time) | -| | * Memory availability (Error free memory time) | -| | * Processor mean-time-to-failure | -| | * Memory mean-time-to-failure | -| | * Number of processing faults per second | -| | | -+---------+-------------------------------------------------------------------+ -| Network | * NIC availability (Error free connection time) | -| | * Link availability (Error free transmission time) | -| | * NIC mean-time-to-failure | -| | * Network timeout duration due to link failure | -| | * Frame loss rate | -| | | -+---------+-------------------------------------------------------------------+ -| Storage | * Disk availability (Error free disk access time) | -| | * Disk mean-time-to-failure | -| | * Number of failed storage read/write operations per second | -| | | -+---------+-------------------------------------------------------------------+ - -.. _table2_4: - -**Table 4 - Yardstick Generic Test Cases** - -+---------+-------------------+----------------+------------------------------+ -| Category| Performance/Speed | Capacity/Scale | Availability/Reliability | -| | | | | -+---------+-------------------+----------------+------------------------------+ -| Compute | TC003 | TC003 | TC013 [1]_ | -| | TC004 | TC004 | TC015 [1]_ | -| | TC014 | TC010 | | -| | TC024 | TC012 | | -| | | | | -+---------+-------------------+----------------+------------------------------+ -| Network | TC002 | TC001 | TC016 [1]_ | -| | TC011 | TC008 | TC018 [1]_ | -| | | TC009 | | -| | | | | -+---------+-------------------+----------------+------------------------------+ -| Storage | TC005 | TC005 | TC017 [1]_ | -| | | | | -+---------+-------------------+----------------+------------------------------+ - -.. note:: The description in this OPNFV document is intended as a reference for - users to understand the scope of the Yardstick Project and the - deliverables of the Yardstick framework. For complete description of - the methodology, refer to the ETSI document. - -.. rubric:: Footnotes -.. [1] To be included in future deliveries. diff --git a/docs/configguide/yardstick_testcases/03-list-of-tcs.rst b/docs/configguide/yardstick_testcases/03-list-of-tcs.rst deleted file mode 100644 index bb3fbbac5..000000000 --- a/docs/configguide/yardstick_testcases/03-list-of-tcs.rst +++ /dev/null @@ -1,91 +0,0 @@ -==================== -Yardstick Test Cases -==================== - -Abstract -======== - -This chapter lists available Yardstick test cases. -Yardstick test cases are divided in two main categories: - -* *Generic NFVI Test Cases* - Test Cases developed to realize the methodology -described in :doc:`02-methodology` - -* *OPNFV Feature Test Cases* - Test Cases developed to verify one or more -aspect of a feature delivered by an OPNFV Project, including the test cases -developed for the :term:`VTC`. - -Generic NFVI Test Case Descriptions -=================================== - -.. toctree:: - :maxdepth: 1 - - opnfv_yardstick_tc001.rst - opnfv_yardstick_tc002.rst - opnfv_yardstick_tc005.rst - opnfv_yardstick_tc008.rst - opnfv_yardstick_tc009.rst - opnfv_yardstick_tc010.rst - opnfv_yardstick_tc011.rst - opnfv_yardstick_tc012.rst - opnfv_yardstick_tc014.rst - opnfv_yardstick_tc024.rst - opnfv_yardstick_tc037.rst - opnfv_yardstick_tc038.rst - -OPNFV Feature Test Cases -======================== - -H A ---- - -.. toctree:: - :maxdepth: 1 - - opnfv_yardstick_tc019.rst - opnfv_yardstick_tc025.rst - -IPv6 ----- - -.. toctree:: - :maxdepth: 1 - - opnfv_yardstick_tc027.rst - -KVM ---- - -.. toctree:: - :maxdepth: 1 - - opnfv_yardstick_tc028.rst - -Parser ------- - -.. toctree:: - :maxdepth: 1 - - opnfv_yardstick_tc040.rst - -virtual Traffic Classifier --------------------------- - -.. toctree:: - :maxdepth: 1 - - opnfv_yardstick_tc006.rst - opnfv_yardstick_tc007.rst - opnfv_yardstick_tc020.rst - opnfv_yardstick_tc021.rst - -Templates -========= - -.. toctree:: - :maxdepth: 1 - - testcase_description_v2_template - Yardstick_task_templates diff --git a/docs/configguide/yardstick_testcases/04-vtc-overview.rst b/docs/configguide/yardstick_testcases/04-vtc-overview.rst deleted file mode 100644 index 95159a9bc..000000000 --- a/docs/configguide/yardstick_testcases/04-vtc-overview.rst +++ /dev/null @@ -1,114 +0,0 @@ -========================== -Virtual Traffic Classifier -========================== - -Abstract -======== - -.. _TNOVA: http://www.t-nova.eu/ -.. _TNOVAresults: http://www.t-nova.eu/results/ -.. _Yardstick: https://wiki.opnfv.org/yardstick - -This chapter provides an overview of the virtual Traffic Classifier, a -contribution to OPNFV Yardstick_ from the EU Project TNOVA_. -Additional documentation is available in TNOVAresults_. - -Overview -======== - -The virtual Traffic Classifier :term:`VNF`, the :term:`VTC`, comprises of a -:term:`VNFC`. The :term:`VNFC` contains both the Traffic Inspection module, and -the Traffic forwarding module, needed to run the VNF. The exploitation of -:term:`DPI` methods for traffic classification is built around two basic -assumptions: - -* third parties unaffiliated with either source or recipient are able to -inspect each IP packet’s payload - -* the classifier knows the relevant syntax of each application’s packet -payloads (protocol signatures, data patterns, etc.). - -The proposed :term:`DPI` based approach will only use an indicative, small -number of the initial packets from each flow in order to identify the content -and not inspect each packet. - -In this respect it follows the :term:`PBFS`. This method uses a table to track -each session based on the 5-tuples (src address, dest address, src port,dest -port, transport protocol) that is maintained for each flow. - -Concepts -======== - -* *Traffic Inspection*: The process of packet analysis and application -identification of network traffic that passes through the :term:`VTC`. - -* *Traffic Forwarding*: The process of packet forwarding from an incoming -network interface to a pre-defined outgoing network interface. - -* *Traffic Rule Application*: The process of packet tagging, based on a -predefined set of rules. Packet tagging may include e.g. :term:`ToS` field -modification. - -Architecture -============ - -The Traffic Inspection module is the most computationally intensive component -of the :term:`VNF`. It implements filtering and packet matching algorithms in -order to support the enhanced traffic forwarding capability of the :term:`VNF`. -The component supports a flow table (exploiting hashing algorithms for fast -indexing of flows) and an inspection engine for traffic classification. - -The implementation used for these experiments exploits the nDPI library. -The packet capturing mechanism is implemented using libpcap. When the -:term:`DPI` engine identifies a new flow, the flow register is updated with the -appropriate information and transmitted across the Traffic Forwarding module, -which then applies any required policy updates. - -The Traffic Forwarding moudle is responsible for routing and packet forwarding. -It accepts incoming network traffic, consults the flow table for classification -information for each incoming flow and then applies pre-defined policies -marking e.g. :term:`ToS`/:term:`DSCP` multimedia traffic for :term:`QoS` -enablement on the forwarded traffic. -It is assumed that the traffic is forwarded using the default policy until it -is identified and new policies are enforced. - -The expected response delay is considered to be negligible, as only a small -number of packets are required to identify each flow. - -Graphical Overview -================== - -.. code-block:: console - - +----------------------------+ - | | - | Virtual Traffic Classifier | - | | - | Analysing/Forwarding | - | ------------> | - | ethA ethB | - | | - +----------------------------+ - | ^ - | | - v | - +----------------------------+ - | | - | Virtual Switch | - | | - +----------------------------+ - -Install -======= - -run the build.sh with root privileges - -Run -=== - -sudo ./pfbridge -a eth1 -b eth2 - -Development Environment -======================= - -Ubuntu 14.04 diff --git a/docs/configguide/yardstick_testcases/Yardstick_task_templates.rst b/docs/configguide/yardstick_testcases/Yardstick_task_templates.rst deleted file mode 100755 index 8185062b2..000000000 --- a/docs/configguide/yardstick_testcases/Yardstick_task_templates.rst +++ /dev/null @@ -1,155 +0,0 @@ -Task Template Syntax -==================== - -Basic template syntax ---------------------- -A nice feature of the input task format used in Yardstick is that it supports -the template syntax based on Jinja2. -This turns out to be extremely useful when, say, you have a fixed structure of -your task but you want to parameterize this task in some way. -For example, imagine your input task file (task.yaml) runs a set of Ping -scenarios: - -:: - - # Sample benchmark task config file - # measure network latency using ping - schema: "yardstick:task:0.1" - - scenarios: - - - type: Ping - options: - packetsize: 200 - host: athena.demo - target: ares.demo - - runner: - type: Duration - duration: 60 - interval: 1 - - sla: - max_rtt: 10 - action: monitor - - context: - ... - -Let's say you want to run the same set of scenarios with the same runner/ -context/sla, but you want to try another packetsize to compare the performance. -The most elegant solution is then to turn the packetsize name into a template -variable: - -:: - - # Sample benchmark task config file - # measure network latency using ping - - schema: "yardstick:task:0.1" - scenarios: - - - type: Ping - options: - packetsize: {{packetsize}} - host: athena.demo - target: ares.demo - - runner: - type: Duration - duration: 60 - interval: 1 - - sla: - max_rtt: 10 - action: monitor - - context: - ... - -and then pass the argument value for {{packetsize}} when starting a task with -this configuration file. -Yardstick provides you with different ways to do that: - -1.Pass the argument values directly in the command-line interface (with either -a JSON or YAML dictionary): - -:: - - yardstick task start samples/ping-template.yaml - --task-args'{"packetsize":"200"}' - -2.Refer to a file that specifies the argument values (JSON/YAML): - -:: - - yardstick task start samples/ping-template.yaml --task-args-file args.yaml - -Using the default values ------------------------- -Note that the Jinja2 template syntax allows you to set the default values for -your parameters. -With default values set, your task file will work even if you don't -parameterize it explicitly while starting a task. -The default values should be set using the {% set ... %} clause (task.yaml). -For example: - -:: - - # Sample benchmark task config file - # measure network latency using ping - schema: "yardstick:task:0.1" - {% set packetsize = packetsize or "100" %} - scenarios: - - - type: Ping - options: - packetsize: {{packetsize}} - host: athena.demo - target: ares.demo - - runner: - type: Duration - duration: 60 - interval: 1 - ... - -If you don't pass the value for {{packetsize}} while starting a task, the -default one will be used. - -Advanced templates ------------------- - -Yardstick makes it possible to use all the power of Jinja2 template syntax, -including the mechanism of built-in functions. -As an example, let us make up a task file that will do a block storage -performance test. -The input task file (fio-template.yaml) below uses the Jinja2 for-endfor -construct to accomplish that: - -:: - - #Test block sizes of 4KB, 8KB, 64KB, 1MB - #Test 5 workloads: read, write, randwrite, randread, rw - schema: "yardstick:task:0.1" - - scenarios: - {% for bs in ['4k', '8k', '64k', '1024k' ] %} - {% for rw in ['read', 'write', 'randwrite', 'randread', 'rw' ] %} - - - type: Fio - options: - filename: /home/ubuntu/data.raw - bs: {{bs}} - rw: {{rw}} - ramp_time: 10 - host: fio.demo - runner: - type: Duration - duration: 60 - interval: 60 - - {% endfor %} - {% endfor %} - context - ... diff --git a/docs/configguide/yardstick_testcases/glossary.rst b/docs/configguide/yardstick_testcases/glossary.rst deleted file mode 100644 index 8ce9a6ba0..000000000 --- a/docs/configguide/yardstick_testcases/glossary.rst +++ /dev/null @@ -1,33 +0,0 @@ -================== -Yardstick Glossary -================== - -.. glossary:: - :sorted: - - DPI - Deep Packet Inspection - - DSCP - Differentiated Services Code Point - - PBFS - Packet Based per Flow State - - QoS - Quality of Service - - VNF - Virtual Network Function - - VNFC - Virtual Network Function Component - - NFVI - Network Function Virtualization Infrastructure - - ToS - Type of Service - - VTC - Virtual Traffic Classifier diff --git a/docs/configguide/yardstick_testcases/index.rst b/docs/configguide/yardstick_testcases/index.rst deleted file mode 100644 index 55d4ea3e1..000000000 --- a/docs/configguide/yardstick_testcases/index.rst +++ /dev/null @@ -1,12 +0,0 @@ -================== -Yardstick Overview -================== - -.. toctree:: - :maxdepth: 2 - - 01-introduction - 02-methodology - 04-vtc-overview - 03-list-of-tcs - glossary diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc001.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc001.rst deleted file mode 100644 index 810bad489..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc001.rst +++ /dev/null @@ -1,79 +0,0 @@ -************************************* -Yardstick Test Case Description TC001 -************************************* - -.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt - -+-----------------------------------------------------------------------------+ -|Network Performance | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC001_NW PERF | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Number of flows and throughput | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS network performance with regards to | -| | flows and throughput, such as if and how different amounts | -| | of flows matter for the throughput between hosts on different| -| | compute blades. Typically e.g. the performance of a vSwitch | -| | depends on the number of flows running through it. Also | -| | performance of other equipment or entities can depend | -| | on the number of flows or the packet sizes used. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc001.yaml | -| | | -| | Packet size: 60 bytes | -| | Number of ports: 10, 50, 100, 500 and 1000, where each | -| | runs for 20 seconds. The whole sequence is run | -| | twice. The client and server are distributed on different HW.| -| | For SLA max_ppm is set to 1000. The amount of configured | -| | ports map to between 110 up to 1001000 flows, respectively. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | pktgen | -| | | -| | (Pktgen is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Docker | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | pktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different packet sizes, amount | -| | of flows and test duration. Default values exist. | -| | | -| | SLA (optional): max_ppm: The number of packets per million | -| | packets sent that are acceptable to loose, not received. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with pktgen included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. pktgen is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc002.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc002.rst deleted file mode 100644 index 56350f5bb..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc002.rst +++ /dev/null @@ -1,78 +0,0 @@ -************************************* -Yardstick Test Case Description TC002 -************************************* - -.. _cirros-image: https://download.cirros-cloud.net - -+-----------------------------------------------------------------------------+ -|Network Latency | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC002_NW LATENCY | -| | | -+--------------+--------------------------------------------------------------+ -|metric | RTT, Round Trip Time | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To do a basic verification that network latency is within | -| | acceptable boundaries when packets travel between hosts | -| | located on same or different compute blades. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs and similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc002.yaml | -| | | -| | Packet size 100 bytes. Total test duration 600 seconds. | -| | One ping each 10 seconds. SLA RTT is set to maximum 10 ms. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | ping | -| | | -| | Ping is normally part of any Linux distribution, hence it | -| | doesn't need to be installed. It is also part of the | -| | Yardstick Docker image. | -| | (For example also a Cirros image can be downloaded from | -| | cirros-image_, it includes ping) | -| | | -+--------------+--------------------------------------------------------------+ -|references | Ping man page | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test case can be configured with different packet sizes, | -| | burst sizes, ping intervals and test duration. | -| | SLA is optional. The SLA in this test case serves as an | -| | example. Considerably lower RTT is expected, and | -| | also normal to achieve in balanced L2 environments. However, | -| | to cover most configurations, both bare metal and fully | -| | virtualized ones, this value should be possible to achieve | -| | and acceptable for black box testing. Many real time | -| | applications start to suffer badly if the RTT time is higher | -| | than this. Some may suffer bad also close to this RTT, while | -| | others may not suffer at all. It is a compromise that may | -| | have to be tuned for different configuration purposes. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with ping included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. Ping is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Test should not PASS if any RTT is above the optional SLA | -| | value, or if there is a test case execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc005.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc005.rst deleted file mode 100644 index 83b6aedd6..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc005.rst +++ /dev/null @@ -1,72 +0,0 @@ -************************************* -Yardstick Test Case Description TC005 -************************************* - -.. _fio: http://www.bluestop.org/fio/HOWTO.txt - -+-----------------------------------------------------------------------------+ -|Storage Performance | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC005_Storage Performance | -| | | -+--------------+--------------------------------------------------------------+ -|metric | IOPS, throughput and latency | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS storage performance with regards to | -| | IOPS, throughput and latency. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs and similar shall be stored for comparison reasons | -| | and product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc005.yaml | -| | | -| | IO types: read, write, randwrite, randread, rw | -| | IO block size: 4KB, 64KB, 1024KB, where each | -| | runs for 30 seconds(10 for ramp time, 20 for runtime). | -| | | -| | For SLA minimum read/write iops is set to 100, minimum | -| | read/write throughput is set to 400 KB/s, and maximum | -| | read/write latency is set to 20000 usec. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | fio | -| | | -| | (fio is not always part of a Linux distribution, hence it | -| | needs to be installed. As an example see the | -| | /yardstick/tools/ directory for how to generate a Linux | -| | image with fio included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | fio_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different read/write types, IO | -| | block size, IO depth, ramp time (runtime required for stable | -| | results) and test duration. Default values exist. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with fio included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The host is installed and fio is invoked and logs are | -| | produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc006.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc006.rst deleted file mode 100644 index b68315078..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc006.rst +++ /dev/null @@ -1,139 +0,0 @@ -************************************* -Yardstick Test Case Description TC006 -************************************* - -.. _DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/ -.. _rfc2544: https://www.ietf.org/rfc/rfc2544.txt - -+-----------------------------------------------------------------------------+ -|Network Performance | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC006_Virtual Traffic Classifier Data Plane | -| | Throughput Benchmarking Test. | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Throughput | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To measure the throughput supported by the virtual Traffic | -| | Classifier according to the RFC2544 methodology for a | -| | user-defined set of vTC deployment configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: file: opnfv_yardstick_tc006.yaml | -| | | -| | packet_size: size of the packets to be used during the | -| | throughput calculation. | -| | Allowe values: [64, 128, 256, 512, 1024, 1280, 1518] | -| | | -| | vnic_type: type of VNIC to be used. | -| | Allowed values are: | -| | - normal: for default OvS port configuration | -| | - direct: for SR-IOV port configuration | -| | Default value: None | -| | | -| | vtc_flavor: OpenStack flavor to be used for the vTC | -| | Default available values are: m1.small, m1.medium, | -| | and m1.large, but the user can create his/her own | -| | flavor and give it as input | -| | Default value: None | -| | | -| | vlan_sender: vlan tag of the network on which the vTC will | -| | receive traffic (VLAN Network 1). | -| | Allowed values: range (1, 4096) | -| | | -| | vlan_receiver: vlan tag of the network on which the vTC | -| | will send traffic back to the packet generator | -| | (VLAN Network 2). | -| | Allowed values: range (1, 4096) | -| | | -| | default_net_name: neutron name of the defaul network that | -| | is used for access to the internet from the vTC | -| | (vNIC 1). | -| | | -| | default_subnet_name: subnet name for vNIC1 | -| | (information available through Neutron). | -| | | -| | vlan_net_1_name: Neutron Name for VLAN Network 1 | -| | (information available through Neutron). | -| | | -| | vlan_subnet_1_name: Subnet Neutron name for VLAN Network 1 | -| | (information available through Neutron). | -| | | -| | vlan_net_2_name: Neutron Name for VLAN Network 2 | -| | (information available through Neutron). | -| | | -| | vlan_subnet_2_name: Subnet Neutron name for VLAN Network 2 | -| | (information available through Neutron). | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | DPDK pktgen | -| | | -| | DPDK Pktgen is not part of a Linux distribution, | -| | hence it needs to be installed by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|references | DPDK Pktgen: DPDKpktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -| | RFC 2544: rfc2544_ | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different flavors, vNIC type | -| | and packet sizes. Default values exist as specified above. | -| | The vNIC type and flavor MUST be specified by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The vTC has been successfully instantiated and configured. | -| | The user has correctly assigned the values to the deployment | -| | configuration parameters. | -| | | -| | - Multicast traffic MUST be enabled on the network. | -| | The Data network switches need to be configured in | -| | order to manage multicast traffic. | -| | - In the case of SR-IOV vNICs use, SR-IOV compatible NICs | -| | must be used on the compute node. | -| | - Yarsdtick needs to be installed on a host connected to the | -| | data network and the host must have 2 DPDK-compatible | -| | NICs. Proper configuration of DPDK and DPDK pktgen is | -| | required before to run the test case. | -| | (For further instructions please refer to the ApexLake | -| | documentation). | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | Description and expected results | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The vTC is deployed, according to the user-defined | -| | configuration | -| | | -+--------------+--------------------------------------------------------------+ -|step 2 | The vTC is correctly deployed and configured as necessary | -| | The initialization script has been correctly executed and | -| | vTC is ready to receive and process the traffic. | -| | | -+--------------+--------------------------------------------------------------+ -|step 3 | Test case is executed with the selected parameters: | -| | - vTC flavor | -| | - vNIC type | -| | - packet size | -| | The traffic is sent to the vTC using the maximum available | -| | traffic rate for 60 seconds. | -| | | -+--------------+--------------------------------------------------------------+ -|step 4 | The vTC instance forwards all the packets back to the packet | -| | generator for 60 seconds, as specified by RFC 2544. | -| | | -| | Steps 3 and 4 are executed different times, with different | -| | rates in order to find the maximum supported traffic rate | -| | according to the current definition of throughput in RFC | -| | 2544. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | The result of the test is a number between 0 and 100 which | -| | represents the throughput in terms of percentage of the | -| | available pktgen NIC bandwidth. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc007.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc007.rst deleted file mode 100644 index a7a4776d5..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc007.rst +++ /dev/null @@ -1,157 +0,0 @@ -************************************* -Yardstick Test Case Description TC007 -************************************* - -.. _DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/ -.. _rfc2544: https://www.ietf.org/rfc/rfc2544.txt - -+-----------------------------------------------------------------------------+ -|Network Performance | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC007_Virtual Traffic Classifier Data Plane | -| | Throughput Benchmarking Test in Presence of Noisy | -| | neighbours | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Throughput | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To measure the throughput supported by the virtual Traffic | -| | Classifier according to the RFC2544 methodology for a | -| | user-defined set of vTC deployment configurations in the | -| | presence of noisy neighbours. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc007.yaml | -| | | -| | packet_size: size of the packets to be used during the | -| | throughput calculation. | -| | Allowe values: [64, 128, 256, 512, 1024, 1280, 1518] | -| | | -| | vnic_type: type of VNIC to be used. | -| | Allowed values are: | -| | - normal: for default OvS port configuration | -| | - direct: for SR-IOV port configuration | -| | | -| | vtc_flavor: OpenStack flavor to be used for the vTC | -| | Default available values are: m1.small, m1.medium, | -| | and m1.large, but the user can create his/her own | -| | flavor and give it as input | -| | | -| | num_of_neighbours: Number of noisy neighbours (VMs) to be | -| | instantiated during the experiment. | -| | Allowed values: range (1, 10) | -| | | -| | amount_of_ram: RAM to be used by each neighbor. | -| | Allowed values: ['250M', '1G', '2G', '3G', '4G', '5G', | -| | '6G', '7G', '8G', '9G', '10G'] | -| | Deault value: 256M | -| | | -| | number_of_cores: Number of noisy neighbours (VMs) to be | -| | instantiated during the experiment. | -| | Allowed values: range (1, 10) | -| | Default value: 1 | -| | | -| | vlan_sender: vlan tag of the network on which the vTC will | -| | receive traffic (VLAN Network 1). | -| | Allowed values: range (1, 4096) | -| | | -| | vlan_receiver: vlan tag of the network on which the vTC | -| | will send traffic back to the packet generator | -| | (VLAN Network 2). | -| | Allowed values: range (1, 4096) | -| | | -| | default_net_name: neutron name of the defaul network that | -| | is used for access to the internet from the vTC | -| | (vNIC 1). | -| | | -| | default_subnet_name: subnet name for vNIC1 | -| | (information available through Neutron). | -| | | -| | vlan_net_1_name: Neutron Name for VLAN Network 1 | -| | (information available through Neutron). | -| | | -| | vlan_subnet_1_name: Subnet Neutron name for VLAN Network 1 | -| | (information available through Neutron). | -| | | -| | vlan_net_2_name: Neutron Name for VLAN Network 2 | -| | (information available through Neutron). | -| | | -| | vlan_subnet_2_name: Subnet Neutron name for VLAN Network 2 | -| | (information available through Neutron). | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | DPDK pktgen | -| | | -| | DPDK Pktgen is not part of a Linux distribution, | -| | hence it needs to be installed by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|references | DPDKpktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -| | rfc2544_ | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different flavors, vNIC type | -| | and packet sizes. Default values exist as specified above. | -| | The vNIC type and flavor MUST be specified by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The vTC has been successfully instantiated and configured. | -| | The user has correctly assigned the values to the deployment | -| | configuration parameters. | -| | | -| | - Multicast traffic MUST be enabled on the network. | -| | The Data network switches need to be configured in | -| | order to manage multicast traffic. | -| | - In the case of SR-IOV vNICs use, SR-IOV compatible NICs | -| | must be used on the compute node. | -| | - Yarsdtick needs to be installed on a host connected to the | -| | data network and the host must have 2 DPDK-compatible | -| | NICs. Proper configuration of DPDK and DPDK pktgen is | -| | required before to run the test case. | -| | (For further instructions please refer to the ApexLake | -| | documentation). | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | Description and expected results | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The noisy neighbours are deployed as required by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|step 2 | The vTC is deployed, according to the configuration required | -| | by the user | -| | | -+--------------+--------------------------------------------------------------+ -|step 3 | The vTC is correctly deployed and configured as necessary. | -| | The initialization script has been correctly executed and | -| | the vTC is ready to receive and process the traffic. | -| | | -+--------------+--------------------------------------------------------------+ -|step 4 | Test case is executed with the parameters specified by the | -| | user: | -| | - vTC flavor | -| | - vNIC type | -| | - packet size | -| | The traffic is sent to the vTC using the maximum available | -| | traffic rate | -| | | -+--------------+--------------------------------------------------------------+ -|step 5 | The vTC instance forwards all the packets back to the | -| | packet generator for 60 seconds, as specified by RFC 2544. | -| | | -| | Steps 4 and 5 are executed different times with different | -| | with different traffic rates, in order to find the maximum | -| | supported traffic rate, accoring to the current definition | -| | of throughput in RFC 2544. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | The result of the test is a number between 0 and 100 which | -| | represents the throughput in terms of percentage of the | -| | available pktgen NIC bandwidth. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc008.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc008.rst deleted file mode 100644 index e176e633a..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc008.rst +++ /dev/null @@ -1,85 +0,0 @@ -************************************* -Yardstick Test Case Description TC008 -************************************* - -.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt - -+-----------------------------------------------------------------------------+ -|Packet Loss Extended Test | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC008_NW PERF, Packet loss Extended Test | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Number of flows, packet size and throughput | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS network performance with regards to | -| | flows and throughput, such as if and how different amounts | -| | of packet sizes and flows matter for the throughput between | -| | VMs on different compute blades. Typically e.g. the | -| | performance of a vSwitch | -| | depends on the number of flows running through it. Also | -| | performance of other equipment or entities can depend | -| | on the number of flows or the packet sizes used. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc008.yaml | -| | | -| | Packet size: 64, 128, 256, 512, 1024, 1280 and 1518 bytes. | -| | | -| | Number of ports: 1, 10, 50, 100, 500 and 1000. The amount of | -| | configured ports map from 2 up to 1001000 flows, | -| | respectively. Each packet_size/port_amount combination is run| -| | ten times, for 20 seconds each. Then the next | -| | packet_size/port_amount combination is run, and so on. | -| | | -| | The client and server are distributed on different HW. | -| | | -| | For SLA max_ppm is set to 1000. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | pktgen | -| | | -| | (Pktgen is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Docker | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | pktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different packet sizes, amount | -| | of flows and test duration. Default values exist. | -| | | -| | SLA (optional): max_ppm: The number of packets per million | -| | packets sent that are acceptable to loose, not received. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with pktgen included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. pktgen is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc009.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc009.rst deleted file mode 100644 index e4002a884..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc009.rst +++ /dev/null @@ -1,84 +0,0 @@ -************************************* -Yardstick Test Case Description TC009 -************************************* - -.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt - -+-----------------------------------------------------------------------------+ -|Packet Loss | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC009_NW PERF, Packet loss | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Number of flows, packets lost and throughput | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS network performance with regards to | -| | flows and throughput, such as if and how different amounts | -| | of flows matter for the throughput between VMs on different | -| | compute blades. | -| | Typically e.g. the performance of a vSwitch | -| | depends on the number of flows running through it. Also | -| | performance of other equipment or entities can depend | -| | on the number of flows or the packet sizes used. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc009.yaml | -| | | -| | Packet size: 64 bytes | -| | | -| | Number of ports: 1, 10, 50, 100, 500 and 1000. The amount of | -| | configured ports map from 2 up to 1001000 flows, | -| | respectively. Each port amount is run ten times, for 20 | -| | seconds each. Then the next port_amount is run, and so on. | -| | | -| | The client and server are distributed on different HW. | -| | | -| | For SLA max_ppm is set to 1000. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | pktgen | -| | | -| | (Pktgen is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Docker | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | pktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different packet sizes, amount | -| | of flows and test duration. Default values exist. | -| | | -| | SLA (optional): max_ppm: The number of packets per million | -| | packets sent that are acceptable to loose, not received. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with pktgen included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. pktgen is | -| | invoked and logs are produced and stored. | -| | | -| | Result: logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc010.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc010.rst deleted file mode 100644 index ebb74ea30..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc010.rst +++ /dev/null @@ -1,77 +0,0 @@ -************************************* -Yardstick Test Case Description TC010 -************************************* - -.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/lat_mem_rd.8.html - -+-----------------------------------------------------------------------------+ -|Memory Latency | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC010_Memory Latency | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Latency in nanoseconds | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | Measure the memory read latency for varying memory sizes and | -| | strides. Whole memory hierarchy is measured including all | -| | levels of cache. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | File: opnfv_yardstick_tc010.yaml | -| | | -| | * SLA (max_latency): 30 nanoseconds | -| | * Stride - 128 bytes | -| | * Stop size - 64 megabytes | -| | * Iterations: 10 - test is run 10 times iteratively. | -| | * Interval: 1 - there is 1 second delay between each | -| | iteration. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | Lmbench | -| | | -| | Lmbench is a suite of operating system microbenchmarks. This | -| | test uses lat_mem_rd tool from that suite. | -| | Lmbench is not always part of a Linux distribution, hence it | -| | needs to be installed in the test image | -| | | -+--------------+--------------------------------------------------------------+ -|references | man-pages_ | -| | | -| | McVoy, Larry W.,and Carl Staelin. "lmbench: Portable Tools | -| | for Performance Analysis." USENIX annual technical | -| | conference 1996. | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different: | -| | | -| | * strides; | -| | * stop_size; | -| | * iterations and intervals. | -| | | -| | There are default values for each above-mentioned option. | -| | | -| | SLA (optional) : max_latency: The maximum memory latency | -| | that is accepted. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with Lmbench included in the image. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The host is installed as client. Lmbench's lat_mem_rd tool | -| | is invoked and logs are produced and stored. | -| | | -| | Result: logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Test fails if the measured memory latency is above the SLA | -| | value or if there is a test case execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc011.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc011.rst deleted file mode 100644 index 6760ce067..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc011.rst +++ /dev/null @@ -1,83 +0,0 @@ -************************************* -Yardstick Test Case Description TC011 -************************************* - -.. _iperf3: https://iperf.fr/ - -+-----------------------------------------------------------------------------+ -|Packet delay variation between VMs | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC011_Packet delay variation between VMs | -| | | -+--------------+--------------------------------------------------------------+ -|metric | jitter: packet delay variation (ms) | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | Measure the packet delay variation sending the packets from | -| | one VM to the other. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | File: opnfv_yardstick_tc011.yaml | -| | | -| | * options: | -| | protocol: udp # The protocol used by iperf3 tools | -| | bandwidth: 20m # It will send the given number of packets | -| | without pausing | -| | * runner: | -| | duration: 30 # Total test duration 30 seconds. | -| | | -| | * SLA (optional): | -| | jitter: 10 (ms) # The maximum amount of jitter that is | -| | accepted. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | iperf3 | -| | | -| | iPerf3 is a tool for active measurements of the maximum | -| | achievable bandwidth on IP networks. It supports tuning of | -| | various parameters related to timing, buffers and protocols. | -| | The UDP protocols can be used to measure jitter delay. | -| | | -| | (iperf3 is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Docker | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | iperf3_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different | -| | | -| | * bandwidth: Test case can be configured with different | -| | bandwidth | -| | | -| | * duration: The test duration can be configured | -| | | -| | * jitter: SLA is optional. The SLA in this test case | -| | serves as an example. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with iperf3 included in the image. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. iperf3 is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Test should not PASS if any jitter is above the optional SLA | -| | value, or if there is a test case execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc012.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc012.rst deleted file mode 100644 index e7889c14e..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc012.rst +++ /dev/null @@ -1,81 +0,0 @@ -************************************* -Yardstick Test Case Description TC012 -************************************* - -.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/bw_mem.8.html - -+-----------------------------------------------------------------------------+ -|Memory Bandwidth | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC012_Memory Bandwidth | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Megabyte per second (MBps) | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | Measure the rate at which data can be read from and written | -| | to the memory (this includes all levels of memory). | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | File: opnfv_yardstick_tc012.yaml | -| | | -| | * SLA (optional): 15000 (MBps) min_bw: The minimum amount of | -| | memory bandwidth that is accepted. | -| | * Size: 10 240 kB - test allocates twice that size (20 480kB)| -| | zeros it and then measures the time it takes to copy from | -| | one side to another. | -| | * Benchmark: rdwr - measures the time to read data into | -| | memory and then write data to the same location. | -| | * Warmup: 0 - the number of iterations to perform before | -| | taking actual measurements. | -| | * Iterations: 10 - test is run 10 times iteratively. | -| | * Interval: 1 - there is 1 second delay between each | -| | iteration. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | Lmbench | -| | | -| | Lmbench is a suite of operating system microbenchmarks. This | -| | test uses bw_mem tool from that suite. | -| | Lmbench is not always part of a Linux distribution, hence it | -| | needs to be installed in the test image. | -| | | -+--------------+--------------------------------------------------------------+ -|references | man-pages_ | -| | | -| | McVoy, Larry W., and Carl Staelin. "lmbench: Portable Tools | -| | for Performance Analysis." USENIX annual technical | -| | conference. 1996. | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different: | -| | | -| | * memory sizes; | -| | * memory operations (such as rd, wr, rdwr, cp, frd, fwr, | -| | fcp, bzero, bcopy); | -| | * number of warmup iterations; | -| | * iterations and intervals. | -| | | -| | There are default values for each above-mentioned option. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with Lmbench included in the image. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The host is installed as client. Lmbench's bw_mem tool is | -| | invoked and logs are produced and stored. | -| | | -| | Result: logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Test fails if the measured memory bandwidth is below the SLA | -| | value or if there is a test case execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc014.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc014.rst deleted file mode 100644 index 68d36ecd2..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc014.rst +++ /dev/null @@ -1,69 +0,0 @@ -************************************* -Yardstick Test Case Description TC014 -************************************* - -.. _unixbench: https://github.com/kdlucas/byte-unixbench/blob/master/UnixBench - -+-----------------------------------------------------------------------------+ -|Processing speed | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC014_Processing speed | -| | | -+--------------+--------------------------------------------------------------+ -|metric | score of single cpu running, score of parallel running | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS processing speed with regards to score | -| | of single cpu running and parallel running | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs and similar shall be stored for comparison reasons | -| | and product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc014.yaml | -| | | -| | run_mode: Run unixbench in quiet mode or verbose mode | -| | test_type: dhry2reg, whetstone and so on | -| | | -| | For SLA with single_score and parallel_score, both can be | -| | set by user, default is NA | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | unixbench | -| | | -| | (unixbench is not always part of a Linux distribution, hence | -| | it needs to be installed. As an example see the | -| | /yardstick/tools/ directory for how to generate a Linux | -| | image with unixbench included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | unixbench_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different test types, dhry2reg, | -| | whetstone and so on. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with unixbench included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as a client. unixbench is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc019.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc019.rst deleted file mode 100644 index 482260b48..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc019.rst +++ /dev/null @@ -1,129 +0,0 @@ -************************************* -Yardstick Test Case Description TC019 -************************************* - -+-----------------------------------------------------------------------------+ -|Control Node Openstack Service High Availability | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC019_HA: Control node Openstack service down| -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | This test case will verify the high availability of the | -| | service provided by OpenStack (like nova-api, neutro-server) | -| | on control node. | -| | | -+--------------+--------------------------------------------------------------+ -|test method | This test case kills the processes of a specific Openstack | -| | service on a selected control node, then checks whether the | -| | request of the related Openstack command is OK and the killed| -| | processes are recovered. | -| | | -+--------------+--------------------------------------------------------------+ -|attackers | In this test case, an attacker called "kill-process" is | -| | needed. This attacker includes three parameters: | -| | 1) fault_type: which is used for finding the attacker's | -| | scripts. It should be always set to "kill-process" in this | -| | test case. | -| | 2) process_name: which is the process name of the specified | -| | OpenStack service. If there are multiple processes use the | -| | same name on the host, all of them are killed by this | -| | attacker. | -| | 3) host: which is the name of a control node being attacked. | -| | | -| | e.g. | -| | -fault_type: "kill-process" | -| | -process_name: "nova-api" | -| | -host: node1 | -| | | -+--------------+--------------------------------------------------------------+ -|monitors | In this test case, two kinds of monitor are needed: | -| | 1. the "openstack-cmd" monitor constantly request a specific | -| | Openstack command, which needs two parameters: | -| | 1) monitor_type: which is used for finding the monitor class | -| | and related scritps. It should be always set to | -| | "openstack-cmd" for this monitor. | -| | 2) command_name: which is the command name used for request | -| | | -| | 2. the "process" monitor check whether a process is running | -| | on a specific node, which needs three parameters: | -| | 1) monitor_type: which used for finding the monitor class and| -| | related scritps. It should be always set to "process" | -| | for this monitor. | -| | 2) process_name: which is the process name for monitor | -| | 3) host: which is the name of the node runing the process | -| | | -| | e.g. | -| | monitor1: | -| | -monitor_type: "openstack-cmd" | -| | -command_name: "nova image-list" | -| | monitor2: | -| | -monitor_type: "process" | -| | -process_name: "nova-api" | -| | -host: node1 | -| | | -+--------------+--------------------------------------------------------------+ -|metrics | In this test case, there are two metrics: | -| | 1)service_outage_time: which indicates the maximum outage | -| | time (seconds) of the specified Openstack command request. | -| | 2)process_recover_time: which indicates the maximun time | -| | (seconds) from the process being killed to recovered | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | Developed by the project. Please see folder: | -| | "yardstick/benchmark/scenarios/availability/ha_tools" | -| | | -+--------------+--------------------------------------------------------------+ -|references | ETSI NFV REL001 | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | This test case needs two configuration files: | -| | 1) test case file: opnfv_yardstick_tc019.yaml | -| | -Attackers: see above "attackers" discription | -| | -waiting_time: which is the time (seconds) from the process | -| | being killed to stoping monitors the monitors | -| | -Monitors: see above "monitors" discription | -| | -SLA: see above "metrics" discription | -| | | -| | 2)POD file: pod.yaml | -| | The POD configuration should record on pod.yaml first. | -| | the "host" item in this test case will use the node name in | -| | the pod.yaml. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | start monitors: | -| | each monitor will run with independently process | -| | | -| | Result: The monitor info will be collected. | -| | | -+--------------+--------------------------------------------------------------+ -|step 2 | do attacker: connect the host through SSH, and then execute | -| | the kill process script with param value specified by | -| | "process_name" | -| | | -| | Result: Process will be killed. | -| | | -+--------------+--------------------------------------------------------------+ -|step 3 | stop monitors after a period of time specified by | -| | "waiting_time" | -| | | -| | Result: The monitor info will be aggregated. | -| | | -+--------------+--------------------------------------------------------------+ -|step 4 | verify the SLA | -| | | -| | Result: The test case is passed or not. | -| | | -+--------------+--------------------------------------------------------------+ -|post-action | It is the action when the test cases exist. It will check the| -| | status of the specified process on the host, and restart the | -| | process if it is not running for next test cases | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc020.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc020.rst deleted file mode 100644 index 9a5130f71..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc020.rst +++ /dev/null @@ -1,136 +0,0 @@ -************************************* -Yardstick Test Case Description TC020 -************************************* - -.. _DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/ -.. _rfc2544: https://www.ietf.org/rfc/rfc2544.txt - -+-----------------------------------------------------------------------------+ -|Network Performance | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC0020_Virtual Traffic Classifier | -| | Instantiation Test | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Failure | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To verify that a newly instantiated vTC is 'alive' and | -| | functional and its instantiation is correctly supported by | -| | the infrastructure. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc020.yaml | -| | | -| | vnic_type: type of VNIC to be used. | -| | Allowed values are: | -| | - normal: for default OvS port configuration | -| | - direct: for SR-IOV port configuration | -| | Default value: None | -| | | -| | vtc_flavor: OpenStack flavor to be used for the vTC | -| | Default available values are: m1.small, m1.medium, | -| | and m1.large, but the user can create his/her own | -| | flavor and give it as input | -| | Default value: None | -| | | -| | vlan_sender: vlan tag of the network on which the vTC will | -| | receive traffic (VLAN Network 1). | -| | Allowed values: range (1, 4096) | -| | | -| | vlan_receiver: vlan tag of the network on which the vTC | -| | will send traffic back to the packet generator | -| | (VLAN Network 2). | -| | Allowed values: range (1, 4096) | -| | | -| | default_net_name: neutron name of the defaul network that | -| | is used for access to the internet from the vTC | -| | (vNIC 1). | -| | | -| | default_subnet_name: subnet name for vNIC1 | -| | (information available through Neutron). | -| | | -| | vlan_net_1_name: Neutron Name for VLAN Network 1 | -| | (information available through Neutron). | -| | | -| | vlan_subnet_1_name: Subnet Neutron name for VLAN Network 1 | -| | (information available through Neutron). | -| | | -| | vlan_net_2_name: Neutron Name for VLAN Network 2 | -| | (information available through Neutron). | -| | | -| | vlan_subnet_2_name: Subnet Neutron name for VLAN Network 2 | -| | (information available through Neutron). | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | DPDK pktgen | -| | | -| | DPDK Pktgen is not part of a Linux distribution, | -| | hence it needs to be installed by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|references | DPDKpktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -| | rfc2544_ | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different flavors, vNIC type | -| | and packet sizes. Default values exist as specified above. | -| | The vNIC type and flavor MUST be specified by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The vTC has been successfully instantiated and configured. | -| | The user has correctly assigned the values to the deployment | -| | configuration parameters. | -| | | -| | - Multicast traffic MUST be enabled on the network. | -| | The Data network switches need to be configured in | -| | order to manage multicast traffic. | -| | Installation and configuration of smcroute is required | -| | before to run the test case. | -| | (For further instructions please refer to the ApexLake | -| | documentation). | -| | - In the case of SR-IOV vNICs use, SR-IOV compatible NICs | -| | must be used on the compute node. | -| | - Yarsdtick needs to be installed on a host connected to the | -| | data network and the host must have 2 DPDK-compatible | -| | NICs. Proper configuration of DPDK and DPDK pktgen is | -| | required before to run the test case. | -| | (For further instructions please refer to the ApexLake | -| | documentation). | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | Description and expected results | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The vTC is deployed, according to the configuration provided | -| | by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|step 2 | The vTC is correctly deployed and configured as necessary. | -| | The initialization script has been correctly executed and | -| | the vTC is ready to receive and process the traffic. | -| | | -+--------------+--------------------------------------------------------------+ -|step 3 | Test case is executed with the parameters specified by the | -| | the user: | -| | - vTC flavor | -| | - vNIC type | -| | A constant rate traffic is sent to the vTC for 10 seconds. | -| | | -+--------------+--------------------------------------------------------------+ -|step 4 | The vTC instance tags all the packets and sends them back to | -| | the packet generator for 10 seconds. | -| | | -| | The framework checks that the packet generator receives | -| | back all the packets with the correct tag from the vTC. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | The vTC is deemed to be successfully instantiated if all | -| | packets are sent back with the right tag as requested, | -| | else it is deemed DoA (Dead on arrival) | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc021.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc021.rst deleted file mode 100644 index a493ddfc0..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc021.rst +++ /dev/null @@ -1,152 +0,0 @@ -************************************* -Yardstick Test Case Description TC021 -************************************* - -.. _DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/ -.. _rfc2544: https://www.ietf.org/rfc/rfc2544.txt - -+-----------------------------------------------------------------------------+ -|Network Performance | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC0021_Virtual Traffic Classifier | -| | Instantiation Test in Presence of Noisy Neighbours | -| | | -+--------------+--------------------------------------------------------------+ -|metric | Failure | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To verify that a newly instantiated vTC is 'alive' and | -| | functional and its instantiation is correctly supported by | -| | the infrastructure in the presence of noisy neighbours. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc021.yaml | -| | | -| | vnic_type: type of VNIC to be used. | -| | Allowed values are: | -| | - normal: for default OvS port configuration | -| | - direct: for SR-IOV port configuration | -| | Default value: None | -| | | -| | vtc_flavor: OpenStack flavor to be used for the vTC | -| | Default available values are: m1.small, m1.medium, | -| | and m1.large, but the user can create his/her own | -| | flavor and give it as input | -| | Default value: None | -| | | -| | num_of_neighbours: Number of noisy neighbours (VMs) to be | -| | instantiated during the experiment. | -| | Allowed values: range (1, 10) | -| | | -| | amount_of_ram: RAM to be used by each neighbor. | -| | Allowed values: ['250M', '1G', '2G', '3G', '4G', '5G', | -| | '6G', '7G', '8G', '9G', '10G'] | -| | Deault value: 256M | -| | | -| | number_of_cores: Number of noisy neighbours (VMs) to be | -| | instantiated during the experiment. | -| | Allowed values: range (1, 10) | -| | Default value: 1 | -| | | -| | vlan_sender: vlan tag of the network on which the vTC will | -| | receive traffic (VLAN Network 1). | -| | Allowed values: range (1, 4096) | -| | | -| | vlan_receiver: vlan tag of the network on which the vTC | -| | will send traffic back to the packet generator | -| | (VLAN Network 2). | -| | Allowed values: range (1, 4096) | -| | | -| | default_net_name: neutron name of the defaul network that | -| | is used for access to the internet from the vTC | -| | (vNIC 1). | -| | | -| | default_subnet_name: subnet name for vNIC1 | -| | (information available through Neutron). | -| | | -| | vlan_net_1_name: Neutron Name for VLAN Network 1 | -| | (information available through Neutron). | -| | | -| | vlan_subnet_1_name: Subnet Neutron name for VLAN Network 1 | -| | (information available through Neutron). | -| | | -| | vlan_net_2_name: Neutron Name for VLAN Network 2 | -| | (information available through Neutron). | -| | | -| | vlan_subnet_2_name: Subnet Neutron name for VLAN Network 2 | -| | (information available through Neutron). | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | DPDK pktgen | -| | | -| | DPDK Pktgen is not part of a Linux distribution, | -| | hence it needs to be installed by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|references | DPDK Pktgen: DPDK Pktgen: DPDKpktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -| | RFC 2544: rfc2544_ | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different flavors, vNIC type | -| | and packet sizes. Default values exist as specified above. | -| | The vNIC type and flavor MUST be specified by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The vTC has been successfully instantiated and configured. | -| | The user has correctly assigned the values to the deployment | -| | configuration parameters. | -| | | -| | - Multicast traffic MUST be enabled on the network. | -| | The Data network switches need to be configured in | -| | order to manage multicast traffic. | -| | Installation and configuration of smcroute is required | -| | before to run the test case. | -| | (For further instructions please refer to the ApexLake | -| | documentation). | -| | - In the case of SR-IOV vNICs use, SR-IOV compatible NICs | -| | must be used on the compute node. | -| | - Yarsdtick needs to be installed on a host connected to the | -| | data network and the host must have 2 DPDK-compatible | -| | NICs. Proper configuration of DPDK and DPDK pktgen is | -| | required before to run the test case. | -| | (For further instructions please refer to the ApexLake | -| | documentation). | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | Description and expected results | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The noisy neighbours are deployed as required by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|step 2 | The vTC is deployed, according to the configuration provided | -| | by the user. | -| | | -+--------------+--------------------------------------------------------------+ -|step 3 | The vTC is correctly deployed and configured as necessary. | -| | The initialization script has been correctly executed and | -| | the vTC is ready to receive and process the traffic. | -| | | -+--------------+--------------------------------------------------------------+ -|step 4 | Test case is executed with the selected parameters: | -| | - vTC flavor | -| | - vNIC type | -| | A constant rate traffic is sent to the vTC for 10 seconds. | -| | | -+--------------+--------------------------------------------------------------+ -|step 5 | The vTC instance tags all the packets and sends them back to | -| | the packet generator for 10 seconds. | -| | | -| | The framework checks if the packet generator receives back | -| | all the packets with the correct tag from the vTC. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | The vTC is deemed to be successfully instantiated if all | -| | packets are sent back with the right tag as requested, | -| | else it is deemed DoA (Dead on arrival) | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc024.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc024.rst deleted file mode 100644 index 1e8d4e6f3..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc024.rst +++ /dev/null @@ -1,64 +0,0 @@ -************************************* -Yardstick Test Case Description TC024 -************************************* - -.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/man1/mpstat.1.html - -+-----------------------------------------------------------------------------+ -| CPU Load | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC024_CPU Load | -| | | -+--------------+--------------------------------------------------------------+ -|metric | CPU load | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the CPU load performance of the IaaS. This test | -| | case should be run in parallel to other Yardstick test cases | -| | and not run as a stand-alone test case. | -| | | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: cpuload.yaml (in the 'samples' directory) | -| | | -| | There is are no additional configurations to be set for this | -| | TC. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | mpstat | -| | | -| | (mpstat is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Glance | -| | image. However, if mpstat is not present the TC instead uses | -| | /proc/stats as source to produce "mpstat" output. | -| | | -+--------------+--------------------------------------------------------------+ -|references | man-pages_ | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Run in background with other test cases. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with mpstat included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The host is installed. The related TC, or TCs, is | -| | invoked and mpstat logs are produced and stored. | -| | | -| | Result: Stored logs | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | None. CPU load results are fetched and stored. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc025.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc025.rst deleted file mode 100644 index 0bc0b78ab..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc025.rst +++ /dev/null @@ -1,118 +0,0 @@ -************************************* -Yardstick Test Case Description TC025 -************************************* - -+-----------------------------------------------------------------------------+ -|OpenStack Controller Node abnormally shutdown High Availability | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC025_HA: OpenStack Controller Node | -| | abnormally shutdown | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | This test case will verify the high availability of | -| | controller node. When one of the controller node abnormally | -| | shutdown, the service provided by it should be OK. | -| | | -+--------------+--------------------------------------------------------------+ -|test method | This test case shutdowns a specified controller node with | -| | some fault injection tools, then checks whether all services | -| | provided by the controller node are OK with some monitor | -| | tools. | -| | | -+--------------+--------------------------------------------------------------+ -|attackers | In this test case, an attacker called "host-shutdown" is | -| | needed. This attacker includes two parameters: | -| | 1) fault_type: which is used for finding the attacker's | -| | scripts. It should be always set to "host-shutdown" in | -| | this test case. | -| | 2) host: the name of a controller node being attacked. | -| | | -| | e.g. | -| | -fault_type: "host-shutdown" | -| | -host: node1 | -| | | -+--------------+--------------------------------------------------------------+ -|monitors | In this test case, one kind of monitor are needed: | -| | 1. the "openstack-cmd" monitor constantly request a specific | -| | Openstack command, which needs two parameters | -| | 1) monitor_type: which is used for finding the monitor class | -| | and related scritps. It should be always set to | -| | "openstack-cmd" for this monitor. | -| | 2) command_name: which is the command name used for request | -| | | -| | There are four instance of the "openstack-cmd" monitor: | -| | monitor1: | -| | -monitor_type: "openstack-cmd" | -| | -api_name: "nova image-list" | -| | monitor2: | -| | -monitor_type: "openstack-cmd" | -| | -api_name: "neutron router-list" | -| | monitor3: | -| | -monitor_type: "openstack-cmd" | -| | -api_name: "heat stack-list" | -| | monitor4: | -| | -monitor_type: "openstack-cmd" | -| | -api_name: "cinder list" | -| | | -+--------------+--------------------------------------------------------------+ -|metrics | In this test case, there is one metric: | -| | 1)service_outage_time: which indicates the maximum outage | -| | time (seconds) of the specified Openstack command request. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | Developed by the project. Please see folder: | -| | "yardstick/benchmark/scenarios/availability/ha_tools" | -| | | -+--------------+--------------------------------------------------------------+ -|references | ETSI NFV REL001 | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | This test case needs two configuration files: | -| | 1) test case file: opnfv_yardstick_tc019.yaml | -| | -Attackers: see above "attackers" discription | -| | -waiting_time: which is the time (seconds) from the process | -| | being killed to stoping monitors the monitors | -| | -Monitors: see above "monitors" discription | -| | -SLA: see above "metrics" discription | -| | | -| | 2)POD file: pod.yaml | -| | The POD configuration should record on pod.yaml first. | -| | the "host" item in this test case will use the node name in | -| | the pod.yaml. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | start monitors: | -| | each monitor will run with independently process | -| | | -| | Result: The monitor info will be collected. | -| | | -+--------------+--------------------------------------------------------------+ -|step 2 | do attacker: connect the host through SSH, and then execute | -| | shutdown script on the host | -| | | -| | Result: The host will be shutdown. | -| | | -+--------------+--------------------------------------------------------------+ -|step 3 | stop monitors after a period of time specified by | -| | "waiting_time" | -| | | -| | Result: All monitor result will be aggregated. | -| | | -+--------------+--------------------------------------------------------------+ -|step 4 | verify the SLA | -| | | -| | Result: The test case is passed or not. | -| | | -+--------------+--------------------------------------------------------------+ -|post-action | It is the action when the test cases exist. It restarts the | -| | specified controller node if it is not restarted. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc027.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc027.rst deleted file mode 100644 index 56c8227df..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc027.rst +++ /dev/null @@ -1,67 +0,0 @@ -************************************* -Yardstick Test Case Description TC027 -************************************* - -.. _ipv6: https://wiki.opnfv.org/ipv6_opnfv_project - -+-----------------------------------------------------------------------------+ -|IPv6 connectivity between nodes on the tenant network | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC002_IPv6 connectivity | -| | | -+--------------+--------------------------------------------------------------+ -|metric | RTT, Round Trip Time | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To do a basic verification that IPv6 connectivity is within | -| | acceptable boundaries when ipv6 packets travel between hosts | -| | located on same or different compute blades. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs and similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc027.yaml | -| | | -| | Packet size 56 bytes. | -| | SLA RTT is set to maximum 10 ms. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | ping6 | -| | | -| | Ping6 is normally part of Linux distribution, hence it | -| | doesn't need to be installed. | -| | | -+--------------+--------------------------------------------------------------+ -|references | ipv6_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test case can be configured with different run step | -| | you can run setup, run benchmakr, teardown independently | -| | SLA is optional. The SLA in this test case serves as an | -| | example. Considerably lower RTT is expected. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with ping6 included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. Ping is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Test should not PASS if any RTT is above the optional SLA | -| | value, or if there is a test case execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc028.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc028.rst deleted file mode 100644 index 8ac5f49f9..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc028.rst +++ /dev/null @@ -1,65 +0,0 @@ -************************************* -Yardstick Test Case Description TC028 -************************************* - -.. _Cyclictest: https://rt.wiki.kernel.org/index.php/Cyclictest - -+-----------------------------------------------------------------------------+ -|KVM Latency measurements | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC028_KVM Latency measurements | -| | | -+--------------+--------------------------------------------------------------+ -|metric | min, avg and max latency | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS KVM virtualization capability with | -| | regards to min, avg and max latency. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs and similar shall be stored for comparison reasons | -| | and product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: samples/cyclictest-node-context.yaml | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | Cyclictest | -| | | -| | (Cyclictest is not always part of a Linux distribution, | -| | hence it needs to be installed. As an example see the | -| | /yardstick/tools/ directory for how to generate a Linux | -| | image with cyclictest included.) | -| | | -+--------------+--------------------------------------------------------------+ -|references | Cyclictest_ | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | This test case is mainly for kvm4nfv project CI verify. | -| | Upgrade host linux kernel, boot a gust vm update it's linux | -| | kernel, and then run the cyclictest to test the new kernel | -| | is work well. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test kernel rpm, test sequence scripts and test guest | -|conditions | image need put the right folders as specified in the test | -| | case yaml file. | -| | The test guest image needs with cyclictest included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The host and guest os kernel is upgraded. Cyclictest is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc037.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc037.rst deleted file mode 100644 index 5c91f6bf1..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc037.rst +++ /dev/null @@ -1,99 +0,0 @@ -************************************* -Yardstick Test Case Description TC037 -************************************* - -.. _cirros: https://download.cirros-cloud.net -.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt - -+-----------------------------------------------------------------------------+ -|Latency, CPU Load, Throughput, Packet Loss | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC037_Latency,CPU Load,Throughput,Packet Loss| -| | | -+--------------+--------------------------------------------------------------+ -|metric | Number of flows, latency, throughput, CPU load, packet loss | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS network performance with regards to | -| | flows and throughput, such as if and how different amounts | -| | of flows matter for the throughput between hosts on different| -| | compute blades. Typically e.g. the performance of a vSwitch | -| | depends on the number of flows running through it. Also | -| | performance of other equipment or entities can depend | -| | on the number of flows or the packet sizes used. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc037.yaml | -| | | -| | Packet size: 64 bytes | -| | Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. | -| | The amount configured ports map from 2 up to 1001000 flows, | -| | respectively. Each port amount is run two times, for 20 | -| | seconds each. Then the next port_amount is run, and so on. | -| | During the test CPU load on both client and server, and the | -| | network latency between the client and server are measured. | -| | The client and server are distributed on different HW. | -| | For SLA max_ppm is set to 1000. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | pktgen | -| | | -| | (Pktgen is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Glance | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -| | ping | -| | | -| | Ping is normally part of any Linux distribution, hence it | -| | doesn't need to be installed. It is also part of the | -| | Yardstick Glance image. | -| | (For example also a cirros_ image can be downloaded, it | -| | includes ping) | -| | | -| | mpstat | -| | | -| | (Mpstat is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Glance | -| | image. | -| | | -+--------------+--------------------------------------------------------------+ -|references | Ping and Mpstat man pages | -| | | -| | pktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different packet sizes, amount | -| | of flows and test duration. Default values exist. | -| | | -| | SLA (optional): max_ppm: The number of packets per million | -| | packets sent that are acceptable to loose, not received. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with pktgen included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. pktgen is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc038.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc038.rst deleted file mode 100644 index 93c2cf3d8..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc038.rst +++ /dev/null @@ -1,99 +0,0 @@ -************************************* -Yardstick Test Case Description TC038 -************************************* - -.. _cirros: https://download.cirros-cloud.net -.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt - -+-----------------------------------------------------------------------------+ -|Latency, CPU Load, Throughput, Packet Loss (Extended measurements) | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC038_Latency,CPU Load,Throughput,Packet Loss| -| | | -+--------------+--------------------------------------------------------------+ -|metric | Number of flows, latency, throughput, CPU load, packet loss | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | To evaluate the IaaS network performance with regards to | -| | flows and throughput, such as if and how different amounts | -| | of flows matter for the throughput between hosts on different| -| | compute blades. Typically e.g. the performance of a vSwitch | -| | depends on the number of flows running through it. Also | -| | performance of other equipment or entities can depend | -| | on the number of flows or the packet sizes used. | -| | The purpose is also to be able to spot trends. Test results, | -| | graphs ans similar shall be stored for comparison reasons and| -| | product evolution understanding between different OPNFV | -| | versions and/or configurations. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc038.yaml | -| | | -| | Packet size: 64 bytes | -| | Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. | -| | The amount configured ports map from 2 up to 1001000 flows, | -| | respectively. Each port amount is run ten times, for 20 | -| | seconds each. Then the next port_amount is run, and so on. | -| | During the test CPU load on both client and server, and the | -| | network latency between the client and server are measured. | -| | The client and server are distributed on different HW. | -| | For SLA max_ppm is set to 1000. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | pktgen | -| | | -| | (Pktgen is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Glance | -| | image. | -| | As an example see the /yardstick/tools/ directory for how | -| | to generate a Linux image with pktgen included.) | -| | | -| | ping | -| | | -| | Ping is normally part of any Linux distribution, hence it | -| | doesn't need to be installed. It is also part of the | -| | Yardstick Glance image. | -| | (For example also a cirros_ image can be downloaded, it | -| | includes ping) | -| | | -| | mpstat | -| | | -| | (Mpstat is not always part of a Linux distribution, hence it | -| | needs to be installed. It is part of the Yardstick Glance | -| | image. | -| | | -+--------------+--------------------------------------------------------------+ -|references | Ping and Mpstat man pages | -| | | -| | pktgen_ | -| | | -| | ETSI-NFV-TST001 | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different packet sizes, amount | -| | of flows and test duration. Default values exist. | -| | | -| | SLA (optional): max_ppm: The number of packets per million | -| | packets sent that are acceptable to loose, not received. | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | The test case image needs to be installed into Glance | -|conditions | with pktgen included in it. | -| | | -| | No POD specific requirements have been identified. | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | The hosts are installed, as server and client. pktgen is | -| | invoked and logs are produced and stored. | -| | | -| | Result: Logs are stored. | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if SLA is not passed, or if there is a test case | -| | execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc040.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc040.rst deleted file mode 100644 index 044ccf193..000000000 --- a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc040.rst +++ /dev/null @@ -1,60 +0,0 @@ -************************************* -Yardstick Test Case Description TC040 -************************************* - -.. _Parser: https://wiki.opnfv.org/parser - -+-----------------------------------------------------------------------------+ -|Verify Parser Yang-to-Tosca | -| | -+--------------+--------------------------------------------------------------+ -|test case id | OPNFV_YARDSTICK_TC040 Verify Parser Yang-to-Tosca | -| | | -+--------------+--------------------------------------------------------------+ -|metric | 1. tosca file which is converted from yang file by Parser | -| | 2. result whether the output is same with expected outcome | -+--------------+--------------------------------------------------------------+ -|test purpose | To verify the function of Yang-to-Tosca in Parser. | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | file: opnfv_yardstick_tc040.yaml | -| | | -| | yangfile: the path of the yangfile which you want to convert | -| | toscafile: the path of the toscafile which is your expected | -| | outcome. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | Parser | -| | | -| | (Parser is not part of a Linux distribution, hence it | -| | needs to be installed. As an example see the | -| | /yardstick/benchmark/scenarios/parser/parser_setup.sh for | -| | how to install it manual. Of course, it will be installed | -| | and uninstalled automatically when you run this test case | -| | by yardstick) | -+--------------+--------------------------------------------------------------+ -|references | Parser_ | -| | | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | Test can be configured with different path of yangfile and | -| | toscafile to fit your real environment to verify Parser | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | No POD specific requirements have been identified. | -|conditions | it can be run without VM | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | parser is installed without VM, running Yang-to-Tosca module | -| | to convert yang file to tosca file, validating output against| -| | expected outcome. | -| | | -| | Result: Logs are stored. | -+--------------+--------------------------------------------------------------+ -|test verdict | Fails only if output is different with expected outcome | -| | or if there is a test case execution problem. | -| | | -+--------------+--------------------------------------------------------------+ diff --git a/docs/configguide/yardstick_testcases/testcase_description_v2_template.rst b/docs/configguide/yardstick_testcases/testcase_description_v2_template.rst deleted file mode 100644 index 1b8754b05..000000000 --- a/docs/configguide/yardstick_testcases/testcase_description_v2_template.rst +++ /dev/null @@ -1,64 +0,0 @@ -.. Template to be used for test case descriptions in Yardstick Project. - Write one .rst per test case. - Upload the .rst for the test case in /docs/source/yardstick directory. - Review in Gerrit. - -************************************* -Yardstick Test Case Description TCXXX -************************************* - -+-----------------------------------------------------------------------------+ -|test case slogan e.g. Network Latency | -| | -+--------------+--------------------------------------------------------------+ -|test case id | e.g. OPNFV_YARDSTICK_TC001_NW Latency | -| | | -+--------------+--------------------------------------------------------------+ -|metric | what will be measured, e.g. latency | -| | | -+--------------+--------------------------------------------------------------+ -|test purpose | describe what is the purpose of the test case | -| | | -+--------------+--------------------------------------------------------------+ -|configuration | what .yaml file to use, state SLA if applicable, state | -| | test duration, list and describe the scenario options used in| -| | this TC and also list the options using default values. | -| | | -+--------------+--------------------------------------------------------------+ -|test tool | e.g. ping | -| | | -+--------------+--------------------------------------------------------------+ -|references | e.g. RFCxxx, ETSI-NFVyyy | -| | | -+--------------+--------------------------------------------------------------+ -|applicability | describe variations of the test case which can be | -| | performend, e.g. run the test for different packet sizes | -| | | -+--------------+--------------------------------------------------------------+ -|pre-test | describe configuration in the tool(s) used to perform | -|conditions | the measurements (e.g. fio, pktgen), POD-specific | -| | configuration required to enable running the test | -| | | -+--------------+--------------------------------------------------------------+ -|test sequence | description and expected result | -| | | -+--------------+--------------------------------------------------------------+ -|step 1 | use this to describe tests that require sveveral steps e.g | -| | collect logs. | -| | | -| | Result: what happens in this step e.g. logs collected | -| | | -+--------------+--------------------------------------------------------------+ -|step 2 | remove interface | -| | | -| | Result: interface down. | -| | | -+--------------+--------------------------------------------------------------+ -|step N | what is done in step N | -| | | -| | Result: what happens | -| | | -+--------------+--------------------------------------------------------------+ -|test verdict | expected behavior, or SLA, pass/fail criteria | -| | | -+--------------+--------------------------------------------------------------+ |