summaryrefslogtreecommitdiffstats
path: root/docs
diff options
context:
space:
mode:
Diffstat (limited to 'docs')
-rw-r--r--docs/release/release-notes/release-notes.rst64
-rw-r--r--docs/testing/user/userguide/04-installation.rst4
-rw-r--r--docs/testing/user/userguide/10-yardstick-user-interface.rst30
-rw-r--r--docs/testing/user/userguide/glossary.rst60
-rw-r--r--docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst3
-rw-r--r--docs/testing/user/userguide/nsb/tc_vfw_rfc2544.rst189
-rw-r--r--docs/testing/user/userguide/nsb/tc_vfw_rfc2544_correlated.rst130
-rw-r--r--docs/testing/user/userguide/nsb/tc_vfw_rfc3511.rst133
8 files changed, 559 insertions, 54 deletions
diff --git a/docs/release/release-notes/release-notes.rst b/docs/release/release-notes/release-notes.rst
index 4b3f12bcf..914daa3a4 100644
--- a/docs/release/release-notes/release-notes.rst
+++ b/docs/release/release-notes/release-notes.rst
@@ -33,6 +33,9 @@ Version History
| November 9, 2018 | 7.0.0 | Yardstick for Gambia release |
| | | |
+-------------------+-----------+---------------------------------+
+| December 14, 2018 | 7.1.0 | Yardstick for Gambia release |
+| | | |
++-------------------+-----------+---------------------------------+
Important Notes
@@ -111,19 +114,19 @@ Release Data
| **Project** | Yardstick |
| | |
+--------------------------------+-----------------------+
-| **Repo/tag** | yardstick/opnfv-7.0.0 |
+| **Repo/tag** | yardstick/opnfv-7.1.0 |
| | |
+--------------------------------+-----------------------+
-| **Yardstick Docker image tag** | opnfv-7.0.0 |
+| **Yardstick Docker image tag** | opnfv-7.1.0 |
| | |
+--------------------------------+-----------------------+
-| **Release designation** | Gambia 7.0 |
+| **Release designation** | Gambia 7.1 |
| | |
+--------------------------------+-----------------------+
-| **Release date** | November 9, 2018 |
+| **Release date** | December 14, 2018 |
| | |
+--------------------------------+-----------------------+
-| **Purpose of the delivery** | OPNFV Gambia 7.0.0 |
+| **Purpose of the delivery** | OPNFV Gambia 7.1.0 |
| | |
+--------------------------------+-----------------------+
@@ -269,7 +272,7 @@ List of Scenarios
New Test cases
--------------
-.. note:: Yardstick Gambia 7.0.0 adds no new test cases.
+.. note:: Yardstick Gambia 7.1.0 adds no new test cases.
* Generic NFVI test cases
@@ -326,7 +329,7 @@ Feature additions
Scenario Matrix
===============
-For Gambia 7.0.0, Yardstick was tested on the following scenarios:
+For Gambia 7.1.0, Yardstick was tested on the following scenarios:
+-------------------------+------+---------+----------+------+
| Scenario | Apex | Compass | Fuel-arm | Fuel |
@@ -370,58 +373,35 @@ Known Issues/Faults
Corrected Faults
----------------
-Gambia 7.0.0:
+Gambia 7.1.0:
+--------------------+--------------------------------------------------------------------------+
| **JIRA REFERENCE** | **DESCRIPTION** |
+====================+==========================================================================+
-| YARDSTICK-1137 | Fix CLI argument handling in nsb_setup.sh |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1220 | Get stats for multiple port simultaneously |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1260 | Added missing functionality to start VM and access it using SSH keys |
-| | in Standalone contexts. |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1298 | Allows for in-line overriding/modification of traffic profile variables |
-| | from the testcase file. |
+| YARDSTICK-1241 | Update NSB PROX devguide. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1368 | Updated existing test cases in Yardstick to minimize changes done |
-| | manually to run standalone tests for Trex. |
+| YARDSTICK-1458 | NSB NFVi PROX Should report realtime port activity not historical data. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1389 | Add status filed for RFC2544 TC iterations |
+| YARDSTICK-1471 | Add Testcase Prox Standalone SRIOV. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1395 | Update 'configure_uwsgi' role to work in baremetal/container modes. |
+| YARDSTICK-1475 | Adding Testcase for Prox Stanalone OvS-DPDK. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1402 | Change IP assignment for VM to static for standalone context |
+| YARDSTICK-1500 | Adding Testcase for Prox L2FWD PktTouch Stanalone OvS-DPDK. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1404 | CPU Utilization for VNF and traffic generator are now graphed on Grafana |
+| YARDSTICK-1517 | Missing opnfv "os-ovn-nofeature-ha" scenario test suite. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1411 | Fix Yardstick Docker image ARM build |
+| YARDSTICK-l526 | Run testcase 074 result overridden by job status. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1414 | Update the pinned sampleVNF version to use a commit instead of a branch |
+| YARDSTICK-1547 | Adding scale up test case for l3fwd OvS-DPDK. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1418 | NSB PROX NFVi test now stops after reaching expected precision |
+| YARDSTICK-1560 | Fix pip environment. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1457 | Fix influxdb "field type conflict" error |
+| YARDSTICK-1561 | L3FWD Gradana Dashboards Out-of-date and incorrect. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1458 | Update Grafana to display "real-time" data instead of historical data. |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1462 | NSB: Add OvS 2.8.1 support in SA context |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1492 | Make OvS server to listen on TCP |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1493 | The RX queues number is hard-codded and cannot be changed |
-+--------------------+--------------------------------------------------------------------------+
-
-Gambia 7.0.0 known restrictions/issues
+Gambia 7.1.0 known restrictions/issues
======================================
-+-----------+-----------------------+------------------+
-| Installer | Scenario | Issue |
-+===========+=======================+==================+
-| apex | os-ovn-nofeature-ha | YARDSTICK-1517 |
-+-----------+-----------------------+------------------+
Useful links
============
diff --git a/docs/testing/user/userguide/04-installation.rst b/docs/testing/user/userguide/04-installation.rst
index 2f8175c25..3ba312ce7 100644
--- a/docs/testing/user/userguide/04-installation.rst
+++ b/docs/testing/user/userguide/04-installation.rst
@@ -613,15 +613,15 @@ Run influxDB::
sudo -EH docker run -d --name influxdb \
-p 8083:8083 -p 8086:8086 --expose 8090 --expose 8099 \
tutum/influxdb
- docker exec -it influxdb influx
Configure influxDB::
+ docker exec -it influxdb influx
> CREATE USER root WITH PASSWORD 'root' WITH ALL PRIVILEGES
> CREATE DATABASE yardstick;
> use yardstick;
> show MEASUREMENTS;
- > quit
+ > exit
Run Grafana::
diff --git a/docs/testing/user/userguide/10-yardstick-user-interface.rst b/docs/testing/user/userguide/10-yardstick-user-interface.rst
index 76890b29a..b3056ec99 100644
--- a/docs/testing/user/userguide/10-yardstick-user-interface.rst
+++ b/docs/testing/user/userguide/10-yardstick-user-interface.rst
@@ -2,16 +2,32 @@
Yardstick User Interface
========================
-This interface provides a user to view the test result
-in table format and also values pinned on to a graph.
+This chapter describes how to generate HTML reports, used to view, store, share
+or publish test results in table and graph formats.
+The following layouts are available:
-Command
-=======
-::
+* The compact HTML report layout is suitable for testcases producing a few
+ metrics over a short period of time. All metrics for all timestamps are
+ displayed in the data table and on the graph.
+
+* The dynamic HTML report layout consists of a wider data table, a graph, and
+ a tree that allows selecting the metrics to be displayed. This layout is
+ suitable for testcases, such as NSB ones, producing a lot of metrics over
+ a longer period of time.
+
+
+Commands
+========
+
+To generate the compact HTML report, run::
yardstick report generate <task-ID> <testcase-filename>
+To generate the dynamic HTML report, run::
+
+ yardstick report generate-nsb <task-ID> <testcase-filename>
+
Description
===========
@@ -27,8 +43,8 @@ Description
The graph is framed with Timestamp on x-axis and output values
(differ from testcase to testcase) on y-axis with the help of
-`Highcharts`_.
+`Chart.js`_.
.. _InfluxDB: https://www.influxdata.com/time-series-platform/influxdb/
.. _Jinja2: http://jinja.pocoo.org/docs/2.10/
-.. _Highcharts: https://www.highcharts.com/products/highcharts/
+.. _Chart.js: https://www.chartjs.org/
diff --git a/docs/testing/user/userguide/glossary.rst b/docs/testing/user/userguide/glossary.rst
index 6a153943c..cef9b69a5 100644
--- a/docs/testing/user/userguide/glossary.rst
+++ b/docs/testing/user/userguide/glossary.rst
@@ -13,23 +13,54 @@ Glossary
API
Application Programming Interface
+ Barometer
+ OPNFV NFVi Service Assurance project. Barometer upstreams changes to
+ collectd, OpenStack, etc to improve features related to NFVi monitoring
+ and service assurance.
+ More info on: https://opnfv-barometer.readthedocs.io/en/latest/
+
+ collectd
+ collectd is a system statistics collection daemon.
+ More info on: https://collectd.org/
+
+ context
+ A context describes the environment in which a yardstick testcase will
+ be run. It can refer to a pre-provisioned environment, or an environment
+ that will be set up using OpenStack or Kubernetes.
+
Docker
Docker provisions and manages containers. Yardstick and many other OPNFV
projects are deployed in containers. Docker is required to launch the
containerized versions of these projects.
- DPI
- Deep Packet Inspection
-
DPDK
Data Plane Development Kit
+ DPI
+ Deep Packet Inspection
+
DSCP
Differentiated Services Code Point
+ flavor
+ A specification of virtual resources used by OpenStack in the creation
+ of a VM instance.
+
+ Grafana
+ A visualization tool, used in Yardstick to retrieve test data from
+ InfluxDB and display it. Grafana works by defining dashboards, which are
+ combinations of visualization panes (e.g. line charts and gauges) and
+ forms that assist the user in formulating SQL-like queries for InfluxDB.
+ More info on: https://grafana.com/
+
IGMP
Internet Group Management Protocol
+ InfluxDB
+ One of the Dispatchers supported by Yardstick, it allows test results to
+ be reported to a time-series database.
+ More info on: https://www.influxdata.com/
+
IOPS
Input/Output Operations Per Second
A performance measurement used to benchmark storage devices.
@@ -43,6 +74,9 @@ Glossary
deployment, scaling and management of containerized applications.
It is one of the contexts supported in Yardstick.
+ MPLS
+ Multiprotocol Label Switching
+
NFV
Network Function Virtualization
NFV is an initiative to take network services which were traditionally run
@@ -56,6 +90,10 @@ Glossary
NIC
Network Interface Controller
+ NSB
+ Network Services Benchmarking. A subset of Yardstick features concerned
+ with NFVI and VNF characterization.
+
OpenStack
OpenStack is a cloud operating system that controls pools of compute,
storage, and networking resources. OpenStack is an open source project
@@ -77,6 +115,18 @@ Glossary
performance in Input/Output Operations Per Second (IOPS), throttling
agreements, and performance expectations at peak load
+ runner
+ The part of a Yardstick testcase that determines how the test will be run
+ (e.g. for x iterations, y seconds or until state z is reached). The runner
+ also determines when the metrics are collected/reported.
+
+ SampleVNF
+ OPNFV project providing a repository of reference VNFs.
+ More info on: https://opnfv-samplevnf.readthedocs.io/en/latest/
+
+ scenario
+ The part of a Yardstick testcase that describes each test step.
+
SLA
Service Level Agreement
An SLA is an agreement between a service provider and a customer to
@@ -92,6 +142,10 @@ Glossary
SUT
System Under Test
+ testcase
+ A task in Yardstick; the yaml file that is read by Yardstick to
+ determine how to run a test.
+
ToS
Type of Service
diff --git a/docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst b/docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst
index 723cd6f99..6c18c7d89 100644
--- a/docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst
+++ b/docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst
@@ -33,3 +33,6 @@ NSB PROX Test Case Descriptions
tc_epc_saegw_tput_relocation_landslide
tc_epc_network_service_request_landslide
tc_epc_ue_service_request_landslide
+ tc_vfw_rfc2544
+ tc_vfw_rfc2544_correlated
+ tc_vfw_rfc3511
diff --git a/docs/testing/user/userguide/nsb/tc_vfw_rfc2544.rst b/docs/testing/user/userguide/nsb/tc_vfw_rfc2544.rst
new file mode 100644
index 000000000..139990bc3
--- /dev/null
+++ b/docs/testing/user/userguide/nsb/tc_vfw_rfc2544.rst
@@ -0,0 +1,189 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, 2018 Intel Corporation.
+
+************************************************
+Yardstick Test Case Description: NSB vFW RFC2544
+************************************************
+
++------------------------------------------------------------------------------+
+| NSB vFW test for VNF characterization |
+| |
++---------------+--------------------------------------------------------------+
+| test case id | tc_{context}_rfc2544_ipv4_1rule_1flow_{pkt_size}_{tg_type} |
+| | |
+| | * context = baremetal, heat, heat_external, ovs, sriov |
+| | heat_sriov_external contexts; |
+| | * tg_type = ixia (context != heat,heat_sriov_external), |
+| | trex; |
+| | * pkt_size = 64B - all contexts; |
+| | 128B, 256B, 512B, 1024B, 1280B, 1518B - |
+| | (context = heat, tg_type = ixia) |
+| | |
++---------------+--------------------------------------------------------------+
+| metric | * Network Throughput; |
+| | * TG Packets Out; |
+| | * TG Packets In; |
+| | * TG Latency; |
+| | * VNF Packets Out; |
+| | * VNF Packets In; |
+| | * VNF Packets Fwd; |
+| | * Dropped packets; |
+| | |
++---------------+--------------------------------------------------------------+
+| test purpose | The VFW RFC2544 tests measure performance characteristics of |
+| | the SUT (multiple ports) and sends UDP bidirectional traffic |
+| | from all TG ports to SampleVNF vFW application. The |
+| | application forwards received traffic based on rules |
+| | provided by the user in the TC configuration and default |
+| | rules created by vFW to send traffic from uplink ports to |
+| | downlink and voice versa. |
+| | |
++---------------+--------------------------------------------------------------+
+| configuration | The 2 ports RFC2544 test cases are listed below: |
+| | |
+| | * tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_ixia.yaml |
+| | * tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_trex.yaml |
+| | * tc_heat_external_rfc2544_ipv4_1rule_1flow_1024B_ixia.yaml |
+| | * tc_heat_external_rfc2544_ipv4_1rule_1flow_1280B_ixia.yaml |
+| | * tc_heat_external_rfc2544_ipv4_1rule_1flow_128B_ixia.yaml |
+| | * tc_heat_external_rfc2544_ipv4_1rule_1flow_1518B_ixia.yaml |
+| | * tc_heat_external_rfc2544_ipv4_1rule_1flow_256B_ixia.yaml |
+| | * tc_heat_external_rfc2544_ipv4_1rule_1flow_512B_ixia.yaml |
+| | * tc_heat_external_rfc2544_ipv4_1rule_1flow_64B_ixia.yaml |
+| | * tc_heat_external_rfc2544_ipv4_1rule_1flow_64B_trex.yaml |
+| | * tc_heat_sriov_external_rfc2544_ipv4_1rule_1flow_64B_trex. |
+| | yaml |
+| | * tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex.yaml |
+| | * tc_ovs_rfc2544_ipv4_1rule_1flow_64B_ixia.yaml |
+| | * tc_ovs_rfc2544_ipv4_1rule_1flow_64B_trex.yaml |
+| | * tc_sriov_rfc2544_ipv4_1rule_1flow_64B_ixia.yaml |
+| | * tc_sriov_rfc2544_ipv4_1rule_1flow_64B_trex.yaml |
+| | |
+| | The 4 ports RFC2544 test cases are listed below: |
+| | |
+| | * tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_ixia_4port.yaml |
+| | * tc_tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_trex_4port. |
+| | yaml |
+| | * tc_tc_heat_external_rfc2544_ipv4_1rule_1flow_64B_trex_4 |
+| | port.yaml |
+| | * tc_tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_4port.yaml |
+| | |
+| | The scale-up RFC2544 test cases are listed below: |
+| | |
+| | * tc_tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_scale-up.yaml |
+| | |
+| | The scale-out RFC2544 test cases are listed below: |
+| | |
+| | * tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_scale_out.yaml |
+| | |
+| | Test duration is set as 30 sec for each test and default |
+| | number of rules are applied. These can be configured |
+| | |
++---------------+--------------------------------------------------------------+
+| test tool | The vFW is a DPDK application that performs basic filtering |
+| | for malformed packets and dynamic packet filtering of |
+| | incoming packets using the connection tracker library. |
+| | |
++---------------+--------------------------------------------------------------+
+| applicability | The vFW RFC2544 test cases can be configured with different: |
+| | |
+| | * packet sizes; |
+| | * test duration; |
+| | * tolerated loss; |
+| | * traffic flows; |
+| | * rules; |
+| | |
+| | Default values exist. |
+| | |
++---------------+--------------------------------------------------------------+
+| pre-test | For OpenStack test case image (yardstick-samplevnf) needs |
+| conditions | to be installed into Glance with vFW and DPDK included in |
+| | it (NSB install). |
+| | |
+| | For Baremetal tests cases vFW and DPDK must be installed on |
+| | the hosts where the test is executed. The pod.yaml file must |
+| | have the necessary system and NIC information. |
+| | |
+| | For standalone (SA) SRIOV/OvS test cases the |
+| | yardstick-samplevnf image needs to be installed on hosts and |
+| | pod.yaml file must be provided with necessary system, NIC |
+| | information. |
+| | |
++---------------+--------------------------------------------------------------+
+| test sequence | Description and expected result |
+| | |
++---------------+--------------------------------------------------------------+
+| step 1 | For Baremetal test: The TG (except IXIA) and VNF are started |
+| | on the hosts based on the pod file. |
+| | |
+| | For Heat test: Two host VMs are booted, as Traffic generator |
+| | and VNF(vFW) based on the test flavor. In case of scale-out |
+| | scenario the multiple VNF VMs will be started. |
+| | |
+| | For Heat external test: vFW VM is booted and TG (except IXIA)|
+| | generator is started on the external host based on the pod |
+| | file. In case of scale-out scenario the multiple VNF VMs |
+| | will be deployed. |
+| | |
+| | For Heat SRIOV external test: vFW VM is booted with network |
+| | interfaces of `direct` type which are mapped to VFs that are |
+| | available to OpenStack. TG (except IXIA) is started on the |
+| | external host based on the pod file. In case of scale-out |
+| | scenario the multiple VNF VMs will be deployed. |
+| | |
+| | For SRIOV test: VF ports are created on host's PFs specified |
+| | in the TC file and VM is booed using those ports and image |
+| | provided in the configuration. TG (except IXIA) is started |
+| | on other host connected to VNF machine based on the pod |
+| | file. The vFW is started in the booted VM. In case of |
+| | scale-out scenario the multiple VNF VMs will be created. |
+| | |
+| | For OvS-DPDK test: OvS DPDK switch is started and bridges |
+| | are created with ports specified in the TC file. DPDK vHost |
+| | ports are added to corresponding bridge and VM is booed |
+| | using those ports and image provided in the configuration. |
+| | TG (except IXIA) is started on other host connected to VNF |
+| | machine based on the pod file. The vFW is started in the |
+| | booted VM. In case of scale-out scenario the multiple VNF |
+| | VMs will be deployed. |
+| | |
++---------------+--------------------------------------------------------------+
+| step 2 | Yardstick is connected with the TG and VNF by using ssh (in |
+| | case of IXIA TG is connected via TCL interface). The test |
+| | will resolve the topology and instantiate all VNFs |
+| | and TG and collect the KPI's/metrics. |
+| | |
++---------------+--------------------------------------------------------------+
+| step 3 | The TG will send packets to the VNFs. If the number of |
+| | dropped packets is more than the tolerated loss the line |
+| | rate or throughput is halved. This is done until the dropped |
+| | packets are within an acceptable tolerated loss. |
+| | |
+| | The KPI is the number of packets per second for different |
+| | packet size with an accepted minimal packet loss for the |
+| | default configuration. |
+| | |
++---------------+--------------------------------------------------------------+
+| step 4 | In Baremetal test: The test quits the application and unbind |
+| | the DPDK ports. |
+| | |
+| | In Heat test: All VNF VMs and TG are deleted on test |
+| | completion. |
+| | |
+| | In SRIOV test: The deployed VM with vFW is destroyed on the |
+| | host and TG (exclude IXIA) is stopped. |
+| | |
+| | In Heat SRIOV test: The deployed VM with vFW is destroyed, |
+| | VFs are released and TG (exclude IXIA) is stopped. |
+| | |
+| | In OvS test: The deployed VM with vFW is destroyed on the |
+| | host and OvS DPDK switch is stopped and ports are unbinded. |
+| | The TG (exclude IXIA) is stopped. |
+| | |
++---------------+--------------------------------------------------------------+
+| test verdict | The test case will achieve a Throughput with an accepted |
+| | minimal tolerated packet loss. |
++---------------+--------------------------------------------------------------+
+
diff --git a/docs/testing/user/userguide/nsb/tc_vfw_rfc2544_correlated.rst b/docs/testing/user/userguide/nsb/tc_vfw_rfc2544_correlated.rst
new file mode 100644
index 000000000..de490900d
--- /dev/null
+++ b/docs/testing/user/userguide/nsb/tc_vfw_rfc2544_correlated.rst
@@ -0,0 +1,130 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, 2018 Intel Corporation.
+
+*************************************************************
+Yardstick Test Case Description: NSB vFW RFC2544 (correlated)
+*************************************************************
+
++------------------------------------------------------------------------------+
+| NSB vFW test for VNF characterization using correlated traffic |
+| |
++---------------+--------------------------------------------------------------+
+| test case id | tc_{context}_rfc2544_ipv4_1rule_1flow_64B_trex_corelated |
+| | |
+| | * context = baremetal, heat |
+| | |
++---------------+--------------------------------------------------------------+
+| metric | * Network Throughput; |
+| | * TG Packets Out; |
+| | * TG Packets In; |
+| | * TG Latency; |
+| | * VNF Packets Out; |
+| | * VNF Packets In; |
+| | * VNF Packets Fwd; |
+| | * Dropped packets; |
+| | |
+| | NOTE: For correlated TCs the TG metrics are available on |
+| | uplink ports. |
+| | |
++---------------+--------------------------------------------------------------+
+| test purpose | The VFW RFC2544 correlated tests measure performance |
+| | characteristics of the SUT (multiple ports) and sends UDP |
+| | traffic from uplink TG ports to SampleVNF vFW application. |
+| | The application forwards received traffic from uplink ports |
+| | to downlink ports based on rules provided by the user in the |
+| | TC configuration and default rules created by vFW. The VNF |
+| | downlink traffic is received by another UDPReplay VNF and it |
+| | is mirrored back to the VNF on the same port. Finally, the |
+| | traffic is received back to the TG uplink port. |
+| | |
++---------------+--------------------------------------------------------------+
+| configuration | The 2 ports RFC2544 correlated test cases are listed below: |
+| | |
+| | * tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_trex_corelated |
+| | _traffic.yaml |
+| | |
+| | Multiple VNF (2, 4, 10) RFC2544 correlated test cases are |
+| | listed below: |
+| | |
+| | * tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_correlated |
+| | _scale_10.yaml |
+| | * tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_correlated_scale |
+| | _2.yaml |
+| | * tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_correlated_scale |
+| | _4.yaml |
+| | |
+| | The scale-out RFC2544 test cases are listed below: |
+| | |
+| | * tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_correlated_scale |
+| | _out.yaml |
+| | |
+| | Test duration is set as 30 sec for each test and default |
+| | number of rules are applied. These can be configured |
+| | |
++---------------+--------------------------------------------------------------+
+| test tool | The vFW is a DPDK application that performs basic filtering |
+| | for malformed packets and dynamic packet filtering of |
+| | incoming packets using the connection tracker library. |
+| | |
++---------------+--------------------------------------------------------------+
+| applicability | The vFW RFC2544 test cases can be configured with different: |
+| | |
+| | * packet sizes; |
+| | * test duration; |
+| | * tolerated loss; |
+| | * traffic flows; |
+| | * rules; |
+| | |
+| | Default values exist. |
+| | |
++---------------+--------------------------------------------------------------+
+| pre-test | For OpenStack test case image (yardstick-samplevnf) needs |
+| conditions | to be installed into Glance with vFW and DPDK included in |
+| | it (NSB install). |
+| | |
+| | For Baremetal tests cases vFW and DPDK must be installed on |
+| | the hosts where the test is executed. The pod.yaml file must |
+| | have the necessary system and NIC information. |
+| | |
++---------------+--------------------------------------------------------------+
+| test sequence | Description and expected result |
+| | |
++---------------+--------------------------------------------------------------+
+| step 1 | For Baremetal test: The TG (except IXIA), vFW and UDPReplay |
+| | VNFs are started on the hosts based on the pod file. |
+| | |
+| | For Heat test: Three host VMs are booted, as Traffic |
+| | generator, vFW and UDPReplay VNF(vFW) based on the test |
+| | flavor. In case of scale-out scenario the multiple vFW VNF |
+| | VMs will be started. |
+| | |
++---------------+--------------------------------------------------------------+
+| step 2 | Yardstick is connected with the TG, vFW and UDPReplay VNF by |
+| | using ssh (in case of IXIA TG is connected via TCL |
+| | interface). The test will resolve the topology and |
+| | instantiate all VNFs and TG and collect the KPI's/metrics. |
+| | |
++---------------+--------------------------------------------------------------+
+| step 3 | The TG will send packets to the VNFs. If the number of |
+| | dropped packets is more than the tolerated loss the line |
+| | rate or throughput is halved. This is done until the dropped |
+| | packets are within an acceptable tolerated loss. |
+| | |
+| | The KPI is the number of packets per second for 64B packet |
+| | size with an accepted minimal packet loss for the default |
+| | configuration. |
+| | |
++---------------+--------------------------------------------------------------+
+| step 4 | In Baremetal test: The test quits the application and unbind |
+| | the DPDK ports. |
+| | |
+| | In Heat test: All VNF VMs and TG are deleted on test |
+| | completion. |
+| | |
++---------------+--------------------------------------------------------------+
+| test verdict | The test case will achieve a Throughput with an accepted |
+| | minimal tolerated packet loss. |
++---------------+--------------------------------------------------------------+
+
diff --git a/docs/testing/user/userguide/nsb/tc_vfw_rfc3511.rst b/docs/testing/user/userguide/nsb/tc_vfw_rfc3511.rst
new file mode 100644
index 000000000..9051fc4df
--- /dev/null
+++ b/docs/testing/user/userguide/nsb/tc_vfw_rfc3511.rst
@@ -0,0 +1,133 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, 2018 Intel Corporation.
+
+*******************************************************
+Yardstick Test Case Description: NSB vFW RFC3511 (HTTP)
+*******************************************************
+
++------------------------------------------------------------------------------+
+| NSB vFW test for VNF characterization based on RFC3511 and IXIA |
+| |
++---------------+--------------------------------------------------------------+
+| test case id | tc_{context}_http_ixload_{http_size}_Requests-65000_{type} |
+| | |
+| | * context = baremetal, heat_external |
+| | * http_size = 1b, 4k, 64k, 256k, 512k, 1024k payload size |
+| | * type = Concurrency, Connections, Throughput |
+| | |
++---------------+--------------------------------------------------------------+
+| metric | * HTTP Total Throughput (Kbps); |
+| | * HTTP Simulated Users; |
+| | * HTTP Concurrent Connections; |
+| | * HTTP Connection Rate; |
+| | * HTTP Transaction Rate |
+| | |
++---------------+--------------------------------------------------------------+
+| test purpose | The vFW RFC3511 tests measure performance characteristics of |
+| | the SUT by sending the HTTP traffic from uplink to downlink |
+| | TG ports through vFW VNF. The application forwards received |
+| | traffic based on rules provided by the user in the TC |
+| | configuration and default rules created by vFW to send |
+| | traffic from uplink ports to downlink and voice versa. |
+| | |
++---------------+--------------------------------------------------------------+
+| configuration | The 2 ports RFC3511 test cases are listed below: |
+| | |
+| | * tc_baremetal_http_ixload_1024k_Requests-65000 |
+| | _Concurrency.yaml |
+| | * tc_baremetal_http_ixload_1b_Requests-65000 |
+| | _Concurrency.yaml |
+| | * tc_baremetal_http_ixload_256k_Requests-65000 |
+| | _Concurrency.yaml |
+| | * tc_baremetal_http_ixload_4k_Requests-65000 |
+| | _Concurrency.yaml |
+| | * tc_baremetal_http_ixload_512k_Requests-65000 |
+| | _Concurrency.yaml |
+| | * tc_baremetal_http_ixload_64k_Requests-65000 |
+| | _Concurrency.yaml |
+| | * tc_heat_external_http_ixload_1b_Requests-10Gbps |
+| | _Throughput.yaml |
+| | * tc_heat_external_http_ixload_1b_Requests-65000 |
+| | _Concurrency.yaml |
+| | * tc_heat_external_http_ixload_1b_Requests-65000 |
+| | _Connections.yaml |
+| | |
+| | The 4 ports RFC3511 test cases are listed below: |
+| | |
+| | * tc_baremetal_http_ixload_1b_Requests-65000 |
+| | _Concurrency_4port.yaml |
+| | |
++---------------+--------------------------------------------------------------+
+| test tool | The vFW is a DPDK application that performs basic filtering |
+| | for malformed packets and dynamic packet filtering of |
+| | incoming packets using the connection tracker library. |
+| | |
++---------------+--------------------------------------------------------------+
+| applicability | The vFW RFC3511 test cases can be configured with different: |
+| | |
+| | * http payload sizes; |
+| | * traffic flows; |
+| | * rules; |
+| | |
+| | Default values exist. |
+| | |
++---------------+--------------------------------------------------------------+
+| pre-test | For OpenStack test case image (yardstick-samplevnf) needs |
+| conditions | to be installed into Glance with vFW and DPDK included in |
+| | it (NSB install). |
+| | |
+| | For Baremetal tests cases vFW and DPDK must be installed on |
+| | the hosts where the test is executed. The pod.yaml file must |
+| | have the necessary system and NIC information. |
+| | |
++---------------+--------------------------------------------------------------+
+| test sequence | Description and expected result |
+| | |
++---------------+--------------------------------------------------------------+
+| step 1 | For Baremetal test: The vFW VNF is started on the hosts |
+| | based on the pod file. |
+| | |
+| | For Heat external test: The vFW VM are deployed and booted. |
+| | |
++---------------+--------------------------------------------------------------+
+| step 2 | Yardstick is connected with the TG (IxLoad) via IxLoad API |
+| | and VNF by using ssh. The test will resolve the topology and |
+| | instantiate all VNFs and TG and collect the KPI's/metrics. |
+| | |
++---------------+--------------------------------------------------------------+
+| step 3 | The TG simulates HTTP traffic based on selected type of TC. |
+| | |
+| | Concurrency: |
+| | The TC attempts to simulate some number of human users. |
+| | The simulated users are gradually brought online until 64K |
+| | users is met (the Ramp-Up phase), then taken offline (the |
+| | Ramp Down phase). |
+| | |
+| | Connections: |
+| | The TC creates some number of HTTP connections per second. |
+| | It will attempt to generate the 64K of HTTP connections |
+| | per second. |
+| | |
+| | Throughput: |
+| | TC simultaneously transmits and receives TCP payload |
+| | (bytes) at a certain rate measured in Megabits per second |
+| | (Mbps), Kilobits per second (Kbps), or Gigabits per |
+| | second. The 10 Gbits is default throughput. |
+| | |
+| | At the end of the TC, the KPIs are collected and stored |
+| | (depends on the selected dispatcher). |
+| | |
++---------------+--------------------------------------------------------------+
+| step 4 | In Baremetal test: The test quits the application and |
+| | unbinds the DPDK ports. |
+| | |
+| | In Heat test: All VNF VMs are deleted and connections to TG |
+| | are terminated. |
+| | |
++---------------+--------------------------------------------------------------+
+| test verdict | The test case will try to achieve the configured HTTP |
+| | Concurrency/Throughput/Connections. |
++---------------+--------------------------------------------------------------+
+