summaryrefslogtreecommitdiffstats
path: root/docs/testing/user/userguide/02-methodology.rst
diff options
context:
space:
mode:
authorLuc Provoost <luc.provoost@intel.com>2020-11-23 18:24:10 +0100
committerLuc Provoost <luc.provoost@intel.com>2020-11-23 18:24:10 +0100
commit2ecb6a9310d26bf861ac97a1f4c1d264b50e6ef7 (patch)
treead4d4277eb39e078352b38f88da2b5a1ef77bcd8 /docs/testing/user/userguide/02-methodology.rst
parentcbfbf14a9feca56cabd5788927fee4cf419bce4e (diff)
Jerma documentation rewrite
First step in a rewrite of the sampleVNF documentation whcih is now focused on the TST009 testing using PROX and rapid. Change-Id: Id95cd44831ae7db518ccdf1d8da55890197d2fcb Signed-off-by: Luc Provoost <luc.provoost@intel.com>
Diffstat (limited to 'docs/testing/user/userguide/02-methodology.rst')
-rw-r--r--docs/testing/user/userguide/02-methodology.rst96
1 files changed, 42 insertions, 54 deletions
diff --git a/docs/testing/user/userguide/02-methodology.rst b/docs/testing/user/userguide/02-methodology.rst
index 01cbb276..30efe79f 100644
--- a/docs/testing/user/userguide/02-methodology.rst
+++ b/docs/testing/user/userguide/02-methodology.rst
@@ -6,81 +6,69 @@
===========
Methodology
===========
+.. _NFV-TST009: https://docbox.etsi.org/ISG/NFV/open/Publications_pdf/Specs-Reports/NFV-TST%20009v3.2.1%20-%20GS%20-%20NFVI_Benchmarks.pdf
Abstract
========
This chapter describes the methodology/overview of SampleVNF project from
-the perspective of a :term:`VNF` and :term:`NFVI` Characterization
+the perspective of :term:`NFVI` Characterization
Overview
========
-This project provides a placeholder for various sample VNF (Virtual Network Function (:term:`VNF`))
-development which includes example reference architecture and optimization methods
-related to VNF/Network service for high performance VNFs.
+This project covers the dataplane benchmarking for Network Function Virtualization
+Infrastructure (:term:`NFVI`)) using the PROX tool, according to ETSI GS NFV-TST009_.
-The sample VNFs are Open Source approximations* of Telco grade :term:`VNF`
-using optimized VNF + NFVi Infrastructure libraries, with Performance Characterization of Sample† Traffic Flows.
-• * Not a commercial product. Encourage the community to contribute and close the feature gaps.
-• † No Vendor/Proprietary Workloads
+The test execution and reporting is driven by the Xtesting framework and is fully automated.
-ETSI-NFV
-========
-
-.. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
-.. _SampleVNFtst: https://wiki.opnfv.org/display/SAM/Technical+Briefs+of+VNFs
-.. _Yardstick_NSB: http://artifacts.opnfv.org/yardstick/docs/testing_user_userguide/index.html#document-13-nsb-overview
-
-SampleVNF Test Infrastructure (NSB (Yardstick_NSB_))in yardstick helps to facilitate
-consistent/repeatable methodologies for characterizing & validating the
-sample VNFs (:term:`VNF`) through OPEN SOURCE VNF approximations.
-
-Network Service Benchmarking in yardstick framework follows ETSI GS NFV-TST001_
-to verify/characterize both :term:`NFVI` & :term:`VNF`
-
-The document ETSI GS NFV-TST001_, "Pre-deployment Testing; Report on Validation
-of NFV Environments and Services", recommends methods for pre-deployment
-testing of the functional components of an NFV environment.
+When executing the tests, traffic will be send between 2 or more PROX VMs and all metrics
+will be collected in the Xtesting database.
+The placement of the test VMs (in which the PROX tool is running), can be controlled by
+Heat stacks, but can also be done through other means. This will be explained in the chapter
+covering the PROX instance deployment, and needs to be done prior to the test execution.
-The SampleVNF project implements the methodology described in chapter 13 of Yardstick_NSB_,
-"Pre-deployment validation of NFV infrastructure".
+The PROX tool is a DPDK based application optimized for high throughput packet handling.
+As such, we will not measure limitations imposed by the tool, but the capacity of the
+NFVI. In the rare case that the PROX tool would impose a limit, a warning will be logged.
-The methodology consists in decomposing the typical :term:`VNF` work-load
-performance metrics into a number of characteristics/performance vectors, which
-each can be represented by distinct test-cases.
-
-.. seealso:: SampleVNFtst_ for material on alignment ETSI TST001 and SampleVNF.
+ETSI-NFV
+========
+The document ETSI GS NFV-TST009_, "Specification of Networking Benchmarks and
+Measurement Methods for NFVI", specifies vendor-agnostic definitions of performance
+metrics and the associated methods of measurement for Benchmarking networks supported
+in the NFVI. Throughput, latency, packet loss and delay variation will be measured.
+The delay variation is not represented by the Frame Delay Variation (FDV) as defined in
+the specification, but by the average latency, the 99 percentile latency, the maximum
+latency and the complete latency distribution histogram.
Metrics
=======
-The metrics, as defined by ETSI GS NFV-TST001, are shown in
-:ref:`Table1 <table2_1>`.
+The metrics, as reported by the tool, and aligned with the definitions in ETSI GS NFV-TST009_,
+are shown in :ref:`Table1 <table2_1>`.
.. _table2_1:
-**Table 1 - Performance/Speed Metrics**
-
-+---------+-------------------------------------------------------------------+
-| Category| Performance/Speed |
-| | |
-+---------+-------------------------------------------------------------------+
-| Network | * Throughput per NFVI node (frames/byte per second) |
-| | * Throughput provided to a VM (frames/byte per second) |
-| | * Latency per traffic flow |
-| | * Latency between VMs |
-| | * Latency between NFVI nodes |
-| | * Packet delay variation (jitter) between VMs |
-| | * Packet delay variation (jitter) between NFVI nodes |
-| | * RFC 3511 benchmark |
-| | |
-+---------+-------------------------------------------------------------------+
+**Table 1 - Network Metrics**
+
++-----------------+---------------------------------------------------------------+
+| Measurement | Description |
+| | |
++-----------------+---------------------------------------------------------------+
+| Throughput | Maximum number of traffic that can be sent between 2 VM |
+| | instances, within the allowed packet loss requirements. |
+| | Results are expressed in Mpps and in Gb/s |
++-----------------+---------------------------------------------------------------+
+| Latency | 99 percentile Round trip latency expressed in micro-seconds |
+| | Note that you can also specify the n-th percentile |
++-----------------+---------------------------------------------------------------+
+| Delay Variation | Average latency, maximum latency and the latency histogram |
++-----------------+---------------------------------------------------------------+
+| Loss | Packets per seconds that were lost on their round trip between|
+| | VMs. Total packet loss numbers are also reported |
++-----------------+---------------------------------------------------------------+
.. note:: The description in this OPNFV document is intended as a reference for
users to understand the scope of the SampleVNF Project and the
deliverables of the SampleVNF framework. For complete description of
the methodology, please refer to the ETSI document.
-
-.. rubric:: Footnotes
-.. [1] To be included in future deliveries.
-