diff options
Diffstat (limited to 'docs/testing/developer')
21 files changed, 0 insertions, 9297 deletions
diff --git a/docs/testing/developer/design/LICENSE b/docs/testing/developer/design/LICENSE deleted file mode 100644 index 7bc572ce..00000000 --- a/docs/testing/developer/design/LICENSE +++ /dev/null @@ -1,2 +0,0 @@ -This work is licensed under a Creative Commons Attribution 4.0 International License. -http://creativecommons.org/licenses/by/4.0 diff --git a/docs/testing/developer/design/factory_and_loader.png b/docs/testing/developer/design/factory_and_loader.png Binary files differdeleted file mode 100644 index 290c0af6..00000000 --- a/docs/testing/developer/design/factory_and_loader.png +++ /dev/null diff --git a/docs/testing/developer/design/traffic_controller.png b/docs/testing/developer/design/traffic_controller.png Binary files differdeleted file mode 100644 index 598296ec..00000000 --- a/docs/testing/developer/design/traffic_controller.png +++ /dev/null diff --git a/docs/testing/developer/design/trafficgen_integration_guide.rst b/docs/testing/developer/design/trafficgen_integration_guide.rst deleted file mode 100644 index 382cedcb..00000000 --- a/docs/testing/developer/design/trafficgen_integration_guide.rst +++ /dev/null @@ -1,238 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Intel Corporation, AT&T and others. - -=================================== -Traffic Generator Integration Guide -=================================== - -Intended Audience -================= - -This document is intended to aid those who want to integrate new traffic -generator into the vsperf code. It is expected, that reader has already -read generic part of :ref:`vsperf-design`. - -Let us create a sample traffic generator called **sample_tg**, step by step. - -Step 1 - create a directory -=========================== - -Implementation of trafficgens is located at tools/pkt_gen/ directory, -where every implementation has its dedicated sub-directory. It is -required to create a new directory for new traffic generator -implementations. - -E.g. - -.. code-block:: console - - $ mkdir tools/pkt_gen/sample_tg - -Step 2 - create a trafficgen module -=================================== - -Every trafficgen class must inherit from generic **ITrafficGenerator** -interface class. VSPERF during its initialization scans content of pkt_gen -directory for all python modules, that inherit from **ITrafficGenerator**. These -modules are automatically added into the list of supported traffic generators. - -Example: - -Let us create a draft of tools/pkt_gen/sample_tg/sample_tg.py module. - -.. code-block:: python - - from tools.pkt_gen import trafficgen - - class SampleTG(trafficgen.ITrafficGenerator): - """ - A sample traffic generator implementation - """ - pass - -VSPERF is immediately aware of the new class: - -.. code-block:: console - - $ ./vsperf --list-trafficgen - -Output should look like: - -.. code-block:: console - - Classes derived from: ITrafficGenerator - ====== - - * Ixia: A wrapper around the IXIA traffic generator. - - * IxNet: A wrapper around IXIA IxNetwork applications. - - * Dummy: A dummy traffic generator whose data is generated by the user. - - * SampleTG: A sample traffic generator implementation - - * TestCenter: Spirent TestCenter - - -Step 3 - configuration -====================== - -All configuration values, required for correct traffic generator function, are passed -from VSPERF to the traffic generator in a dictionary. Default values shared among -all traffic generators are defined in **conf/03_traffic.conf** within **TRAFFIC** -dictionary. Default values are loaded by **ITrafficGenerator** interface class -automatically, so it is not needed to load them explicitly. In case that there are -any traffic generator specific default values, then they should be set within class -specific **__init__** function. - -VSPERF passes test specific configuration within **traffic** dictionary to every -start and send function. So implementation of these functions must ensure, -that default values are updated with the testcase specific values. Proper merge -of values is assured by call of **merge_spec** function from **conf** module. - -Example of **merge_spec** usage in **tools/pkt_gen/sample_tg/sample_tg.py** module: - -.. code-block:: python - - from conf import merge_spec - - def start_rfc2544_throughput(self, traffic=None, duration=30): - self._params = {} - self._params['traffic'] = self.traffic_defaults.copy() - if traffic: - self._params['traffic'] = merge_spec( - self._params['traffic'], traffic) - - -Step 4 - generic functions -========================== - -There are some generic functions, which every traffic generator should provide. -Although these functions are mainly optional, at least empty implementation must -be provided. This is required, so that developer is explicitly aware of these -functions. - -The **connect** function is called from the traffic generator controller from its -**__enter__** method. This function should assure proper connection initialization -between DUT and traffic generator. In case, that such implementation is not needed, -empty implementation is required. - -The **disconnect** function should perform clean up of any connection specific -actions called from the **connect** function. - -Example in **tools/pkt_gen/sample_tg/sample_tg.py** module: - -.. code-block:: python - - def connect(self): - pass - - def disconnect(self): - pass - -.. _step-5-supported-traffic-types: - -Step 5 - supported traffic types -================================ - -Currently VSPERF supports three different types of tests for traffic generators, -these are identified in vsperf through the traffic type, which include: - - * RFC2544 throughput - Send fixed size packets at different rates, using - traffic configuration, until minimum rate at which no packet loss is - detected is found. Methods with its implementation have suffix - **_rfc2544_throughput**. - - * RFC2544 back2back - Send fixed size packets at a fixed rate, using traffic - configuration, for specified time interval. Methods with its - implementation have suffix **_rfc2544_back2back**. - - * continuous flow - Send fixed size packets at given framerate, using traffic - configuration, for specified time interval. Methods with its - implementation have suffix **_cont_traffic**. - -In general, both synchronous and asynchronous interfaces must be implemented -for each traffic type. Synchronous functions start with prefix **send_**. -Asynchronous with prefixes **start_** and **wait_** in case of throughput -and back2back and **start_** and **stop_** in case of continuous traffic type. - -Example of synchronous interfaces: - -.. code-block:: python - - def send_rfc2544_throughput(self, traffic=None, tests=1, duration=20, - lossrate=0.0): - def send_rfc2544_back2back(self, traffic=None, tests=1, duration=20, - lossrate=0.0): - def send_cont_traffic(self, traffic=None, duration=20): - -Example of asynchronous interfaces: - -.. code-block:: python - - def start_rfc2544_throughput(self, traffic=None, tests=1, duration=20, - lossrate=0.0): - def wait_rfc2544_throughput(self): - - def start_rfc2544_back2back(self, traffic=None, tests=1, duration=20, - lossrate=0.0): - def wait_rfc2544_back2back(self): - - def start_cont_traffic(self, traffic=None, duration=20): - def stop_cont_traffic(self): - -Description of parameters used by **send**, **start**, **wait** and **stop** -functions: - - * param **traffic**: A dictionary with detailed definition of traffic - pattern. It contains following parameters to be implemented by - traffic generator. - - Note: Traffic dictionary has also virtual switch related parameters, - which are not listed below. - - Note: There are parameters specific to testing of tunnelling protocols, - which are discussed in detail at :ref:`integration-tests` userguide. - - * param **traffic_type**: One of the supported traffic types, - e.g. **rfc2544_throughput**, **rfc2544_continuous** - or **rfc2544_back2back**. - * param **frame_rate**: Defines desired percentage of frame - rate used during continuous stream tests. - * param **bidir**: Specifies if generated traffic will be full-duplex - (true) or half-duplex (false). - * param **multistream**: Defines number of flows simulated by traffic - generator. Value 0 disables MultiStream feature. - * param **stream_type**: Stream Type defines ISO OSI network layer - used for simulation of multiple streams. - Supported values: - - * **L2** - iteration of destination MAC address - * **L3** - iteration of destination IP address - * **L4** - iteration of destination port of selected transport protocol - - * param **l2**: A dictionary with data link layer details, e.g. **srcmac**, - **dstmac** and **framesize**. - * param **l3**: A dictionary with network layer details, e.g. **srcip**, - **dstip** and **proto**. - * param **l3**: A dictionary with transport layer details, e.g. **srcport**, - **dstport**. - * param **vlan**: A dictionary with vlan specific parameters, - e.g. **priority**, **cfi**, **id** and vlan on/off switch **enabled**. - - * param **tests**: Number of times the test is executed. - * param **duration**: Duration of continuous test or per iteration duration - in case of RFC2544 throughput or back2back traffic types. - * param **lossrate**: Acceptable lossrate percentage. - -Step 6 - passing back results -============================= - -It is expected that methods **send**, **wait** and **stop** will return -values measured by traffic generator within a dictionary. Dictionary keys -are defined in **ResultsConstants** implemented in -**core/results/results_constants.py**. Please check sections for RFC2544 -Throughput & Continuous and for Back2Back. The same key names should -be used by all traffic generator implementations. - diff --git a/docs/testing/developer/design/vsperf.png b/docs/testing/developer/design/vsperf.png Binary files differdeleted file mode 100644 index 4af2ac62..00000000 --- a/docs/testing/developer/design/vsperf.png +++ /dev/null diff --git a/docs/testing/developer/design/vswitchperf_design.rst b/docs/testing/developer/design/vswitchperf_design.rst deleted file mode 100644 index da7ec6fd..00000000 --- a/docs/testing/developer/design/vswitchperf_design.rst +++ /dev/null @@ -1,870 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Intel Corporation, AT&T and others. - -.. _vsperf-design: - -====================== -VSPERF Design Document -====================== - -Intended Audience -================= - -This document is intended to aid those who want to modify the vsperf code. Or -to extend it - for example to add support for new traffic generators, -deployment scenarios and so on. - -Usage -===== - -Example Connectivity to DUT ---------------------------- - -Establish connectivity to the VSPERF DUT Linux host, such as the DUT in Pod 3, -by following the steps in `Testbed POD3 -<https://wiki.opnfv.org/get_started/pod_3_-_characterize_vswitch_performance>`__ - -The steps cover booking the DUT and establishing the VSPERF environment. - -Example Command Lines ---------------------- - -List all the cli options: - -.. code-block:: console - - $ ./vsperf -h - -Run all tests that have ``tput`` in their name - ``phy2phy_tput``, ``pvp_tput`` etc.: - -.. code-block:: console - - $ ./vsperf --tests 'tput' - -As above but override default configuration with settings in '10_custom.conf'. -This is useful as modifying configuration directly in the configuration files -in ``conf/NN_*.py`` shows up as changes under git source control: - -.. code-block:: console - - $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf --tests 'tput' - -Override specific test parameters. Useful for shortening the duration of tests -for development purposes: - -.. code-block:: console - - $ ./vsperf --test-params 'TRAFFICGEN_DURATION=10;TRAFFICGEN_RFC2544_TESTS=1;' \ - 'TRAFFICGEN_PKT_SIZES=(64,)' pvp_tput - -Typical Test Sequence -===================== - -This is a typical flow of control for a test. - -.. image:: vsperf.png - -.. _design-configuration: - -Configuration -============= - -The conf package contains the configuration files (``*.conf``) for all system -components, it also provides a ``settings`` object that exposes all of these -settings. - -Settings are not passed from component to component. Rather they are available -globally to all components once they import the conf package. - -.. code-block:: python - - from conf import settings - ... - log_file = settings.getValue('LOG_FILE_DEFAULT') - -Settings files (``*.conf``) are valid python code so can be set to complex -types such as lists and dictionaries as well as scalar types: - -.. code-block:: python - - first_packet_size = settings.getValue('PACKET_SIZE_LIST')[0] - -Configuration Procedure and Precedence --------------------------------------- - -Configuration files follow a strict naming convention that allows them to be -processed in a specific order. All the .conf files are named ``NN_name.conf``, -where NN is a decimal number. The files are processed in order from 00_name.conf -to 99_name.conf so that if the name setting is given in both a lower and higher -numbered conf file then the higher numbered file is the effective setting as it -is processed after the setting in the lower numbered file. - -The values in the file specified by ``--conf-file`` takes precedence over all -the other configuration files and does not have to follow the naming -convention. - -.. _paths-documentation: - -Configuration of PATHS dictionary ---------------------------------- - -VSPERF uses external tools like Open vSwitch and Qemu for execution of testcases. These -tools may be downloaded and built automatically (see :ref:`vsperf-installation-script`) -or installed manually by user from binary packages. It is also possible to use a combination -of both approaches, but it is essential to correctly set paths to all required tools. -These paths are stored within a PATHS dictionary, which is evaluated before execution -of each testcase, in order to setup testcase specific environment. Values selected for testcase -execution are internally stored inside TOOLS dictionary, which is used by VSPERF to execute -external tools, load kernel modules, etc. - -The default configuration of PATHS dictionary is spread among three different configuration files -to follow logical grouping of configuration options. Basic description of PATHS dictionary -is placed inside ``conf/00_common.conf``. The configuration specific to DPDK and vswitches -is located at ``conf/02_vswitch.conf``. The last part related to the Qemu is defined inside -``conf/04_vnf.conf``. Default configuration values can be used in case, that all required -tools were downloaded and built automatically by vsperf itself. In case, that some of -tools were installed manually from binary packages, then it will be necessary to modify -the content of PATHS dictionary accordingly. - -Dictionary has a specific section of configuration options for every tool type, it means: - - * ``PATHS['vswitch']`` - contains a separete dictionary for each of vswitches supported by VSPEF - - Example: - - .. code-block:: python - - PATHS['vswitch'] = { - 'OvsDpdkVhost': { ... }, - 'OvsVanilla' : { ... }, - ... - } - - * ``PATHS['dpdk']`` - contains paths to the dpdk sources, kernel modules and tools (e.g. testpmd) - - Example: - - .. code-block:: python - - PATHS['dpdk'] = { - 'type' : 'src', - 'src': { - 'path': os.path.join(ROOT_DIR, 'src/dpdk/dpdk/'), - 'modules' : ['uio', os.path.join(RTE_TARGET, 'kmod/igb_uio.ko')], - 'bind-tool': 'tools/dpdk*bind.py', - 'testpmd': os.path.join(RTE_TARGET, 'app', 'testpmd'), - }, - ... - } - - * ``PATHS['qemu']`` - contains paths to the qemu sources and executable file - - Example: - - .. code-block:: python - - PATHS['qemu'] = { - 'type' : 'bin', - 'bin': { - 'qemu-system': 'qemu-system-x86_64' - }, - ... - } - -Every section specific to the particular vswitch, dpdk or qemu may contain following types -of configuration options: - - * option ``type`` - is a string, which defines the type of configured paths ('src' or 'bin') - to be selected for a given section: - - * value ``src`` means, that VSPERF will use vswitch, DPDK or QEMU built from sources - e.g. by execution of ``systems/build_base_machine.sh`` script during VSPERF - installation - - * value ``bin`` means, that VSPERF will use vswitch, DPDK or QEMU binaries installed - directly in the operating system, e.g. via OS specific packaging system - - * option ``path`` - is a string with a valid system path; Its content is checked for - existence, prefixed with section name and stored into TOOLS for later use - e.g. ``TOOLS['dpdk_src']`` or ``TOOLS['vswitch_src']`` - - * option ``modules`` - is list of strings with names of kernel modules; Every module name - from given list is checked for a '.ko' suffix. In case that it matches and if it is not - an absolute path to the module, then module name is prefixed with value of ``path`` - option defined for the same section - - Example: - - .. code-block:: python - - """ - snippet of PATHS definition from the configuration file: - """ - PATHS['vswitch'] = { - 'OvsVanilla' = { - 'type' : 'src', - 'src': { - 'path': '/tmp/vsperf/src_vanilla/ovs/ovs/', - 'modules' : ['datapath/linux/openvswitch.ko'], - ... - }, - ... - } - ... - } - - """ - Final content of TOOLS dictionary used during runtime: - """ - TOOLS['vswitch_modules'] = ['/tmp/vsperf/src_vanilla/ovs/ovs/datapath/linux/openvswitch.ko'] - - * all other options are strings with names and paths to specific tools; If a given string - contains a relative path and option ``path`` is defined for a given section, then string - content will be prefixed with content of the ``path``. Otherwise the name of the tool will be - searched within standard system directories. In case that filename contains OS specific - wildcards, then they will be expanded to the real path. At the end of the processing, every - absolute path will be checked for its existence. In case that temporary path (i.e. path with - a ``_tmp`` suffix) does not exist, then log will be written and vsperf will continue. If any - other path will not exist, then vsperf execution will be terminated with a runtime error. - - Example: - - .. code-block:: python - - """ - snippet of PATHS definition from the configuration file: - """ - PATHS['vswitch'] = { - 'OvsDpdkVhost': { - 'type' : 'src', - 'src': { - 'path': '/tmp/vsperf/src_vanilla/ovs/ovs/', - 'ovs-vswitchd': 'vswitchd/ovs-vswitchd', - 'ovsdb-server': 'ovsdb/ovsdb-server', - ... - } - ... - } - ... - } - - """ - Final content of TOOLS dictionary used during runtime: - """ - TOOLS['ovs-vswitchd'] = '/tmp/vsperf/src_vanilla/ovs/ovs/vswitchd/ovs-vswitchd' - TOOLS['ovsdb-server'] = '/tmp/vsperf/src_vanilla/ovs/ovs/ovsdb/ovsdb-server' - -Note: In case that ``bin`` type is set for DPDK, then ``TOOLS['dpdk_src']`` will be set to -the value of ``PATHS['dpdk']['src']['path']``. The reason is, that VSPERF uses downloaded -DPDK sources to copy DPDK and testpmd into the GUEST, where testpmd is built. In case, -that DPDK sources are not available, then vsperf will continue with test execution, -but testpmd can't be used as a guest loopback. This is useful in case, that other guest -loopback applications (e.g. buildin or l2fwd) are used. - -Note: In case of RHEL 7.3 OS usage, binary package configuration is required -for Vanilla OVS tests. With the installation of a supported rpm for OVS there is -a section in the ``conf\10_custom.conf`` file that can be used. - -.. _configuration-of-traffic-dictionary: - -Configuration of TRAFFIC dictionary ------------------------------------ - -TRAFFIC dictionary is used for configuration of traffic generator. Default values -can be found in configuration file ``conf/03_traffic.conf``. These default values -can be modified by (first option has the highest priorty): - - 1. ``Parameters`` section of testcase defintion - 2. command line options specified by ``--test-params`` argument - 3. custom configuration file - -It is to note, that in case of option 1 and 2, it is possible to specify only -values, which should be changed. In case of custom configuration file, it is -required to specify whole ``TRAFFIC`` dictionary with its all values or explicitly -call and update() method of ``TRAFFIC`` dictionary. - -Detailed description of ``TRAFFIC`` dictionary items follows: - -.. code-block:: console - - 'traffic_type' - One of the supported traffic types. - E.g. rfc2544_throughput, rfc2544_back2back - or rfc2544_continuous - Data type: str - Default value: "rfc2544_throughput". - 'bidir' - Specifies if generated traffic will be full-duplex (True) - or half-duplex (False) - Data type: str - Supported values: "True", "False" - Default value: "False". - 'frame_rate' - Defines desired percentage of frame rate used during - continuous stream tests. - Data type: int - Default value: 100. - 'multistream' - Defines number of flows simulated by traffic generator. - Value 0 disables multistream feature - Data type: int - Supported values: 0-65535 - Default value: 0. - 'stream_type' - Stream type is an extension of the "multistream" feature. - If multistream is disabled, then stream type will be - ignored. Stream type defines ISO OSI network layer used - for simulation of multiple streams. - Data type: str - Supported values: - "L2" - iteration of destination MAC address - "L3" - iteration of destination IP address - "L4" - iteration of destination port - of selected transport protocol - Default value: "L4". - 'pre_installed_flows' - - Pre-installed flows is an extension of the "multistream" - feature. If enabled, it will implicitly insert a flow - for each stream. If multistream is disabled, then - pre-installed flows will be ignored. - Note: It is supported only for p2p deployment scenario. - Data type: str - Supported values: - "Yes" - flows will be inserted into OVS - "No" - flows won't be inserted into OVS - Default value: "No". - 'flow_type' - Defines flows complexity. - Data type: str - Supported values: - "port" - flow is defined by ingress ports - "IP" - flow is defined by ingress ports - and src and dst IP addresses - Default value: "port" - 'l2' - A dictionary with l2 network layer details. Supported - values are: - 'srcmac' - Specifies source MAC address filled by traffic generator. - NOTE: It can be modified by vsperf in some scenarios. - Data type: str - Default value: "00:00:00:00:00:00". - 'dstmac' - Specifies destination MAC address filled by traffic generator. - NOTE: It can be modified by vsperf in some scenarios. - Data type: str - Default value: "00:00:00:00:00:00". - 'framesize' - Specifies default frame size. This value should not be - changed directly. It will be overridden during testcase - execution by values specified by list TRAFFICGEN_PKT_SIZES. - Data type: int - Default value: 64 - 'l3' - A dictionary with l3 network layer details. Supported - values are: - 'srcip' - Specifies source MAC address filled by traffic generator. - NOTE: It can be modified by vsperf in some scenarios. - Data type: str - Default value: "1.1.1.1". - 'dstip' - Specifies destination MAC address filled by traffic generator. - NOTE: It can be modified by vsperf in some scenarios. - Data type: str - Default value: "90.90.90.90". - 'proto' - Specifies deflaut protocol type. - Please check particular traffic generator implementation - for supported protocol types. - Data type: str - Default value: "udp". - 'l4' - A dictionary with l4 network layer details. Supported - values are: - 'srcport' - Specifies source port of selected transport protocol. - NOTE: It can be modified by vsperf in some scenarios. - Data type: int - Default value: 3000 - 'dstport' - Specifies destination port of selected transport protocol. - NOTE: It can be modified by vsperf in some scenarios. - Data type: int - Default value: 3001 - 'vlan' - A dictionary with vlan encapsulation details. Supported - values are: - 'enabled' - Specifies if vlan encapsulation should be enabled or - disabled. - Data type: bool - Default value: False - 'id' - Specifies vlan id. - Data type: int (NOTE: must fit to 12 bits) - Default value: 0 - 'priority' - Specifies a vlan priority (PCP header field). - Data type: int (NOTE: must fit to 3 bits) - Default value: 0 - 'cfi' - Specifies if frames can or cannot be dropped during - congestion (DEI header field). - Data type: int (NOTE: must fit to 1 bit) - Default value: 0 - -.. _configuration-of-guest-options: - -Configuration of GUEST options ------------------------------- - -VSPERF is able to setup scenarios involving a number of VMs in series or in parallel. -All configuration options related to a particular VM instance are defined as -lists and prefixed with ``GUEST_`` label. It is essential, that there is enough -items in all ``GUEST_`` options to cover all VM instances involved in the test. -In case there is not enough items, then VSPERF will use the first item of -particular ``GUEST_`` option to expand the list to required length. - -Example of option expansion for 4 VMs: - - .. code-block:: python - - """ - Original values: - """ - GUEST_SMP = ['2'] - GUEST_MEMORY = ['2048', '4096'] - - """ - Values after automatic expansion: - """ - GUEST_SMP = ['2', '2', '2', '2'] - GUEST_MEMORY = ['2048', '4096', '2048', '2048'] - - -First option can contain macros starting with ``#`` to generate VM specific values. -These macros can be used only for options of ``list`` or ``str`` types with ``GUEST_`` -prefix. - -Example of macros and their expnasion for 2 VMs: - - .. code-block:: python - - """ - Original values: - """ - GUEST_SHARE_DIR = ['/tmp/qemu#VMINDEX_share'] - GUEST_BRIDGE_IP = ['#IP(1.1.1.5)/16'] - - """ - Values after automatic expansion: - """ - GUEST_SHARE_DIR = ['/tmp/qemu0_share', '/tmp/qemu1_share'] - GUEST_BRIDGE_IP = ['1.1.1.5/16', '1.1.1.6/16'] - -Additional examples are available at ``04_vnf.conf``. - -Note: In case, that macro is detected in the first item of the list, then -all other items are ignored and list content is created automatically. - -Multiple macros can be used inside one configuration option definition, but macros -cannot be used inside other macros. The only exception is macro ``#VMINDEX``, which -is expanded first and thus it can be used inside other macros. - -Following macros are supported: - - * ``#VMINDEX`` - it is replaced by index of VM being executed; This macro - is expanded first, so it can be used inside other macros. - - Example: - - .. code-block:: python - - GUEST_SHARE_DIR = ['/tmp/qemu#VMINDEX_share'] - - * ``#MAC(mac_address[, step])`` - it will iterate given ``mac_address`` - with optional ``step``. In case that step is not defined, then it is set to 1. - It means, that first VM will use the value of ``mac_address``, second VM - value of ``mac_address`` increased by ``step``, etc. - - Example: - - .. code-block:: python - - GUEST_NICS = [[{'mac' : '#MAC(00:00:00:00:00:01,2)'}]] - - * ``#IP(ip_address[, step])`` - it will iterate given ``ip_address`` - with optional ``step``. In case that step is not defined, then it is set to 1. - It means, that first VM will use the value of ``ip_address``, second VM - value of ``ip_address`` increased by ``step``, etc. - - Example: - - .. code-block:: python - - GUEST_BRIDGE_IP = ['#IP(1.1.1.5)/16'] - - * ``#EVAL(expression)`` - it will evaluate given ``expression`` as python code; - Only simple expressions should be used. Call of the functions is not supported. - - Example: - - .. code-block:: python - - GUEST_CORE_BINDING = [('#EVAL(6+2*#VMINDEX)', '#EVAL(7+2*#VMINDEX)')] - -Other Configuration -------------------- - -``conf.settings`` also loads configuration from the command line and from the environment. - -.. _pxp-deployment: - -PXP Deployment -============== - -Every testcase uses one of the supported deployment scenarios to setup test environment. -The controller responsible for a given scenario configures flows in the vswitch to route -traffic among physical interfaces connected to the traffic generator and virtual -machines. VSPERF supports several deployments including PXP deployment, which can -setup various scenarios with multiple VMs. - -These scenarios are realized by VswitchControllerPXP class, which can configure and -execute given number of VMs in serial or parallel configurations. Every VM can be -configured with just one or an even number of interfaces. In case that VM has more than -2 interfaces, then traffic is properly routed among pairs of interfaces. - -Example of traffic routing for VM with 4 NICs in serial configuration: - -.. code-block:: console - - +------------------------------------------+ - | VM with 4 NICs | - | +---------------+ +---------------+ | - | | Application | | Application | | - | +---------------+ +---------------+ | - | ^ | ^ | | - | | v | v | - | +---------------+ +---------------+ | - | | logical ports | | logical ports | | - | | 0 1 | | 2 3 | | - +--+---------------+----+---------------+--+ - ^ : ^ : - | | | | - : v : v - +-----------+---------------+----+---------------+----------+ - | vSwitch | 0 1 | | 2 3 | | - | | logical ports | | logical ports | | - | previous +---------------+ +---------------+ next | - | VM or PHY ^ | ^ | VM or PHY| - | port -----+ +------------+ +---> port | - +-----------------------------------------------------------+ - -It is also possible to define different number of interfaces for each VM to better -simulate real scenarios. - -Example of traffic routing for 2 VMs in serial configuration, where 1st VM has -4 NICs and 2nd VM 2 NICs: - -.. code-block:: console - - +------------------------------------------+ +---------------------+ - | 1st VM with 4 NICs | | 2nd VM with 2 NICs | - | +---------------+ +---------------+ | | +---------------+ | - | | Application | | Application | | | | Application | | - | +---------------+ +---------------+ | | +---------------+ | - | ^ | ^ | | | ^ | | - | | v | v | | | v | - | +---------------+ +---------------+ | | +---------------+ | - | | logical ports | | logical ports | | | | logical ports | | - | | 0 1 | | 2 3 | | | | 0 1 | | - +--+---------------+----+---------------+--+ +--+---------------+--+ - ^ : ^ : ^ : - | | | | | | - : v : v : v - +-----------+---------------+----+---------------+-------+---------------+----------+ - | vSwitch | 0 1 | | 2 3 | | 4 5 | | - | | logical ports | | logical ports | | logical ports | | - | previous +---------------+ +---------------+ +---------------+ next | - | VM or PHY ^ | ^ | ^ | VM or PHY| - | port -----+ +------------+ +---------------+ +----> port | - +-----------------------------------------------------------------------------------+ - -The number of VMs involved in the test and the type of their connection is defined -by deployment name as follows: - - * ``pvvp[number]`` - configures scenario with VMs connected in series with - optional ``number`` of VMs. In case that ``number`` is not specified, then - 2 VMs will be used. - - Example of 2 VMs in a serial configuration: - - .. code-block:: console - - +----------------------+ +----------------------+ - | 1st VM | | 2nd VM | - | +---------------+ | | +---------------+ | - | | Application | | | | Application | | - | +---------------+ | | +---------------+ | - | ^ | | | ^ | | - | | v | | | v | - | +---------------+ | | +---------------+ | - | | logical ports | | | | logical ports | | - | | 0 1 | | | | 0 1 | | - +---+---------------+--+ +---+---------------+--+ - ^ : ^ : - | | | | - : v : v - +---+---------------+---------+---------------+--+ - | | 0 1 | | 3 4 | | - | | logical ports | vSwitch | logical ports | | - | +---------------+ +---------------+ | - | ^ | ^ | | - | | +-----------------+ v | - | +----------------------------------------+ | - | | physical ports | | - | | 0 1 | | - +---+----------------------------------------+---+ - ^ : - | | - : v - +------------------------------------------------+ - | | - | traffic generator | - | | - +------------------------------------------------+ - - * ``pvpv[number]`` - configures scenario with VMs connected in parallel with - optional ``number`` of VMs. In case that ``number`` is not specified, then - 2 VMs will be used. Multistream feature is used to route traffic to particular - VMs (or NIC pairs of every VM). It means, that VSPERF will enable multistream - feaure and sets the number of streams to the number of VMs and their NIC - pairs. Traffic will be dispatched based on Stream Type, i.e. by UDP port, - IP address or MAC address. - - Example of 2 VMs in a parallel configuration, where traffic is dispatched - based on the UDP port. - - .. code-block:: console - - +----------------------+ +----------------------+ - | 1st VM | | 2nd VM | - | +---------------+ | | +---------------+ | - | | Application | | | | Application | | - | +---------------+ | | +---------------+ | - | ^ | | | ^ | | - | | v | | | v | - | +---------------+ | | +---------------+ | - | | logical ports | | | | logical ports | | - | | 0 1 | | | | 0 1 | | - +---+---------------+--+ +---+---------------+--+ - ^ : ^ : - | | | | - : v : v - +---+---------------+---------+---------------+--+ - | | 0 1 | | 3 4 | | - | | logical ports | vSwitch | logical ports | | - | +---------------+ +---------------+ | - | ^ | ^ : | - | | ......................: : | - | UDP | UDP : | : | - | port| port: +--------------------+ : | - | 0 | 1 : | : | - | | : v v | - | +----------------------------------------+ | - | | physical ports | | - | | 0 1 | | - +---+----------------------------------------+---+ - ^ : - | | - : v - +------------------------------------------------+ - | | - | traffic generator | - | | - +------------------------------------------------+ - - -PXP deployment is backward compatible with PVP deployment, where ``pvp`` is -an alias for ``pvvp1`` and it executes just one VM. - -The number of interfaces used by VMs is defined by configuration option -``GUEST_NICS_NR``. In case that more than one pair of interfaces is defined -for VM, then: - - * for ``pvvp`` (serial) scenario every NIC pair is connected in serial - before connection to next VM is created - * for ``pvpv`` (parallel) scenario every NIC pair is directly connected - to the physical ports and unique traffic stream is assigned to it - -Examples: - - * Deployment ``pvvp10`` will start 10 VMs and connects them in series - * Deployment ``pvpv4`` will start 4 VMs and connects them in parallel - * Deployment ``pvpv1`` and GUEST_NICS_NR = [4] will start 1 VM with - 4 interfaces and every NIC pair is directly connected to the - physical ports - * Deployment ``pvvp`` and GUEST_NICS_NR = [2, 4] will start 2 VMs; - 1st VM will have 2 interfaces and 2nd VM 4 interfaces. These interfaces - will be connected in serial, i.e. traffic will flow as follows: - PHY1 -> VM1_1 -> VM1_2 -> VM2_1 -> VM2_2 -> VM2_3 -> VM2_4 -> PHY2 - -Note: In case that only 1 or more than 2 NICs are configured for VM, -then ``testpmd`` should be used as forwarding application inside the VM. -As it is able to forward traffic between multiple VM NIC pairs. - -Note: In case of ``linux_bridge``, all NICs are connected to the same -bridge inside the VM. - -VM, vSwitch, Traffic Generator Independence -=========================================== - -VSPERF supports different vSwithes, Traffic Generators, VNFs -and Forwarding Applications by using standard object-oriented polymorphism: - - * Support for vSwitches is implemented by a class inheriting from IVSwitch. - * Support for Traffic Generators is implemented by a class inheriting from - ITrafficGenerator. - * Support for VNF is implemented by a class inheriting from IVNF. - * Support for Forwarding Applications is implemented by a class inheriting - from IPktFwd. - -By dealing only with the abstract interfaces the core framework can support -many implementations of different vSwitches, Traffic Generators, VNFs -and Forwarding Applications. - -IVSwitch --------- - -.. code-block:: python - - class IVSwitch: - start(self) - stop(self) - add_switch(switch_name) - del_switch(switch_name) - add_phy_port(switch_name) - add_vport(switch_name) - get_ports(switch_name) - del_port(switch_name, port_name) - add_flow(switch_name, flow) - del_flow(switch_name, flow=None) - -ITrafficGenerator ------------------ - -.. code-block:: python - - class ITrafficGenerator: - connect() - disconnect() - - send_burst_traffic(traffic, numpkts, time, framerate) - - send_cont_traffic(traffic, time, framerate) - start_cont_traffic(traffic, time, framerate) - stop_cont_traffic(self): - - send_rfc2544_throughput(traffic, tests, duration, lossrate) - start_rfc2544_throughput(traffic, tests, duration, lossrate) - wait_rfc2544_throughput(self) - - send_rfc2544_back2back(traffic, tests, duration, lossrate) - start_rfc2544_back2back(traffic, , tests, duration, lossrate) - wait_rfc2544_back2back() - -Note ``send_xxx()`` blocks whereas ``start_xxx()`` does not and must be followed by a subsequent call to ``wait_xxx()``. - -IVnf ----- - -.. code-block:: python - - class IVnf: - start(memory, cpus, - monitor_path, shared_path_host, - shared_path_guest, guest_prompt) - stop() - execute(command) - wait(guest_prompt) - execute_and_wait (command) - -IPktFwd --------- - - .. code-block:: python - - class IPktFwd: - start() - stop() - - -Controllers ------------ - -Controllers are used in conjunction with abstract interfaces as way -of decoupling the control of vSwtiches, VNFs, TrafficGenerators -and Forwarding Applications from other components. - -The controlled classes provide basic primitive operations. The Controllers -sequence and co-ordinate these primitive operation in to useful actions. For -instance the vswitch_controller_p2p can be used to bring any vSwitch (that -implements the primitives defined in IVSwitch) into the configuration required -by the Phy-to-Phy Deployment Scenario. - -In order to support a new vSwitch only a new implementation of IVSwitch needs -be created for the new vSwitch to be capable of fulfilling all the Deployment -Scenarios provided for by existing or future vSwitch Controllers. - -Similarly if a new Deployment Scenario is required it only needs to be written -once as a new vSwitch Controller and it will immediately be capable of -controlling all existing and future vSwitches in to that Deployment Scenario. - -Similarly the Traffic Controllers can be used to co-ordinate basic operations -provided by implementers of ITrafficGenerator to provide useful tests. Though -traffic generators generally already implement full test cases i.e. they both -generate suitable traffic and analyse returned traffic in order to implement a -test which has typically been predefined in an RFC document. However the -Traffic Controller class allows for the possibility of further enhancement - -such as iterating over tests for various packet sizes or creating new tests. - -Traffic Controller's Role -------------------------- - -.. image:: traffic_controller.png - - -Loader & Component Factory --------------------------- - -The working of the Loader package (which is responsible for *finding* arbitrary -classes based on configuration data) and the Component Factory which is -responsible for *choosing* the correct class for a particular situation - e.g. -Deployment Scenario can be seen in this diagram. - -.. image:: factory_and_loader.png - -Routing Tables -============== - -Vsperf uses a standard set of routing tables in order to allow tests to easily -mix and match Deployment Scenarios (PVP, P2P topology), Tuple Matching and -Frame Modification requirements. - -.. code-block:: console - - +--------------+ - | | - | Table 0 | table#0 - Match table. Flows designed to force 5 & 10 - | | tuple matches go here. - | | - +--------------+ - | - | - v - +--------------+ table#1 - Routing table. Flow entries to forward - | | packets between ports goes here. - | Table 1 | The chosen port is communicated to subsequent tables by - | | setting the metadata value to the egress port number. - | | Generally this table is set-up by by the - +--------------+ vSwitchController. - | - | - v - +--------------+ table#2 - Frame modification table. Frame modification - | | flow rules are isolated in this table so that they can - | Table 2 | be turned on or off without affecting the routing or - | | tuple-matching flow rules. This allows the frame - | | modification and tuple matching required by the tests - | | in the VSWITCH PERFORMANCE FOR TELCO NFV test - +--------------+ specification to be independent of the Deployment - | Scenario set up by the vSwitchController. - | - v - +--------------+ - | | - | Table 3 | table#3 - Egress table. Egress packets on the ports - | | setup in Table 1. - +--------------+ - - diff --git a/docs/testing/developer/requirements/LICENSE b/docs/testing/developer/requirements/LICENSE deleted file mode 100644 index 7bc572ce..00000000 --- a/docs/testing/developer/requirements/LICENSE +++ /dev/null @@ -1,2 +0,0 @@ -This work is licensed under a Creative Commons Attribution 4.0 International License. -http://creativecommons.org/licenses/by/4.0 diff --git a/docs/testing/developer/requirements/ietf_draft/LICENSE b/docs/testing/developer/requirements/ietf_draft/LICENSE deleted file mode 100644 index 7fc9ae14..00000000 --- a/docs/testing/developer/requirements/ietf_draft/LICENSE +++ /dev/null @@ -1,12 +0,0 @@ -Copyright (c) 2016 IETF Trust and the persons identified as the -document authors. All rights reserved. - -This document is subject to BCP 78 and the IETF Trust's Legal -Provisions Relating to IETF Documents -(http://trustee.ietf.org/license-info) in effect on the date of -publication of this document. Please review these documents -carefully, as they describe your rights and restrictions with respect -to this document. Code Components extracted from this document must -include Simplified BSD License text as described in Section 4.e of -the Trust Legal Provisions and are provided without warranty as -described in the Simplified BSD License. diff --git a/docs/testing/developer/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-00.xml b/docs/testing/developer/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-00.xml deleted file mode 100644 index 2259b23c..00000000 --- a/docs/testing/developer/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-00.xml +++ /dev/null @@ -1,1016 +0,0 @@ -<?xml version="1.0" encoding="US-ASCII"?> -<!DOCTYPE rfc SYSTEM "rfc2629.dtd"> -<?rfc toc="yes"?> -<?rfc tocompact="yes"?> -<?rfc tocdepth="3"?> -<?rfc tocindent="yes"?> -<?rfc symrefs="yes"?> -<?rfc sortrefs="yes"?> -<?rfc comments="yes"?> -<?rfc inline="yes"?> -<?rfc compact="yes"?> -<?rfc subcompact="no"?> -<rfc category="info" docName="draft-ietf-bmwg-vswitch-opnfv-00" - ipr="trust200902"> - <front> - <title abbrev="Benchmarking vSwitches">Benchmarking Virtual Switches in - OPNFV</title> - - <author fullname="Maryam Tahhan" initials="M." surname="Tahhan"> - <organization>Intel</organization> - - <address> - <postal> - <street/> - - <city/> - - <region/> - - <code/> - - <country/> - </postal> - - <phone/> - - <facsimile/> - - <email>maryam.tahhan@intel.com</email> - - <uri/> - </address> - </author> - - <author fullname="Billy O'Mahony" initials="B." surname="O'Mahony"> - <organization>Intel</organization> - - <address> - <postal> - <street/> - - <city/> - - <region/> - - <code/> - - <country/> - </postal> - - <phone/> - - <facsimile/> - - <email>billy.o.mahony@intel.com</email> - - <uri/> - </address> - </author> - - <author fullname="Al Morton" initials="A." surname="Morton"> - <organization>AT&T Labs</organization> - - <address> - <postal> - <street>200 Laurel Avenue South</street> - - <city>Middletown,</city> - - <region>NJ</region> - - <code>07748</code> - - <country>USA</country> - </postal> - - <phone>+1 732 420 1571</phone> - - <facsimile>+1 732 368 1192</facsimile> - - <email>acmorton@att.com</email> - - <uri>http://home.comcast.net/~acmacm/</uri> - </address> - </author> - - <date day="8" month="July" year="2016"/> - - <abstract> - <t>This memo describes the progress of the Open Platform for NFV (OPNFV) - project on virtual switch performance "VSWITCHPERF". This project - intends to build on the current and completed work of the Benchmarking - Methodology Working Group in IETF, by referencing existing literature. - The Benchmarking Methodology Working Group has traditionally conducted - laboratory characterization of dedicated physical implementations of - internetworking functions. Therefore, this memo begins to describe the - additional considerations when virtual switches are implemented in - general-purpose hardware. The expanded tests and benchmarks are also - influenced by the OPNFV mission to support virtualization of the "telco" - infrastructure.</t> - </abstract> - - <note title="Requirements Language"> - <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", - "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this - document are to be interpreted as described in <xref - target="RFC2119">RFC 2119</xref>.</t> - - <t/> - </note> - </front> - - <middle> - <section title="Introduction"> - <t>Benchmarking Methodology Working Group (BMWG) has traditionally - conducted laboratory characterization of dedicated physical - implementations of internetworking functions. The Black-box Benchmarks - of Throughput, Latency, Forwarding Rates and others have served our - industry for many years. Now, Network Function Virtualization (NFV) has - the goal to transform how internetwork functions are implemented, and - therefore has garnered much attention.</t> - - <t>This memo summarizes the progress of the Open Platform for NFV - (OPNFV) project on virtual switch performance characterization, - "VSWITCHPERF", through the Brahmaputra (second) release <xref - target="BrahRel"/>. This project intends to build on the current and - completed work of the Benchmarking Methodology Working Group in IETF, by - referencing existing literature. For example, currently the most often - referenced RFC is <xref target="RFC2544"/> (which depends on <xref - target="RFC1242"/>) and foundation of the benchmarking work in OPNFV is - common and strong.</t> - - <t>See - https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases - for more background, and the OPNFV website for general information: - https://www.opnfv.org/</t> - - <t>The authors note that OPNFV distinguishes itself from other open - source compute and networking projects through its emphasis on existing - "telco" services as opposed to cloud-computing. There are many ways in - which telco requirements have different emphasis on performance - dimensions when compared to cloud computing: support for and transfer of - isochronous media streams is one example.</t> - - <t>Note also that the move to NFV Infrastructure has resulted in many - new benchmarking initiatives across the industry. The authors are - currently doing their best to maintain alignment with many other - projects, and this Internet Draft is one part of the efforts. We - acknowledge the early work in <xref - target="I-D.huang-bmwg-virtual-network-performance"/>, and useful - discussion with the authors.</t> - </section> - - <section title="Scope"> - <t>The primary purpose and scope of the memo is to inform the industry - of work-in-progress that builds on the body of extensive BMWG literature - and experience, and describe the extensions needed for benchmarking - virtual switches. Inital feedback indicates that many of these - extensions may be applicable beyond the current scope (to hardware - switches in the NFV Infrastructure and to virtual routers, for example). - Additionally, this memo serves as a vehicle to include more detail and - commentary from BMWG and other Open Source communities, under BMWG's - chartered work to characterize the NFV Infrastructure (a virtual switch - is an important aspect of that infrastructure).</t> - </section> - - <section title="Benchmarking Considerations"> - <t>This section highlights some specific considerations (from <xref - target="I-D.ietf-bmwg-virtual-net"/>)related to Benchmarks for virtual - switches. The OPNFV project is sharing its present view on these areas, - as they develop their specifications in the Level Test Design (LTD) - document.</t> - - <section title="Comparison with Physical Network Functions"> - <t>To compare the performance of virtual designs and implementations - with their physical counterparts, identical benchmarks are needed. - BMWG has developed specifications for many network functions this memo - re-uses existing benchmarks through references, and expands them - during development of new methods. A key configuration aspect is the - number of parallel cores required to achieve comparable performance - with a given physical device, or whether some limit of scale was - reached before the cores could achieve the comparable level.</t> - - <t>It's unlikely that the virtual switch will be the only application - running on the SUT, so CPU utilization, Cache utilization, and Memory - footprint should also be recorded for the virtual implementations of - internetworking functions.</t> - </section> - - <section title="Continued Emphasis on Black-Box Benchmarks"> - <t>External observations remain essential as the basis for Benchmarks. - Internal observations with fixed specification and interpretation will - be provided in parallel to assist the development of operations - procedures when the technology is deployed.</t> - </section> - - <section title="New Configuration Parameters"> - <t>A key consideration when conducting any sort of benchmark is trying - to ensure the consistency and repeatability of test results. When - benchmarking the performance of a vSwitch there are many factors that - can affect the consistency of results, one key factor is matching the - various hardware and software details of the SUT. This section lists - some of the many new parameters which this project believes are - critical to report in order to achieve repeatability.</t> - - <t>Hardware details including:</t> - - <t><list style="symbols"> - <t>Platform details</t> - - <t>Processor details</t> - - <t>Memory information (type and size)</t> - - <t>Number of enabled cores</t> - - <t>Number of cores used for the test</t> - - <t>Number of physical NICs, as well as their details - (manufacturer, versions, type and the PCI slot they are plugged - into)</t> - - <t>NIC interrupt configuration</t> - - <t>BIOS version, release date and any configurations that were - modified</t> - - <t>CPU microcode level</t> - - <t>Memory DIMM configurations (quad rank performance may not be - the same as dual rank) in size, freq and slot locations</t> - - <t>PCI configuration parameters (payload size, early ack - option...)</t> - - <t>Power management at all levels (ACPI sleep states, processor - package, OS...)</t> - </list>Software details including:</t> - - <t><list style="symbols"> - <t>OS parameters and behavior (text vs graphical no one typing at - the console on one system)</t> - - <t>OS version (for host and VNF)</t> - - <t>Kernel version (for host and VNF)</t> - - <t>GRUB boot parameters (for host and VNF)</t> - - <t>Hypervisor details (Type and version)</t> - - <t>Selected vSwitch, version number or commit id used</t> - - <t>vSwitch launch command line if it has been parameterised</t> - - <t>Memory allocation to the vSwitch</t> - - <t>which NUMA node it is using, and how many memory channels</t> - - <t>DPDK or any other SW dependency version number or commit id - used</t> - - <t>Memory allocation to a VM - if it's from Hugpages/elsewhere</t> - - <t>VM storage type: snapshot/independent persistent/independent - non-persistent</t> - - <t>Number of VMs</t> - - <t>Number of Virtual NICs (vNICs), versions, type and driver</t> - - <t>Number of virtual CPUs and their core affinity on the host</t> - - <t>Number vNIC interrupt configuration</t> - - <t>Thread affinitization for the applications (including the - vSwitch itself) on the host</t> - - <t>Details of Resource isolation, such as CPUs designated for - Host/Kernel (isolcpu) and CPUs designated for specific processes - (taskset). - Test duration. - Number of flows.</t> - </list></t> - - <t>Test Traffic Information:<list style="symbols"> - <t>Traffic type - UDP, TCP, IMIX / Other</t> - - <t>Packet Sizes</t> - - <t>Deployment Scenario</t> - </list></t> - - <t/> - </section> - - <section title="Flow classification"> - <t>Virtual switches group packets into flows by processing and - matching particular packet or frame header information, or by matching - packets based on the input ports. Thus a flow can be thought of a - sequence of packets that have the same set of header field values - (5-tuple) or have arrived on the same port. Performance results can - vary based on the parameters the vSwitch uses to match for a flow. The - recommended flow classification parameters for any vSwitch performance - tests are: the input port, the source IP address, the destination IP - address and the Ethernet protocol type field. It is essential to - increase the flow timeout time on a vSwitch before conducting any - performance tests that do not measure the flow setup time. Normally - the first packet of a particular stream will install the flow in the - virtual switch which adds an additional latency, subsequent packets of - the same flow are not subject to this latency if the flow is already - installed on the vSwitch.</t> - </section> - - <section title="Benchmarks using Baselines with Resource Isolation"> - <t>This outline describes measurement of baseline with isolated - resources at a high level, which is the intended approach at this - time.</t> - - <t><list style="numbers"> - <t>Baselines: <list style="symbols"> - <t>Optional: Benchmark platform forwarding capability without - a vswitch or VNF for at least 72 hours (serves as a means of - platform validation and a means to obtain the base performance - for the platform in terms of its maximum forwarding rate and - latency). <figure> - <preamble>Benchmark platform forwarding - capability</preamble> - - <artwork align="right"><![CDATA[ __ - +--------------------------------------------------+ | - | +------------------------------------------+ | | - | | | | | - | | Simple Forwarding App | | Host - | | | | | - | +------------------------------------------+ | | - | | NIC | | | - +---+------------------------------------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - - <postamble/> - </figure></t> - - <t>Benchmark VNF forwarding capability with direct - connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72 - hours (serves as a means of VNF validation and a means to - obtain the base performance for the VNF in terms of its - maximum forwarding rate and latency). The metrics gathered - from this test will serve as a key comparison point for - vSwitch bypass technologies performance and vSwitch - performance. <figure align="right"> - <preamble>Benchmark VNF forwarding capability</preamble> - - <artwork><![CDATA[ __ - +--------------------------------------------------+ | - | +------------------------------------------+ | | - | | | | | - | | VNF | | | - | | | | | - | +------------------------------------------+ | | - | | Passthrough/SR-IOV | | Host - | +------------------------------------------+ | | - | | NIC | | | - +---+------------------------------------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - - <postamble/> - </figure></t> - - <t>Benchmarking with isolated resources alone, with other - resources (both HW&SW) disabled Example, vSw and VM are - SUT</t> - - <t>Benchmarking with isolated resources alone, leaving some - resources unused</t> - - <t>Benchmark with isolated resources and all resources - occupied</t> - </list></t> - - <t>Next Steps<list style="symbols"> - <t>Limited sharing</t> - - <t>Production scenarios</t> - - <t>Stressful scenarios</t> - </list></t> - </list></t> - </section> - </section> - - <section title="VSWITCHPERF Specification Summary"> - <t>The overall specification in preparation is referred to as a Level - Test Design (LTD) document, which will contain a suite of performance - tests. The base performance tests in the LTD are based on the - pre-existing specifications developed by BMWG to test the performance of - physical switches. These specifications include:</t> - - <t><list style="symbols"> - <t><xref target="RFC2544"/> Benchmarking Methodology for Network - Interconnect Devices</t> - - <t><xref target="RFC2889"/> Benchmarking Methodology for LAN - Switching</t> - - <t><xref target="RFC6201"/> Device Reset Characterization</t> - - <t><xref target="RFC5481"/> Packet Delay Variation Applicability - Statement</t> - </list></t> - - <t>Some of the above/newer RFCs are being applied in benchmarking for - the first time, and represent a development challenge for test equipment - developers. Fortunately, many members of the testing system community - have engaged on the VSPERF project, including an open source test - system.</t> - - <t>In addition to this, the LTD also re-uses the terminology defined - by:</t> - - <t><list style="symbols"> - <t><xref target="RFC2285"/> Benchmarking Terminology for LAN - Switching Devices</t> - - <t><xref target="RFC5481"/> Packet Delay Variation Applicability - Statement</t> - </list></t> - - <t/> - - <t>Specifications to be included in future updates of the LTD - include:<list style="symbols"> - <t><xref target="RFC3918"/> Methodology for IP Multicast - Benchmarking</t> - - <t><xref target="RFC4737"/> Packet Reordering Metrics</t> - </list></t> - - <t>As one might expect, the most fundamental internetworking - characteristics of Throughput and Latency remain important when the - switch is virtualized, and these benchmarks figure prominently in the - specification.</t> - - <t>When considering characteristics important to "telco" network - functions, we must begin to consider additional performance metrics. In - this case, the project specifications have referenced metrics from the - IETF IP Performance Metrics (IPPM) literature. This means that the <xref - target="RFC2544"/> test of Latency is replaced by measurement of a - metric derived from IPPM's <xref target="RFC2679"/>, where a set of - statistical summaries will be provided (mean, max, min, etc.). Further - metrics planned to be benchmarked include packet delay variation as - defined by <xref target="RFC5481"/> , reordering, burst behaviour, DUT - availability, DUT capacity and packet loss in long term testing at - Throughput level, where some low-level of background loss may be present - and characterized.</t> - - <t>Tests have been (or will be) designed to collect the metrics - below:</t> - - <t><list style="symbols"> - <t>Throughput Tests to measure the maximum forwarding rate (in - frames per second or fps) and bit rate (in Mbps) for a constant load - (as defined by <xref target="RFC1242"/>) without traffic loss.</t> - - <t>Packet and Frame Delay Distribution Tests to measure average, min - and max packet and frame delay for constant loads.</t> - - <t>Packet Delay Tests to understand latency distribution for - different packet sizes and over an extended test run to uncover - outliers.</t> - - <t>Scalability Tests to understand how the virtual switch performs - as the number of flows, active ports, complexity of the forwarding - logic’s configuration… it has to deal with - increases.</t> - - <t>Stream Performance Tests (TCP, UDP) to measure bulk data transfer - performance, i.e. how fast systems can send and receive data through - the switch.</t> - - <t>Control Path and Datapath Coupling Tests, to understand how - closely coupled the datapath and the control path are as well as the - effect of this coupling on the performance of the DUT (example: - delay of the initial packet of a flow).</t> - - <t>CPU and Memory Consumption Tests to understand the virtual - switch’s footprint on the system, usually conducted as - auxiliary measurements with benchmarks above. They include: CPU - utilization, Cache utilization and Memory footprint.</t> - - <t>The so-called "Soak" tests, where the selected test is conducted - over a long period of time (with an ideal duration of 24 hours, and - at least 6 hours). The purpose of soak tests is to capture transient - changes in performance which may occur due to infrequent processes - or the low probability coincidence of two or more processes. The - performance must be evaluated periodically during continuous - testing, and this results in use of <xref target="RFC2889"/> Frame - Rate metrics instead of <xref target="RFC2544"/> Throughput (which - requires stopping traffic to allow time for all traffic to exit - internal queues).</t> - </list></t> - - <t>Future/planned test specs include:<list style="symbols"> - <t>Request/Response Performance Tests (TCP, UDP) which measure the - transaction rate through the switch.</t> - - <t>Noisy Neighbour Tests, to understand the effects of resource - sharing on the performance of a virtual switch.</t> - - <t>Tests derived from examination of ETSI NFV Draft GS IFA003 - requirements <xref target="IFA003"/> on characterization of - acceleration technologies applied to vswitches.</t> - </list>The flexibility of deployment of a virtual switch within a - network means that the BMWG IETF existing literature needs to be used to - characterize the performance of a switch in various deployment - scenarios. The deployment scenarios under consideration include:</t> - - <t><figure> - <preamble>Physical port to virtual switch to physical - port</preamble> - - <artwork><![CDATA[ __ - +--------------------------------------------------+ | - | +--------------------+ | | - | | | | | - | | v | | Host - | +--------------+ +--------------+ | | - | | phy port | vSwitch | phy port | | | - +---+--------------+------------+--------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure></t> - - <t><figure> - <preamble>Physical port to virtual switch to VNF to virtual switch - to physical port</preamble> - - <artwork><![CDATA[ __ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | ^ : | | - | | | | | Guest - | : v | | - | +---------------+ +---------------+ | | - | | logical port 0| | logical port 1| | | - +---+---------------+-----------+---------------+---+ __| - ^ : - | | - : v __ - +---+---------------+----------+---------------+---+ | - | | logical port 0| | logical port 1| | | - | +---------------+ +---------------+ | | - | ^ : | | - | | | | | Host - | : v | | - | +--------------+ +--------------+ | | - | | phy port | vSwitch | phy port | | | - +---+--------------+------------+--------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>Physical port to virtual switch to VNF to virtual switch - to VNF to virtual switch to physical port</preamble> - - <artwork><![CDATA[ __ - +----------------------+ +----------------------+ | - | Guest 1 | | Guest 2 | | - | +---------------+ | | +---------------+ | | - | | Application | | | | Application | | | - | +---------------+ | | +---------------+ | | - | ^ | | | ^ | | | - | | v | | | v | | Guests - | +---------------+ | | +---------------+ | | - | | logical ports | | | | logical ports | | | - | | 0 1 | | | | 0 1 | | | - +---+---------------+--+ +---+---------------+--+__| - ^ : ^ : - | | | | - : v : v _ - +---+---------------+---------+---------------+--+ | - | | 0 1 | | 3 4 | | | - | | logical ports | | logical ports | | | - | +---------------+ +---------------+ | | - | ^ | ^ | | | Host - | | |-----------------| v | | - | +--------------+ +--------------+ | | - | | phy ports | vSwitch | phy ports | | | - +---+--------------+----------+--------------+---+_| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>Physical port to virtual switch to VNF</preamble> - - <artwork><![CDATA[ __ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | ^ | | - | | | | Guest - | : | | - | +---------------+ | | - | | logical port 0| | | - +---+---------------+-------------------------------+ __| - ^ - | - : __ - +---+---------------+------------------------------+ | - | | logical port 0| | | - | +---------------+ | | - | ^ | | - | | | | Host - | : | | - | +--------------+ | | - | | phy port | vSwitch | | - +---+--------------+------------ -------------- ---+ __| - ^ - | - : - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>VNF to virtual switch to physical port</preamble> - - <artwork><![CDATA[ __ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | : | | - | | | | Guest - | v | | - | +---------------+ | | - | | logical port | | | - +-------------------------------+---------------+---+ __| - : - | - v __ - +------------------------------+---------------+---+ | - | | logical port | | | - | +---------------+ | | - | : | | - | | | | Host - | v | | - | +--------------+ | | - | vSwitch | phy port | | | - +-------------------------------+--------------+---+ __| - : - | - v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>VNF to virtual switch to VNF</preamble> - - <artwork><![CDATA[ __ - +----------------------+ +----------------------+ | - | Guest 1 | | Guest 2 | | - | +---------------+ | | +---------------+ | | - | | Application | | | | Application | | | - | +---------------+ | | +---------------+ | | - | | | | ^ | | - | v | | | | | Guests - | +---------------+ | | +---------------+ | | - | | logical ports | | | | logical ports | | | - | | 0 | | | | 0 | | | - +---+---------------+--+ +---+---------------+--+__| - : ^ - | | - v : _ - +---+---------------+---------+---------------+--+ | - | | 1 | | 1 | | | - | | logical ports | | logical ports | | | - | +---------------+ +---------------+ | | - | | ^ | | Host - | L-----------------+ | | - | | | - | vSwitch | | - +------------------------------------------------+_|]]></artwork> - </figure></t> - - <t>A set of Deployment Scenario figures is available on the VSPERF Test - Methodology Wiki page <xref target="TestTopo"/>.</t> - </section> - - <section title="3x3 Matrix Coverage"> - <t>This section organizes the many existing test specifications into the - "3x3" matrix (introduced in <xref target="I-D.ietf-bmwg-virtual-net"/>). - Because the LTD specification ID names are quite long, this section is - organized into lists for each occupied cell of the matrix (not all are - occupied, also the matrix has grown to 3x4 to accommodate scale metrics - when displaying the coverage of many metrics/benchmarks). The current - version of the LTD specification is available <xref target="LTD"/>.</t> - - <t>The tests listed below assess the activation of paths in the data - plane, rather than the control plane.</t> - - <t>A complete list of tests with short summaries is available on the - VSPERF "LTD Test Spec Overview" Wiki page <xref target="LTDoverV"/>.</t> - - <section title="Speed of Activation"> - <t><list style="symbols"> - <t>Activation.RFC2889.AddressLearningRate</t> - - <t>PacketLatency.InitialPacketProcessingLatency</t> - </list></t> - </section> - - <section title="Accuracy of Activation section"> - <t><list style="symbols"> - <t>CPDP.Coupling.Flow.Addition</t> - </list></t> - </section> - - <section title="Reliability of Activation"> - <t><list style="symbols"> - <t>Throughput.RFC2544.SystemRecoveryTime</t> - - <t>Throughput.RFC2544.ResetTime</t> - </list></t> - </section> - - <section title="Scale of Activation"> - <t><list style="symbols"> - <t>Activation.RFC2889.AddressCachingCapacity</t> - </list></t> - </section> - - <section title="Speed of Operation"> - <t><list style="symbols"> - <t>Throughput.RFC2544.PacketLossRate</t> - - <t>CPU.RFC2544.0PacketLoss</t> - - <t>Throughput.RFC2544.PacketLossRateFrameModification</t> - - <t>Throughput.RFC2544.BackToBackFrames</t> - - <t>Throughput.RFC2889.MaxForwardingRate</t> - - <t>Throughput.RFC2889.ForwardPressure</t> - - <t>Throughput.RFC2889.BroadcastFrameForwarding</t> - </list></t> - </section> - - <section title="Accuracy of Operation"> - <t><list style="symbols"> - <t>Throughput.RFC2889.ErrorFramesFiltering</t> - - <t>Throughput.RFC2544.Profile</t> - </list></t> - </section> - - <section title="Reliability of Operation"> - <t><list style="symbols"> - <t>Throughput.RFC2889.Soak</t> - - <t>Throughput.RFC2889.SoakFrameModification</t> - - <t>PacketDelayVariation.RFC3393.Soak</t> - </list></t> - </section> - - <section title="Scalability of Operation"> - <t><list style="symbols"> - <t>Scalability.RFC2544.0PacketLoss</t> - - <t>MemoryBandwidth.RFC2544.0PacketLoss.Scalability</t> - </list></t> - </section> - - <section title="Summary"> - <t><figure> - <artwork><![CDATA[|------------------------------------------------------------------------| -| | | | | | -| | SPEED | ACCURACY | RELIABILITY | SCALE | -| | | | | | -|------------------------------------------------------------------------| -| | | | | | -| Activation | X | X | X | X | -| | | | | | -|------------------------------------------------------------------------| -| | | | | | -| Operation | X | X | X | X | -| | | | | | -|------------------------------------------------------------------------| -| | | | | | -| De-activation | | | | | -| | | | | | -|------------------------------------------------------------------------|]]></artwork> - </figure></t> - </section> - </section> - - <section title="Security Considerations"> - <t>Benchmarking activities as described in this memo are limited to - technology characterization of a Device Under Test/System Under Test - (DUT/SUT) using controlled stimuli in a laboratory environment, with - dedicated address space and the constraints specified in the sections - above.</t> - - <t>The benchmarking network topology will be an independent test setup - and MUST NOT be connected to devices that may forward the test traffic - into a production network, or misroute traffic to the test management - network.</t> - - <t>Further, benchmarking is performed on a "black-box" basis, relying - solely on measurements observable external to the DUT/SUT.</t> - - <t>Special capabilities SHOULD NOT exist in the DUT/SUT specifically for - benchmarking purposes. Any implications for network security arising - from the DUT/SUT SHOULD be identical in the lab and in production - networks.</t> - </section> - - <section anchor="IANA" title="IANA Considerations"> - <t>No IANA Action is requested at this time.</t> - </section> - - <section title="Acknowledgements"> - <t>The authors appreciate and acknowledge comments from Scott Bradner, - Marius Georgescu, Ramki Krishnan, Doug Montgomery, Martin Klozik, - Christian Trautman, and others for their reviews.</t> - </section> - </middle> - - <back> - <references title="Normative References"> - <?rfc ?> - - <?rfc include="reference.RFC.2119"?> - - <?rfc ?> - - <?rfc include="reference.RFC.2330"?> - - <?rfc include='reference.RFC.2544'?> - - <?rfc include="reference.RFC.2679"?> - - <?rfc include='reference.RFC.2680'?> - - <?rfc include='reference.RFC.3393'?> - - <?rfc include='reference.RFC.3432'?> - - <?rfc include='reference.RFC.2681'?> - - <?rfc include='reference.RFC.5905'?> - - <?rfc include='reference.RFC.4689'?> - - <?rfc include='reference.RFC.4737'?> - - <?rfc include='reference.RFC.5357'?> - - <?rfc include='reference.RFC.2889'?> - - <?rfc include='reference.RFC.3918'?> - - <?rfc include='reference.RFC.6201'?> - - <?rfc include='reference.RFC.2285'?> - - <reference anchor="NFV.PER001"> - <front> - <title>Network Function Virtualization: Performance and Portability - Best Practices</title> - - <author fullname="ETSI NFV" initials="" surname=""> - <organization/> - </author> - - <date month="June" year="2014"/> - </front> - - <seriesInfo name="Group Specification" - value="ETSI GS NFV-PER 001 V1.1.1 (2014-06)"/> - - <format type="PDF"/> - </reference> - </references> - - <references title="Informative References"> - <?rfc include='reference.RFC.1242'?> - - <?rfc include='reference.RFC.5481'?> - - <?rfc include='reference.RFC.6049'?> - - <?rfc include='reference.RFC.6248'?> - - <?rfc include='reference.RFC.6390'?> - - <?rfc include='reference.I-D.ietf-bmwg-virtual-net'?> - - <?rfc include='reference.I-D.huang-bmwg-virtual-network-performance'?> - - <reference anchor="TestTopo"> - <front> - <title>Test Topologies - https://wiki.opnfv.org/vsperf/test_methodology</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="LTDoverV"> - <front> - <title>LTD Test Spec Overview - https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="LTD"> - <front> - <title>LTD Test Specification - http://artifacts.opnfv.org/vswitchperf/docs/requirements/index.html</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="BrahRel"> - <front> - <title>Brahmaputra, Second OPNFV Release - https://www.opnfv.org/brahmaputra</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="IFA003"> - <front> - <title>https://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA003_Acceleration_-_vSwitch_Spec/</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - </references> - </back> -</rfc> diff --git a/docs/testing/developer/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-01.xml b/docs/testing/developer/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-01.xml deleted file mode 100644 index c8a3d99b..00000000 --- a/docs/testing/developer/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-01.xml +++ /dev/null @@ -1,1027 +0,0 @@ -<?xml version="1.0" encoding="US-ASCII"?> -<!DOCTYPE rfc SYSTEM "rfc2629.dtd"> -<?rfc toc="yes"?> -<?rfc tocompact="yes"?> -<?rfc tocdepth="3"?> -<?rfc tocindent="yes"?> -<?rfc symrefs="yes"?> -<?rfc sortrefs="yes"?> -<?rfc comments="yes"?> -<?rfc inline="yes"?> -<?rfc compact="yes"?> -<?rfc subcompact="no"?> -<rfc category="info" docName="draft-ietf-bmwg-vswitch-opnfv-01" - ipr="trust200902"> - <front> - <title abbrev="Benchmarking vSwitches">Benchmarking Virtual Switches in - OPNFV</title> - - <author fullname="Maryam Tahhan" initials="M." surname="Tahhan"> - <organization>Intel</organization> - - <address> - <postal> - <street/> - - <city/> - - <region/> - - <code/> - - <country/> - </postal> - - <phone/> - - <facsimile/> - - <email>maryam.tahhan@intel.com</email> - - <uri/> - </address> - </author> - - <author fullname="Billy O'Mahony" initials="B." surname="O'Mahony"> - <organization>Intel</organization> - - <address> - <postal> - <street/> - - <city/> - - <region/> - - <code/> - - <country/> - </postal> - - <phone/> - - <facsimile/> - - <email>billy.o.mahony@intel.com</email> - - <uri/> - </address> - </author> - - <author fullname="Al Morton" initials="A." surname="Morton"> - <organization>AT&T Labs</organization> - - <address> - <postal> - <street>200 Laurel Avenue South</street> - - <city>Middletown,</city> - - <region>NJ</region> - - <code>07748</code> - - <country>USA</country> - </postal> - - <phone>+1 732 420 1571</phone> - - <facsimile>+1 732 368 1192</facsimile> - - <email>acmorton@att.com</email> - - <uri>http://home.comcast.net/~acmacm/</uri> - </address> - </author> - - <date day="10" month="October" year="2016"/> - - <abstract> - <t>This memo describes the progress of the Open Platform for NFV (OPNFV) - project on virtual switch performance "VSWITCHPERF". This project - intends to build on the current and completed work of the Benchmarking - Methodology Working Group in IETF, by referencing existing literature. - The Benchmarking Methodology Working Group has traditionally conducted - laboratory characterization of dedicated physical implementations of - internetworking functions. Therefore, this memo begins to describe the - additional considerations when virtual switches are implemented in - general-purpose hardware. The expanded tests and benchmarks are also - influenced by the OPNFV mission to support virtualization of the "telco" - infrastructure.</t> - </abstract> - - <note title="Requirements Language"> - <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", - "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this - document are to be interpreted as described in <xref - target="RFC2119">RFC 2119</xref>.</t> - - <t/> - </note> - </front> - - <middle> - <section title="Introduction"> - <t>Benchmarking Methodology Working Group (BMWG) has traditionally - conducted laboratory characterization of dedicated physical - implementations of internetworking functions. The Black-box Benchmarks - of Throughput, Latency, Forwarding Rates and others have served our - industry for many years. Now, Network Function Virtualization (NFV) has - the goal to transform how internetwork functions are implemented, and - therefore has garnered much attention.</t> - - <t>This memo summarizes the progress of the Open Platform for NFV - (OPNFV) project on virtual switch performance characterization, - "VSWITCHPERF", through the Brahmaputra (second) release <xref - target="BrahRel"/>. This project intends to build on the current and - completed work of the Benchmarking Methodology Working Group in IETF, by - referencing existing literature. For example, currently the most often - referenced RFC is <xref target="RFC2544"/> (which depends on <xref - target="RFC1242"/>) and foundation of the benchmarking work in OPNFV is - common and strong.</t> - - <t>See - https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases - for more background, and the OPNFV website for general information: - https://www.opnfv.org/</t> - - <t>The authors note that OPNFV distinguishes itself from other open - source compute and networking projects through its emphasis on existing - "telco" services as opposed to cloud-computing. There are many ways in - which telco requirements have different emphasis on performance - dimensions when compared to cloud computing: support for and transfer of - isochronous media streams is one example.</t> - - <t>Note also that the move to NFV Infrastructure has resulted in many - new benchmarking initiatives across the industry. The authors are - currently doing their best to maintain alignment with many other - projects, and this Internet Draft is one part of the efforts. We - acknowledge the early work in <xref - target="I-D.huang-bmwg-virtual-network-performance"/>, and useful - discussion with the authors.</t> - </section> - - <section title="Scope"> - <t>The primary purpose and scope of the memo is to inform the industry - of work-in-progress that builds on the body of extensive BMWG literature - and experience, and describe the extensions needed for benchmarking - virtual switches. Inital feedback indicates that many of these - extensions may be applicable beyond the current scope (to hardware - switches in the NFV Infrastructure and to virtual routers, for example). - Additionally, this memo serves as a vehicle to include more detail and - commentary from BMWG and other Open Source communities, under BMWG's - chartered work to characterize the NFV Infrastructure (a virtual switch - is an important aspect of that infrastructure).</t> - - <t>The benchmarking covered in this memo should be applicable to many - types of vswitches, and remain vswitch-agnostic to great degree. There - has been no attempt to track and test all features of any specific - vswitch implementation.</t> - </section> - - <section title="Benchmarking Considerations"> - <t>This section highlights some specific considerations (from <xref - target="I-D.ietf-bmwg-virtual-net"/>)related to Benchmarks for virtual - switches. The OPNFV project is sharing its present view on these areas, - as they develop their specifications in the Level Test Design (LTD) - document.</t> - - <section title="Comparison with Physical Network Functions"> - <t>To compare the performance of virtual designs and implementations - with their physical counterparts, identical benchmarks are needed. - BMWG has developed specifications for many network functions this memo - re-uses existing benchmarks through references, and expands them - during development of new methods. A key configuration aspect is the - number of parallel cores required to achieve comparable performance - with a given physical device, or whether some limit of scale was - reached before the cores could achieve the comparable level.</t> - - <t>It's unlikely that the virtual switch will be the only application - running on the SUT, so CPU utilization, Cache utilization, and Memory - footprint should also be recorded for the virtual implementations of - internetworking functions.</t> - </section> - - <section title="Continued Emphasis on Black-Box Benchmarks"> - <t>External observations remain essential as the basis for Benchmarks. - Internal observations with fixed specification and interpretation will - be provided in parallel to assist the development of operations - procedures when the technology is deployed.</t> - </section> - - <section title="New Configuration Parameters"> - <t>A key consideration when conducting any sort of benchmark is trying - to ensure the consistency and repeatability of test results. When - benchmarking the performance of a vSwitch there are many factors that - can affect the consistency of results, one key factor is matching the - various hardware and software details of the SUT. This section lists - some of the many new parameters which this project believes are - critical to report in order to achieve repeatability.</t> - - <t>Hardware details including:</t> - - <t><list style="symbols"> - <t>Platform details</t> - - <t>Processor details</t> - - <t>Memory information (type and size)</t> - - <t>Number of enabled cores</t> - - <t>Number of cores used for the test</t> - - <t>Number of physical NICs, as well as their details - (manufacturer, versions, type and the PCI slot they are plugged - into)</t> - - <t>NIC interrupt configuration</t> - - <t>BIOS version, release date and any configurations that were - modified</t> - - <t>CPU microcode level</t> - - <t>Memory DIMM configurations (quad rank performance may not be - the same as dual rank) in size, freq and slot locations</t> - - <t>PCI configuration parameters (payload size, early ack - option...)</t> - - <t>Power management at all levels (ACPI sleep states, processor - package, OS...)</t> - </list>Software details including:</t> - - <t><list style="symbols"> - <t>OS parameters and behavior (text vs graphical no one typing at - the console on one system)</t> - - <t>OS version (for host and VNF)</t> - - <t>Kernel version (for host and VNF)</t> - - <t>GRUB boot parameters (for host and VNF)</t> - - <t>Hypervisor details (Type and version)</t> - - <t>Selected vSwitch, version number or commit id used</t> - - <t>vSwitch launch command line if it has been parameterised</t> - - <t>Memory allocation to the vSwitch</t> - - <t>which NUMA node it is using, and how many memory channels</t> - - <t>DPDK or any other SW dependency version number or commit id - used</t> - - <t>Memory allocation to a VM - if it's from Hugpages/elsewhere</t> - - <t>VM storage type: snapshot/independent persistent/independent - non-persistent</t> - - <t>Number of VMs</t> - - <t>Number of Virtual NICs (vNICs), versions, type and driver</t> - - <t>Number of virtual CPUs and their core affinity on the host</t> - - <t>Number vNIC interrupt configuration</t> - - <t>Thread affinitization for the applications (including the - vSwitch itself) on the host</t> - - <t>Details of Resource isolation, such as CPUs designated for - Host/Kernel (isolcpu) and CPUs designated for specific processes - (taskset). - Test duration. - Number of flows.</t> - </list></t> - - <t>Test Traffic Information:<list style="symbols"> - <t>Traffic type - UDP, TCP, IMIX / Other</t> - - <t>Packet Sizes</t> - - <t>Deployment Scenario</t> - </list></t> - - <t/> - </section> - - <section title="Flow classification"> - <t>Virtual switches group packets into flows by processing and - matching particular packet or frame header information, or by matching - packets based on the input ports. Thus a flow can be thought of a - sequence of packets that have the same set of header field values - (5-tuple) or have arrived on the same port. Performance results can - vary based on the parameters the vSwitch uses to match for a flow. The - recommended flow classification parameters for any vSwitch performance - tests are: the input port, the source IP address, the destination IP - address and the Ethernet protocol type field. It is essential to - increase the flow timeout time on a vSwitch before conducting any - performance tests that do not measure the flow setup time. Normally - the first packet of a particular stream will install the flow in the - virtual switch which adds an additional latency, subsequent packets of - the same flow are not subject to this latency if the flow is already - installed on the vSwitch.</t> - </section> - - <section title="Benchmarks using Baselines with Resource Isolation"> - <t>This outline describes measurement of baseline with isolated - resources at a high level, which is the intended approach at this - time.</t> - - <t><list style="numbers"> - <t>Baselines: <list style="symbols"> - <t>Optional: Benchmark platform forwarding capability without - a vswitch or VNF for at least 72 hours (serves as a means of - platform validation and a means to obtain the base performance - for the platform in terms of its maximum forwarding rate and - latency). <figure> - <preamble>Benchmark platform forwarding - capability</preamble> - - <artwork align="right"><![CDATA[ __ - +--------------------------------------------------+ | - | +------------------------------------------+ | | - | | | | | - | | Simple Forwarding App | | Host - | | | | | - | +------------------------------------------+ | | - | | NIC | | | - +---+------------------------------------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - - <postamble/> - </figure></t> - - <t>Benchmark VNF forwarding capability with direct - connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72 - hours (serves as a means of VNF validation and a means to - obtain the base performance for the VNF in terms of its - maximum forwarding rate and latency). The metrics gathered - from this test will serve as a key comparison point for - vSwitch bypass technologies performance and vSwitch - performance. <figure align="right"> - <preamble>Benchmark VNF forwarding capability</preamble> - - <artwork><![CDATA[ __ - +--------------------------------------------------+ | - | +------------------------------------------+ | | - | | | | | - | | VNF | | | - | | | | | - | +------------------------------------------+ | | - | | Passthrough/SR-IOV | | Host - | +------------------------------------------+ | | - | | NIC | | | - +---+------------------------------------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - - <postamble/> - </figure></t> - - <t>Benchmarking with isolated resources alone, with other - resources (both HW&SW) disabled Example, vSw and VM are - SUT</t> - - <t>Benchmarking with isolated resources alone, leaving some - resources unused</t> - - <t>Benchmark with isolated resources and all resources - occupied</t> - </list></t> - - <t>Next Steps<list style="symbols"> - <t>Limited sharing</t> - - <t>Production scenarios</t> - - <t>Stressful scenarios</t> - </list></t> - </list></t> - </section> - </section> - - <section title="VSWITCHPERF Specification Summary"> - <t>The overall specification in preparation is referred to as a Level - Test Design (LTD) document, which will contain a suite of performance - tests. The base performance tests in the LTD are based on the - pre-existing specifications developed by BMWG to test the performance of - physical switches. These specifications include:</t> - - <t><list style="symbols"> - <t><xref target="RFC2544"/> Benchmarking Methodology for Network - Interconnect Devices</t> - - <t><xref target="RFC2889"/> Benchmarking Methodology for LAN - Switching</t> - - <t><xref target="RFC6201"/> Device Reset Characterization</t> - - <t><xref target="RFC5481"/> Packet Delay Variation Applicability - Statement</t> - </list></t> - - <t>Some of the above/newer RFCs are being applied in benchmarking for - the first time, and represent a development challenge for test equipment - developers. Fortunately, many members of the testing system community - have engaged on the VSPERF project, including an open source test - system.</t> - - <t>In addition to this, the LTD also re-uses the terminology defined - by:</t> - - <t><list style="symbols"> - <t><xref target="RFC2285"/> Benchmarking Terminology for LAN - Switching Devices</t> - - <t><xref target="RFC5481"/> Packet Delay Variation Applicability - Statement</t> - </list></t> - - <t/> - - <t>Specifications to be included in future updates of the LTD - include:<list style="symbols"> - <t><xref target="RFC3918"/> Methodology for IP Multicast - Benchmarking</t> - - <t><xref target="RFC4737"/> Packet Reordering Metrics</t> - </list></t> - - <t>As one might expect, the most fundamental internetworking - characteristics of Throughput and Latency remain important when the - switch is virtualized, and these benchmarks figure prominently in the - specification.</t> - - <t>When considering characteristics important to "telco" network - functions, we must begin to consider additional performance metrics. In - this case, the project specifications have referenced metrics from the - IETF IP Performance Metrics (IPPM) literature. This means that the <xref - target="RFC2544"/> test of Latency is replaced by measurement of a - metric derived from IPPM's <xref target="RFC2679"/>, where a set of - statistical summaries will be provided (mean, max, min, etc.). Further - metrics planned to be benchmarked include packet delay variation as - defined by <xref target="RFC5481"/> , reordering, burst behaviour, DUT - availability, DUT capacity and packet loss in long term testing at - Throughput level, where some low-level of background loss may be present - and characterized.</t> - - <t>Tests have been (or will be) designed to collect the metrics - below:</t> - - <t><list style="symbols"> - <t>Throughput Tests to measure the maximum forwarding rate (in - frames per second or fps) and bit rate (in Mbps) for a constant load - (as defined by <xref target="RFC1242"/>) without traffic loss.</t> - - <t>Packet and Frame Delay Distribution Tests to measure average, min - and max packet and frame delay for constant loads.</t> - - <t>Packet Delay Tests to understand latency distribution for - different packet sizes and over an extended test run to uncover - outliers.</t> - - <t>Scalability Tests to understand how the virtual switch performs - as the number of flows, active ports, complexity of the forwarding - logic’s configuration… it has to deal with - increases.</t> - - <t>Stream Performance Tests (TCP, UDP) to measure bulk data transfer - performance, i.e. how fast systems can send and receive data through - the switch.</t> - - <t>Control Path and Datapath Coupling Tests, to understand how - closely coupled the datapath and the control path are as well as the - effect of this coupling on the performance of the DUT (example: - delay of the initial packet of a flow).</t> - - <t>CPU and Memory Consumption Tests to understand the virtual - switch’s footprint on the system, usually conducted as - auxiliary measurements with benchmarks above. They include: CPU - utilization, Cache utilization and Memory footprint.</t> - - <t>The so-called "Soak" tests, where the selected test is conducted - over a long period of time (with an ideal duration of 24 hours, but - only long enough to determine that stability issues exist when - found; there is no requirement to continue a test when a DUT - exhibits instability over time). The key performance characteristics - and benchmarks for a DUT are determined (using short duration tests) - prior to conducting soak tests. The purpose of soak tests is to - capture transient changes in performance which may occur due to - infrequent processes, memory leaks, or the low probability - coincidence of two or more processes. The stability of the DUT is - the paramount consideration, so performance must be evaluated - periodically during continuous testing, and this results in use of - <xref target="RFC2889"/> Frame Rate metrics instead of <xref - target="RFC2544"/> Throughput (which requires stopping traffic to - allow time for all traffic to exit internal queues), for - example.</t> - </list></t> - - <t>Future/planned test specs include:<list style="symbols"> - <t>Request/Response Performance Tests (TCP, UDP) which measure the - transaction rate through the switch.</t> - - <t>Noisy Neighbour Tests, to understand the effects of resource - sharing on the performance of a virtual switch.</t> - - <t>Tests derived from examination of ETSI NFV Draft GS IFA003 - requirements <xref target="IFA003"/> on characterization of - acceleration technologies applied to vswitches.</t> - </list>The flexibility of deployment of a virtual switch within a - network means that the BMWG IETF existing literature needs to be used to - characterize the performance of a switch in various deployment - scenarios. The deployment scenarios under consideration include:</t> - - <t><figure> - <preamble>Physical port to virtual switch to physical - port</preamble> - - <artwork><![CDATA[ __ - +--------------------------------------------------+ | - | +--------------------+ | | - | | | | | - | | v | | Host - | +--------------+ +--------------+ | | - | | phy port | vSwitch | phy port | | | - +---+--------------+------------+--------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure></t> - - <t><figure> - <preamble>Physical port to virtual switch to VNF to virtual switch - to physical port</preamble> - - <artwork><![CDATA[ __ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | ^ : | | - | | | | | Guest - | : v | | - | +---------------+ +---------------+ | | - | | logical port 0| | logical port 1| | | - +---+---------------+-----------+---------------+---+ __| - ^ : - | | - : v __ - +---+---------------+----------+---------------+---+ | - | | logical port 0| | logical port 1| | | - | +---------------+ +---------------+ | | - | ^ : | | - | | | | | Host - | : v | | - | +--------------+ +--------------+ | | - | | phy port | vSwitch | phy port | | | - +---+--------------+------------+--------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>Physical port to virtual switch to VNF to virtual switch - to VNF to virtual switch to physical port</preamble> - - <artwork><![CDATA[ __ - +----------------------+ +----------------------+ | - | Guest 1 | | Guest 2 | | - | +---------------+ | | +---------------+ | | - | | Application | | | | Application | | | - | +---------------+ | | +---------------+ | | - | ^ | | | ^ | | | - | | v | | | v | | Guests - | +---------------+ | | +---------------+ | | - | | logical ports | | | | logical ports | | | - | | 0 1 | | | | 0 1 | | | - +---+---------------+--+ +---+---------------+--+__| - ^ : ^ : - | | | | - : v : v _ - +---+---------------+---------+---------------+--+ | - | | 0 1 | | 3 4 | | | - | | logical ports | | logical ports | | | - | +---------------+ +---------------+ | | - | ^ | ^ | | | Host - | | |-----------------| v | | - | +--------------+ +--------------+ | | - | | phy ports | vSwitch | phy ports | | | - +---+--------------+----------+--------------+---+_| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>Physical port to virtual switch to VNF</preamble> - - <artwork><![CDATA[ __ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | ^ | | - | | | | Guest - | : | | - | +---------------+ | | - | | logical port 0| | | - +---+---------------+-------------------------------+ __| - ^ - | - : __ - +---+---------------+------------------------------+ | - | | logical port 0| | | - | +---------------+ | | - | ^ | | - | | | | Host - | : | | - | +--------------+ | | - | | phy port | vSwitch | | - +---+--------------+------------ -------------- ---+ __| - ^ - | - : - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>VNF to virtual switch to physical port</preamble> - - <artwork><![CDATA[ __ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | : | | - | | | | Guest - | v | | - | +---------------+ | | - | | logical port | | | - +-------------------------------+---------------+---+ __| - : - | - v __ - +------------------------------+---------------+---+ | - | | logical port | | | - | +---------------+ | | - | : | | - | | | | Host - | v | | - | +--------------+ | | - | vSwitch | phy port | | | - +-------------------------------+--------------+---+ __| - : - | - v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>VNF to virtual switch to VNF</preamble> - - <artwork><![CDATA[ __ - +----------------------+ +----------------------+ | - | Guest 1 | | Guest 2 | | - | +---------------+ | | +---------------+ | | - | | Application | | | | Application | | | - | +---------------+ | | +---------------+ | | - | | | | ^ | | - | v | | | | | Guests - | +---------------+ | | +---------------+ | | - | | logical ports | | | | logical ports | | | - | | 0 | | | | 0 | | | - +---+---------------+--+ +---+---------------+--+__| - : ^ - | | - v : _ - +---+---------------+---------+---------------+--+ | - | | 1 | | 1 | | | - | | logical ports | | logical ports | | | - | +---------------+ +---------------+ | | - | | ^ | | Host - | L-----------------+ | | - | | | - | vSwitch | | - +------------------------------------------------+_|]]></artwork> - </figure></t> - - <t>A set of Deployment Scenario figures is available on the VSPERF Test - Methodology Wiki page <xref target="TestTopo"/>.</t> - </section> - - <section title="3x3 Matrix Coverage"> - <t>This section organizes the many existing test specifications into the - "3x3" matrix (introduced in <xref target="I-D.ietf-bmwg-virtual-net"/>). - Because the LTD specification ID names are quite long, this section is - organized into lists for each occupied cell of the matrix (not all are - occupied, also the matrix has grown to 3x4 to accommodate scale metrics - when displaying the coverage of many metrics/benchmarks). The current - version of the LTD specification is available <xref target="LTD"/>.</t> - - <t>The tests listed below assess the activation of paths in the data - plane, rather than the control plane.</t> - - <t>A complete list of tests with short summaries is available on the - VSPERF "LTD Test Spec Overview" Wiki page <xref target="LTDoverV"/>.</t> - - <section title="Speed of Activation"> - <t><list style="symbols"> - <t>Activation.RFC2889.AddressLearningRate</t> - - <t>PacketLatency.InitialPacketProcessingLatency</t> - </list></t> - </section> - - <section title="Accuracy of Activation section"> - <t><list style="symbols"> - <t>CPDP.Coupling.Flow.Addition</t> - </list></t> - </section> - - <section title="Reliability of Activation"> - <t><list style="symbols"> - <t>Throughput.RFC2544.SystemRecoveryTime</t> - - <t>Throughput.RFC2544.ResetTime</t> - </list></t> - </section> - - <section title="Scale of Activation"> - <t><list style="symbols"> - <t>Activation.RFC2889.AddressCachingCapacity</t> - </list></t> - </section> - - <section title="Speed of Operation"> - <t><list style="symbols"> - <t>Throughput.RFC2544.PacketLossRate</t> - - <t>CPU.RFC2544.0PacketLoss</t> - - <t>Throughput.RFC2544.PacketLossRateFrameModification</t> - - <t>Throughput.RFC2544.BackToBackFrames</t> - - <t>Throughput.RFC2889.MaxForwardingRate</t> - - <t>Throughput.RFC2889.ForwardPressure</t> - - <t>Throughput.RFC2889.BroadcastFrameForwarding</t> - </list></t> - </section> - - <section title="Accuracy of Operation"> - <t><list style="symbols"> - <t>Throughput.RFC2889.ErrorFramesFiltering</t> - - <t>Throughput.RFC2544.Profile</t> - </list></t> - </section> - - <section title="Reliability of Operation"> - <t><list style="symbols"> - <t>Throughput.RFC2889.Soak</t> - - <t>Throughput.RFC2889.SoakFrameModification</t> - - <t>PacketDelayVariation.RFC3393.Soak</t> - </list></t> - </section> - - <section title="Scalability of Operation"> - <t><list style="symbols"> - <t>Scalability.RFC2544.0PacketLoss</t> - - <t>MemoryBandwidth.RFC2544.0PacketLoss.Scalability</t> - </list></t> - </section> - - <section title="Summary"> - <t><figure> - <artwork><![CDATA[|------------------------------------------------------------------------| -| | | | | | -| | SPEED | ACCURACY | RELIABILITY | SCALE | -| | | | | | -|------------------------------------------------------------------------| -| | | | | | -| Activation | X | X | X | X | -| | | | | | -|------------------------------------------------------------------------| -| | | | | | -| Operation | X | X | X | X | -| | | | | | -|------------------------------------------------------------------------| -| | | | | | -| De-activation | | | | | -| | | | | | -|------------------------------------------------------------------------|]]></artwork> - </figure></t> - </section> - </section> - - <section title="Security Considerations"> - <t>Benchmarking activities as described in this memo are limited to - technology characterization of a Device Under Test/System Under Test - (DUT/SUT) using controlled stimuli in a laboratory environment, with - dedicated address space and the constraints specified in the sections - above.</t> - - <t>The benchmarking network topology will be an independent test setup - and MUST NOT be connected to devices that may forward the test traffic - into a production network, or misroute traffic to the test management - network.</t> - - <t>Further, benchmarking is performed on a "black-box" basis, relying - solely on measurements observable external to the DUT/SUT.</t> - - <t>Special capabilities SHOULD NOT exist in the DUT/SUT specifically for - benchmarking purposes. Any implications for network security arising - from the DUT/SUT SHOULD be identical in the lab and in production - networks.</t> - </section> - - <section anchor="IANA" title="IANA Considerations"> - <t>No IANA Action is requested at this time.</t> - </section> - - <section title="Acknowledgements"> - <t>The authors appreciate and acknowledge comments from Scott Bradner, - Marius Georgescu, Ramki Krishnan, Doug Montgomery, Martin Klozik, - Christian Trautman, and others for their reviews.</t> - </section> - </middle> - - <back> - <references title="Normative References"> - <?rfc ?> - - <?rfc include="reference.RFC.2119"?> - - <?rfc ?> - - <?rfc include="reference.RFC.2330"?> - - <?rfc include='reference.RFC.2544'?> - - <?rfc include="reference.RFC.2679"?> - - <?rfc include='reference.RFC.2680'?> - - <?rfc include='reference.RFC.3393'?> - - <?rfc include='reference.RFC.3432'?> - - <?rfc include='reference.RFC.2681'?> - - <?rfc include='reference.RFC.5905'?> - - <?rfc include='reference.RFC.4689'?> - - <?rfc include='reference.RFC.4737'?> - - <?rfc include='reference.RFC.5357'?> - - <?rfc include='reference.RFC.2889'?> - - <?rfc include='reference.RFC.3918'?> - - <?rfc include='reference.RFC.6201'?> - - <?rfc include='reference.RFC.2285'?> - - <reference anchor="NFV.PER001"> - <front> - <title>Network Function Virtualization: Performance and Portability - Best Practices</title> - - <author fullname="ETSI NFV" initials="" surname=""> - <organization/> - </author> - - <date month="June" year="2014"/> - </front> - - <seriesInfo name="Group Specification" - value="ETSI GS NFV-PER 001 V1.1.1 (2014-06)"/> - - <format type="PDF"/> - </reference> - </references> - - <references title="Informative References"> - <?rfc include='reference.RFC.1242'?> - - <?rfc include='reference.RFC.5481'?> - - <?rfc include='reference.RFC.6049'?> - - <?rfc include='reference.RFC.6248'?> - - <?rfc include='reference.RFC.6390'?> - - <?rfc include='reference.I-D.ietf-bmwg-virtual-net'?> - - <?rfc include='reference.I-D.huang-bmwg-virtual-network-performance'?> - - <reference anchor="TestTopo"> - <front> - <title>Test Topologies - https://wiki.opnfv.org/vsperf/test_methodology</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="LTDoverV"> - <front> - <title>LTD Test Spec Overview - https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="LTD"> - <front> - <title>LTD Test Specification - http://artifacts.opnfv.org/vswitchperf/brahmaputra/docs/requirements/index.html</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="BrahRel"> - <front> - <title>Brahmaputra, Second OPNFV Release - https://www.opnfv.org/brahmaputra</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="IFA003"> - <front> - <title>https://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA003_Acceleration_-_vSwitch_Spec/</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - </references> - </back> -</rfc> diff --git a/docs/testing/developer/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml b/docs/testing/developer/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml deleted file mode 100644 index b5f7f833..00000000 --- a/docs/testing/developer/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml +++ /dev/null @@ -1,964 +0,0 @@ -<?xml version="1.0" encoding="US-ASCII"?> -<!DOCTYPE rfc SYSTEM "rfc2629.dtd"> -<?rfc toc="yes"?> -<?rfc tocompact="yes"?> -<?rfc tocdepth="3"?> -<?rfc tocindent="yes"?> -<?rfc symrefs="yes"?> -<?rfc sortrefs="yes"?> -<?rfc comments="yes"?> -<?rfc inline="yes"?> -<?rfc compact="yes"?> -<?rfc subcompact="no"?> -<rfc category="info" docName="draft-vsperf-bmwg-vswitch-opnfv-01" - ipr="trust200902"> - <front> - <title abbrev="Benchmarking vSwitches">Benchmarking Virtual Switches in - OPNFV</title> - - <author fullname="Maryam Tahhan" initials="M." surname="Tahhan"> - <organization>Intel</organization> - - <address> - <postal> - <street/> - - <city/> - - <region/> - - <code/> - - <country/> - </postal> - - <phone/> - - <facsimile/> - - <email>maryam.tahhan@intel.com</email> - - <uri/> - </address> - </author> - - <author fullname="Billy O'Mahony" initials="B." surname="O'Mahony"> - <organization>Intel</organization> - - <address> - <postal> - <street/> - - <city/> - - <region/> - - <code/> - - <country/> - </postal> - - <phone/> - - <facsimile/> - - <email>billy.o.mahony@intel.com</email> - - <uri/> - </address> - </author> - - <author fullname="Al Morton" initials="A." surname="Morton"> - <organization>AT&T Labs</organization> - - <address> - <postal> - <street>200 Laurel Avenue South</street> - - <city>Middletown,</city> - - <region>NJ</region> - - <code>07748</code> - - <country>USA</country> - </postal> - - <phone>+1 732 420 1571</phone> - - <facsimile>+1 732 368 1192</facsimile> - - <email>acmorton@att.com</email> - - <uri>http://home.comcast.net/~acmacm/</uri> - </address> - </author> - - <date day="14" month="October" year="2015"/> - - <abstract> - <t>This memo describes the progress of the Open Platform for NFV (OPNFV) - project on virtual switch performance "VSWITCHPERF". This project - intends to build on the current and completed work of the Benchmarking - Methodology Working Group in IETF, by referencing existing literature. - The Benchmarking Methodology Working Group has traditionally conducted - laboratory characterization of dedicated physical implementations of - internetworking functions. Therefore, this memo begins to describe the - additional considerations when virtual switches are implemented in - general-purpose hardware. The expanded tests and benchmarks are also - influenced by the OPNFV mission to support virtualization of the "telco" - infrastructure.</t> - </abstract> - - <note title="Requirements Language"> - <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", - "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this - document are to be interpreted as described in <xref - target="RFC2119">RFC 2119</xref>.</t> - - <t/> - </note> - </front> - - <middle> - <section title="Introduction"> - <t>Benchmarking Methodology Working Group (BMWG) has traditionally - conducted laboratory characterization of dedicated physical - implementations of internetworking functions. The Black-box Benchmarks - of Throughput, Latency, Forwarding Rates and others have served our - industry for many years. Now, Network Function Virtualization (NFV) has - the goal to transform how internetwork functions are implemented, and - therefore has garnered much attention.</t> - - <t>This memo describes the progress of the Open Platform for NFV (OPNFV) - project on virtual switch performance characterization, "VSWITCHPERF". - This project intends to build on the current and completed work of the - Benchmarking Methodology Working Group in IETF, by referencing existing - literature. For example, currently the most often referenced RFC is - <xref target="RFC2544"/> (which depends on <xref target="RFC1242"/>) and - foundation of the benchmarking work in OPNFV is common and strong.</t> - - <t>See - https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases - for more background, and the OPNFV website for general information: - https://www.opnfv.org/</t> - - <t>The authors note that OPNFV distinguishes itself from other open - source compute and networking projects through its emphasis on existing - "telco" services as opposed to cloud-computing. There are many ways in - which telco requirements have different emphasis on performance - dimensions when compared to cloud computing: support for and transfer of - isochronous media streams is one example.</t> - - <t>Note also that the move to NFV Infrastructure has resulted in many - new benchmarking initiatives across the industry, and the authors are - currently doing their best to maintain alignment with many other - projects, and this Internet Draft is evidence of the efforts.</t> - </section> - - <section title="Scope"> - <t>The primary purpose and scope of the memo is to inform BMWG of - work-in-progress that builds on the body of extensive literature and - experience. Additionally, once the initial information conveyed here is - received, this memo may be expanded to include more detail and - commentary from both BMWG and OPNFV communities, under BMWG's chartered - work to characterize the NFV Infrastructure (a virtual switch is an - important aspect of that infrastructure).</t> - </section> - - <section title="Benchmarking Considerations"> - <t>This section highlights some specific considerations (from <xref - target="I-D.ietf-bmwg-virtual-net"/>)related to Benchmarks for virtual - switches. The OPNFV project is sharing its present view on these areas, - as they develop their specifications in the Level Test Design (LTD) - document.</t> - - <section title="Comparison with Physical Network Functions"> - <t>To compare the performance of virtual designs and implementations - with their physical counterparts, identical benchmarks are needed. - BMWG has developed specifications for many network functions this memo - re-uses existing benchmarks through references, and expands them - during development of new methods. A key configuration aspect is the - number of parallel cores required to achieve comparable performance - with a given physical device, or whether some limit of scale was - reached before the cores could achieve the comparable level.</t> - - <t>It's unlikely that the virtual switch will be the only application - running on the SUT, so CPU utilization, Cache utilization, and Memory - footprint should also be recorded for the virtual implementations of - internetworking functions.</t> - </section> - - <section title="Continued Emphasis on Black-Box Benchmarks"> - <t>External observations remain essential as the basis for Benchmarks. - Internal observations with fixed specification and interpretation will - be provided in parallel to assist the development of operations - procedures when the technology is deployed.</t> - </section> - - <section title="New Configuration Parameters"> - <t>A key consideration when conducting any sort of benchmark is trying - to ensure the consistency and repeatability of test results. When - benchmarking the performance of a vSwitch there are many factors that - can affect the consistency of results, one key factor is matching the - various hardware and software details of the SUT. This section lists - some of the many new parameters which this project believes are - critical to report in order to achieve repeatability.</t> - - <t>Hardware details including:</t> - - <t><list style="symbols"> - <t>Platform details</t> - - <t>Processor details</t> - - <t>Memory information (type and size)</t> - - <t>Number of enabled cores</t> - - <t>Number of cores used for the test</t> - - <t>Number of physical NICs, as well as their details - (manufacturer, versions, type and the PCI slot they are plugged - into)</t> - - <t>NIC interrupt configuration</t> - - <t>BIOS version, release date and any configurations that were - modified</t> - - <t>CPU microcode level</t> - - <t>Memory DIMM configurations (quad rank performance may not be - the same as dual rank) in size, freq and slot locations</t> - - <t>PCI configuration parameters (payload size, early ack - option...)</t> - - <t>Power management at all levels (ACPI sleep states, processor - package, OS...)</t> - </list>Software details including:</t> - - <t><list style="symbols"> - <t>OS parameters and behavior (text vs graphical no one typing at - the console on one system)</t> - - <t>OS version (for host and VNF)</t> - - <t>Kernel version (for host and VNF)</t> - - <t>GRUB boot parameters (for host and VNF)</t> - - <t>Hypervisor details (Type and version)</t> - - <t>Selected vSwitch, version number or commit id used</t> - - <t>vSwitch launch command line if it has been parameterised</t> - - <t>Memory allocation to the vSwitch</t> - - <t>which NUMA node it is using, and how many memory channels</t> - - <t>DPDK or any other SW dependency version number or commit id - used</t> - - <t>Memory allocation to a VM - if it's from Hugpages/elsewhere</t> - - <t>VM storage type: snapshot/independent persistent/independent - non-persistent</t> - - <t>Number of VMs</t> - - <t>Number of Virtual NICs (vNICs), versions, type and driver</t> - - <t>Number of virtual CPUs and their core affinity on the host</t> - - <t>Number vNIC interrupt configuration</t> - - <t>Thread affinitization for the applications (including the - vSwitch itself) on the host</t> - - <t>Details of Resource isolation, such as CPUs designated for - Host/Kernel (isolcpu) and CPUs designated for specific processes - (taskset). - Test duration. - Number of flows.</t> - </list></t> - - <t>Test Traffic Information:<list style="symbols"> - <t>Traffic type - UDP, TCP, IMIX / Other</t> - - <t>Packet Sizes</t> - - <t>Deployment Scenario</t> - </list></t> - - <t/> - </section> - - <section title="Flow classification"> - <t>Virtual switches group packets into flows by processing and - matching particular packet or frame header information, or by matching - packets based on the input ports. Thus a flow can be thought of a - sequence of packets that have the same set of header field values or - have arrived on the same port. Performance results can vary based on - the parameters the vSwitch uses to match for a flow. The recommended - flow classification parameters for any vSwitch performance tests are: - the input port, the source IP address, the destination IP address and - the Ethernet protocol type field. It is essential to increase the flow - timeout time on a vSwitch before conducting any performance tests that - do not measure the flow setup time. Normally the first packet of a - particular stream will install the flow in the virtual switch which - adds an additional latency, subsequent packets of the same flow are - not subject to this latency if the flow is already installed on the - vSwitch.</t> - </section> - - <section title="Benchmarks using Baselines with Resource Isolation"> - <t>This outline describes measurement of baseline with isolated - resources at a high level, which is the intended approach at this - time.</t> - - <t><list style="numbers"> - <t>Baselines: <list style="symbols"> - <t>Optional: Benchmark platform forwarding capability without - a vswitch or VNF for at least 72 hours (serves as a means of - platform validation and a means to obtain the base performance - for the platform in terms of its maximum forwarding rate and - latency). <figure> - <preamble>Benchmark platform forwarding - capability</preamble> - - <artwork align="right"><![CDATA[ __ - +--------------------------------------------------+ | - | +------------------------------------------+ | | - | | | | | - | | Simple Forwarding App | | Host - | | | | | - | +------------------------------------------+ | | - | | NIC | | | - +---+------------------------------------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - - <postamble/> - </figure></t> - - <t>Benchmark VNF forwarding capability with direct - connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72 - hours (serves as a means of VNF validation and a means to - obtain the base performance for the VNF in terms of its - maximum forwarding rate and latency). The metrics gathered - from this test will serve as a key comparison point for - vSwitch bypass technologies performance and vSwitch - performance. <figure align="right"> - <preamble>Benchmark VNF forwarding capability</preamble> - - <artwork><![CDATA[ __ - +--------------------------------------------------+ | - | +------------------------------------------+ | | - | | | | | - | | VNF | | | - | | | | | - | +------------------------------------------+ | | - | | Passthrough/SR-IOV | | Host - | +------------------------------------------+ | | - | | NIC | | | - +---+------------------------------------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - - <postamble/> - </figure></t> - - <t>Benchmarking with isolated resources alone, with other - resources (both HW&SW) disabled Example, vSw and VM are - SUT</t> - - <t>Benchmarking with isolated resources alone, leaving some - resources unused</t> - - <t>Benchmark with isolated resources and all resources - occupied</t> - </list></t> - - <t>Next Steps<list style="symbols"> - <t>Limited sharing</t> - - <t>Production scenarios</t> - - <t>Stressful scenarios</t> - </list></t> - </list></t> - </section> - </section> - - <section title="VSWITCHPERF Specification Summary"> - <t>The overall specification in preparation is referred to as a Level - Test Design (LTD) document, which will contain a suite of performance - tests. The base performance tests in the LTD are based on the - pre-existing specifications developed by BMWG to test the performance of - physical switches. These specifications include:</t> - - <t><list style="symbols"> - <t><xref target="RFC2544"/> Benchmarking Methodology for Network - Interconnect Devices</t> - - <t><xref target="RFC2889"/> Benchmarking Methodology for LAN - Switching</t> - - <t><xref target="RFC6201"/> Device Reset Characterization</t> - - <t><xref target="RFC5481"/> Packet Delay Variation Applicability - Statement</t> - </list></t> - - <t>Some of the above/newer RFCs are being applied in benchmarking for - the first time, and represent a development challenge for test equipment - developers. Fortunately, many members of the testing system community - have engaged on the VSPERF project, including an open source test - system.</t> - - <t>In addition to this, the LTD also re-uses the terminology defined - by:</t> - - <t><list style="symbols"> - <t><xref target="RFC2285"/> Benchmarking Terminology for LAN - Switching Devices</t> - - <t><xref target="RFC5481"/> Packet Delay Variation Applicability - Statement</t> - </list></t> - - <t/> - - <t>Specifications to be included in future updates of the LTD - include:<list style="symbols"> - <t><xref target="RFC3918"/> Methodology for IP Multicast - Benchmarking</t> - - <t><xref target="RFC4737"/> Packet Reordering Metrics</t> - </list></t> - - <t>As one might expect, the most fundamental internetworking - characteristics of Throughput and Latency remain important when the - switch is virtualized, and these benchmarks figure prominently in the - specification.</t> - - <t>When considering characteristics important to "telco" network - functions, we must begin to consider additional performance metrics. In - this case, the project specifications have referenced metrics from the - IETF IP Performance Metrics (IPPM) literature. This means that the <xref - target="RFC2544"/> test of Latency is replaced by measurement of a - metric derived from IPPM's <xref target="RFC2679"/>, where a set of - statistical summaries will be provided (mean, max, min, etc.). Further - metrics planned to be benchmarked include packet delay variation as - defined by <xref target="RFC5481"/> , reordering, burst behaviour, DUT - availability, DUT capacity and packet loss in long term testing at - Throughput level, where some low-level of background loss may be present - and characterized.</t> - - <t>Tests have been (or will be) designed to collect the metrics - below:</t> - - <t><list style="symbols"> - <t>Throughput Tests to measure the maximum forwarding rate (in - frames per second or fps) and bit rate (in Mbps) for a constant load - (as defined by RFC1242) without traffic loss.</t> - - <t>Packet and Frame Delay Distribution Tests to measure average, min - and max packet and frame delay for constant loads.</t> - - <t>Packet Delay Tests to understand latency distribution for - different packet sizes and over an extended test run to uncover - outliers.</t> - - <t>Scalability Tests to understand how the virtual switch performs - as the number of flows, active ports, complexity of the forwarding - logic’s configuration… it has to deal with - increases.</t> - - <t>Stream Performance Tests (TCP, UDP) to measure bulk data transfer - performance, i.e. how fast systems can send and receive data through - the switch.</t> - - <t>Control Path and Datapath Coupling Tests, to understand how - closely coupled the datapath and the control path are as well as the - effect of this coupling on the performance of the DUT (example: - delay of the initial packet of a flow).</t> - - <t>CPU and Memory Consumption Tests to understand the virtual - switch’s footprint on the system, usually conducted as - auxiliary measurements with benchmarks above. They include: CPU - utilization, Cache utilization and Memory footprint.</t> - </list></t> - - <t>Future/planned test specs include:<list style="symbols"> - <t>Request/Response Performance Tests (TCP, UDP) which measure the - transaction rate through the switch.</t> - - <t>Noisy Neighbour Tests, to understand the effects of resource - sharing on the performance of a virtual switch.</t> - - <t>Tests derived from examination of ETSI NFV Draft GS IFA003 - requirements <xref target="IFA003"/> on characterization of - acceleration technologies applied to vswitches.</t> - </list>The flexibility of deployment of a virtual switch within a - network means that the BMWG IETF existing literature needs to be used to - characterize the performance of a switch in various deployment - scenarios. The deployment scenarios under consideration include:</t> - - <t><figure> - <preamble>Physical port to virtual switch to physical - port</preamble> - - <artwork><![CDATA[ __ - +--------------------------------------------------+ | - | +--------------------+ | | - | | | | | - | | v | | Host - | +--------------+ +--------------+ | | - | | phy port | vSwitch | phy port | | | - +---+--------------+------------+--------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure></t> - - <t><figure> - <preamble>Physical port to virtual switch to VNF to virtual switch - to physical port</preamble> - - <artwork><![CDATA[ __ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | ^ : | | - | | | | | Guest - | : v | | - | +---------------+ +---------------+ | | - | | logical port 0| | logical port 1| | | - +---+---------------+-----------+---------------+---+ __| - ^ : - | | - : v __ - +---+---------------+----------+---------------+---+ | - | | logical port 0| | logical port 1| | | - | +---------------+ +---------------+ | | - | ^ : | | - | | | | | Host - | : v | | - | +--------------+ +--------------+ | | - | | phy port | vSwitch | phy port | | | - +---+--------------+------------+--------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>Physical port to virtual switch to VNF to virtual switch - to VNF to virtual switch to physical port</preamble> - - <artwork><![CDATA[ __ - +----------------------+ +----------------------+ | - | Guest 1 | | Guest 2 | | - | +---------------+ | | +---------------+ | | - | | Application | | | | Application | | | - | +---------------+ | | +---------------+ | | - | ^ | | | ^ | | | - | | v | | | v | | Guests - | +---------------+ | | +---------------+ | | - | | logical ports | | | | logical ports | | | - | | 0 1 | | | | 0 1 | | | - +---+---------------+--+ +---+---------------+--+__| - ^ : ^ : - | | | | - : v : v _ - +---+---------------+---------+---------------+--+ | - | | 0 1 | | 3 4 | | | - | | logical ports | | logical ports | | | - | +---------------+ +---------------+ | | - | ^ | ^ | | | Host - | | |-----------------| v | | - | +--------------+ +--------------+ | | - | | phy ports | vSwitch | phy ports | | | - +---+--------------+----------+--------------+---+_| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>Physical port to virtual switch to VNF</preamble> - - <artwork><![CDATA[ __ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | ^ | | - | | | | Guest - | : | | - | +---------------+ | | - | | logical port 0| | | - +---+---------------+-------------------------------+ __| - ^ - | - : __ - +---+---------------+------------------------------+ | - | | logical port 0| | | - | +---------------+ | | - | ^ | | - | | | | Host - | : | | - | +--------------+ | | - | | phy port | vSwitch | | - +---+--------------+------------ -------------- ---+ __| - ^ - | - : - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>VNF to virtual switch to physical port</preamble> - - <artwork><![CDATA[ __ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | : | | - | | | | Guest - | v | | - | +---------------+ | | - | | logical port | | | - +-------------------------------+---------------+---+ __| - : - | - v __ - +------------------------------+---------------+---+ | - | | logical port | | | - | +---------------+ | | - | : | | - | | | | Host - | v | | - | +--------------+ | | - | vSwitch | phy port | | | - +-------------------------------+--------------+---+ __| - : - | - v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>VNF to virtual switch to VNF</preamble> - - <artwork><![CDATA[ __ - +----------------------+ +----------------------+ | - | Guest 1 | | Guest 2 | | - | +---------------+ | | +---------------+ | | - | | Application | | | | Application | | | - | +---------------+ | | +---------------+ | | - | | | | ^ | | - | v | | | | | Guests - | +---------------+ | | +---------------+ | | - | | logical ports | | | | logical ports | | | - | | 0 | | | | 0 | | | - +---+---------------+--+ +---+---------------+--+__| - : ^ - | | - v : _ - +---+---------------+---------+---------------+--+ | - | | 1 | | 1 | | | - | | logical ports | | logical ports | | | - | +---------------+ +---------------+ | | - | | ^ | | Host - | L-----------------+ | | - | | | - | vSwitch | | - +------------------------------------------------+_|]]></artwork> - </figure></t> - - <t>A set of Deployment Scenario figures is available on the VSPERF Test - Methodology Wiki page <xref target="TestTopo"/>. </t> - </section> - - <section title="3x3 Matrix Coverage"> - <t>This section organizes the many existing test specifications into the - "3x3" matrix (introduced in <xref target="I-D.ietf-bmwg-virtual-net"/>). - Because the LTD specification ID names are quite long, this section is - organized into lists for each occupied cell of the matrix (not all are - occupied, also the matrix has grown to 3x4 to accommodate scale metrics - when displaying the coverage of many metrics/benchmarks).</t> - - <t>The tests listed below assess the activation of paths in the data - plane, rather than the control plane.</t> - - <t>A complete list of tests with short summaries is available on the - VSPERF "LTD Test Spec Overview" Wiki page <xref target="LTDoverV"/>.</t> - - <section title="Speed of Activation"> - <t><list style="symbols"> - <t>Activation.RFC2889.AddressLearningRate</t> - - <t>PacketLatency.InitialPacketProcessingLatency</t> - </list></t> - </section> - - <section title="Accuracy of Activation section"> - <t><list style="symbols"> - <t>CPDP.Coupling.Flow.Addition</t> - </list></t> - </section> - - <section title="Reliability of Activation"> - <t><list style="symbols"> - <t>Throughput.RFC2544.SystemRecoveryTime</t> - - <t>Throughput.RFC2544.ResetTime</t> - </list></t> - </section> - - <section title="Scale of Activation"> - <t><list style="symbols"> - <t>Activation.RFC2889.AddressCachingCapacity</t> - </list></t> - </section> - - <section title="Speed of Operation"> - <t><list style="symbols"> - <t>Throughput.RFC2544.PacketLossRate</t> - - <t>CPU.RFC2544.0PacketLoss</t> - - <t>Throughput.RFC2544.PacketLossRateFrameModification</t> - - <t>Throughput.RFC2544.BackToBackFrames</t> - - <t>Throughput.RFC2889.MaxForwardingRate</t> - - <t>Throughput.RFC2889.ForwardPressure</t> - - <t>Throughput.RFC2889.BroadcastFrameForwarding</t> - </list></t> - </section> - - <section title="Accuracy of Operation"> - <t><list style="symbols"> - <t>Throughput.RFC2889.ErrorFramesFiltering</t> - - <t>Throughput.RFC2544.Profile</t> - </list></t> - </section> - - <section title="Reliability of Operation"> - <t><list style="symbols"> - <t>Throughput.RFC2889.Soak</t> - - <t>Throughput.RFC2889.SoakFrameModification</t> - - <t>PacketDelayVariation.RFC3393.Soak</t> - </list></t> - </section> - - <section title="Scalability of Operation"> - <t><list style="symbols"> - <t>Scalability.RFC2544.0PacketLoss</t> - - <t>MemoryBandwidth.RFC2544.0PacketLoss.Scalability</t> - </list></t> - </section> - - <section title="Summary"> - <t><figure> - <artwork><![CDATA[|------------------------------------------------------------------------| -| | | | | | -| | SPEED | ACCURACY | RELIABILITY | SCALE | -| | | | | | -|------------------------------------------------------------------------| -| | | | | | -| Activation | X | X | X | X | -| | | | | | -|------------------------------------------------------------------------| -| | | | | | -| Operation | X | X | X | X | -| | | | | | -|------------------------------------------------------------------------| -| | | | | | -| De-activation | | | | | -| | | | | | -|------------------------------------------------------------------------|]]></artwork> - </figure></t> - </section> - </section> - - <section title="Security Considerations"> - <t>Benchmarking activities as described in this memo are limited to - technology characterization of a Device Under Test/System Under Test - (DUT/SUT) using controlled stimuli in a laboratory environment, with - dedicated address space and the constraints specified in the sections - above.</t> - - <t>The benchmarking network topology will be an independent test setup - and MUST NOT be connected to devices that may forward the test traffic - into a production network, or misroute traffic to the test management - network.</t> - - <t>Further, benchmarking is performed on a "black-box" basis, relying - solely on measurements observable external to the DUT/SUT.</t> - - <t>Special capabilities SHOULD NOT exist in the DUT/SUT specifically for - benchmarking purposes. Any implications for network security arising - from the DUT/SUT SHOULD be identical in the lab and in production - networks.</t> - </section> - - <section anchor="IANA" title="IANA Considerations"> - <t>No IANA Action is requested at this time.</t> - </section> - - <section title="Acknowledgements"> - <t>The authors acknowledge</t> - </section> - </middle> - - <back> - <references title="Normative References"> - <?rfc ?> - - <?rfc include="reference.RFC.2119"?> - - <?rfc include="reference.RFC.2330"?> - - <?rfc include='reference.RFC.2544'?> - - <?rfc include="reference.RFC.2679"?> - - <?rfc include='reference.RFC.2680'?> - - <?rfc include='reference.RFC.3393'?> - - <?rfc include='reference.RFC.3432'?> - - <?rfc include='reference.RFC.2681'?> - - <?rfc include='reference.RFC.5905'?> - - <?rfc include='reference.RFC.4689'?> - - <?rfc include='reference.RFC.4737'?> - - <?rfc include='reference.RFC.5357'?> - - <?rfc include='reference.RFC.2889'?> - - <?rfc include='reference.RFC.3918'?> - - <?rfc include='reference.RFC.6201'?> - - <?rfc include='reference.RFC.2285'?> - - <reference anchor="NFV.PER001"> - <front> - <title>Network Function Virtualization: Performance and Portability - Best Practices</title> - - <author fullname="ETSI NFV" initials="" surname=""> - <organization/> - </author> - - <date month="June" year="2014"/> - </front> - - <seriesInfo name="Group Specification" - value="ETSI GS NFV-PER 001 V1.1.1 (2014-06)"/> - - <format type="PDF"/> - </reference> - </references> - - <references title="Informative References"> - <?rfc include='reference.RFC.1242'?> - - <?rfc include='reference.RFC.5481'?> - - <?rfc include='reference.RFC.6049'?> - - <?rfc include='reference.RFC.6248'?> - - <?rfc include='reference.RFC.6390'?> - - <?rfc include='reference.I-D.ietf-bmwg-virtual-net'?> - - <reference anchor="TestTopo"> - <front> - <title>Test Topologies - https://wiki.opnfv.org/vsperf/test_methodology</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="LTDoverV"> - <front> - <title>LTD Test Spec Overview - https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review </title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="IFA003"> - <front> - <title>https://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA003_Acceleration_-_vSwitch_Spec/</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - </references> - </back> -</rfc> diff --git a/docs/testing/developer/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.xml b/docs/testing/developer/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.xml deleted file mode 100644 index a9405a77..00000000 --- a/docs/testing/developer/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.xml +++ /dev/null @@ -1,1016 +0,0 @@ -<?xml version="1.0" encoding="US-ASCII"?> -<!DOCTYPE rfc SYSTEM "rfc2629.dtd"> -<?rfc toc="yes"?> -<?rfc tocompact="yes"?> -<?rfc tocdepth="3"?> -<?rfc tocindent="yes"?> -<?rfc symrefs="yes"?> -<?rfc sortrefs="yes"?> -<?rfc comments="yes"?> -<?rfc inline="yes"?> -<?rfc compact="yes"?> -<?rfc subcompact="no"?> -<rfc category="info" docName="draft-vsperf-bmwg-vswitch-opnfv-02" - ipr="trust200902"> - <front> - <title abbrev="Benchmarking vSwitches">Benchmarking Virtual Switches in - OPNFV</title> - - <author fullname="Maryam Tahhan" initials="M." surname="Tahhan"> - <organization>Intel</organization> - - <address> - <postal> - <street/> - - <city/> - - <region/> - - <code/> - - <country/> - </postal> - - <phone/> - - <facsimile/> - - <email>maryam.tahhan@intel.com</email> - - <uri/> - </address> - </author> - - <author fullname="Billy O'Mahony" initials="B." surname="O'Mahony"> - <organization>Intel</organization> - - <address> - <postal> - <street/> - - <city/> - - <region/> - - <code/> - - <country/> - </postal> - - <phone/> - - <facsimile/> - - <email>billy.o.mahony@intel.com</email> - - <uri/> - </address> - </author> - - <author fullname="Al Morton" initials="A." surname="Morton"> - <organization>AT&T Labs</organization> - - <address> - <postal> - <street>200 Laurel Avenue South</street> - - <city>Middletown,</city> - - <region>NJ</region> - - <code>07748</code> - - <country>USA</country> - </postal> - - <phone>+1 732 420 1571</phone> - - <facsimile>+1 732 368 1192</facsimile> - - <email>acmorton@att.com</email> - - <uri>http://home.comcast.net/~acmacm/</uri> - </address> - </author> - - <date day="20" month="March" year="2016"/> - - <abstract> - <t>This memo describes the progress of the Open Platform for NFV (OPNFV) - project on virtual switch performance "VSWITCHPERF". This project - intends to build on the current and completed work of the Benchmarking - Methodology Working Group in IETF, by referencing existing literature. - The Benchmarking Methodology Working Group has traditionally conducted - laboratory characterization of dedicated physical implementations of - internetworking functions. Therefore, this memo begins to describe the - additional considerations when virtual switches are implemented in - general-purpose hardware. The expanded tests and benchmarks are also - influenced by the OPNFV mission to support virtualization of the "telco" - infrastructure.</t> - </abstract> - - <note title="Requirements Language"> - <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", - "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this - document are to be interpreted as described in <xref - target="RFC2119">RFC 2119</xref>.</t> - - <t/> - </note> - </front> - - <middle> - <section title="Introduction"> - <t>Benchmarking Methodology Working Group (BMWG) has traditionally - conducted laboratory characterization of dedicated physical - implementations of internetworking functions. The Black-box Benchmarks - of Throughput, Latency, Forwarding Rates and others have served our - industry for many years. Now, Network Function Virtualization (NFV) has - the goal to transform how internetwork functions are implemented, and - therefore has garnered much attention.</t> - - <t>This memo summarizes the progress of the Open Platform for NFV - (OPNFV) project on virtual switch performance characterization, - "VSWITCHPERF", through the Brahmaputra (second) release <xref - target="BrahRel"/>. This project intends to build on the current and - completed work of the Benchmarking Methodology Working Group in IETF, by - referencing existing literature. For example, currently the most often - referenced RFC is <xref target="RFC2544"/> (which depends on <xref - target="RFC1242"/>) and foundation of the benchmarking work in OPNFV is - common and strong.</t> - - <t>See - https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases - for more background, and the OPNFV website for general information: - https://www.opnfv.org/</t> - - <t>The authors note that OPNFV distinguishes itself from other open - source compute and networking projects through its emphasis on existing - "telco" services as opposed to cloud-computing. There are many ways in - which telco requirements have different emphasis on performance - dimensions when compared to cloud computing: support for and transfer of - isochronous media streams is one example.</t> - - <t>Note also that the move to NFV Infrastructure has resulted in many - new benchmarking initiatives across the industry. The authors are - currently doing their best to maintain alignment with many other - projects, and this Internet Draft is one part of the efforts. We - acknowledge the early work in <xref - target="I-D.huang-bmwg-virtual-network-performance"/>, and useful - discussion with the authors.</t> - </section> - - <section title="Scope"> - <t>The primary purpose and scope of the memo is to inform the industry - of work-in-progress that builds on the body of extensive BMWG literature - and experience, and describe the extensions needed for benchmarking - virtual switches. Inital feedback indicates that many of these - extensions may be applicable beyond the current scope (to hardware - switches in the NFV Infrastructure and to virtual routers, for example). - Additionally, this memo serves as a vehicle to include more detail and - commentary from BMWG and other Open Source communities, under BMWG's - chartered work to characterize the NFV Infrastructure (a virtual switch - is an important aspect of that infrastructure).</t> - </section> - - <section title="Benchmarking Considerations"> - <t>This section highlights some specific considerations (from <xref - target="I-D.ietf-bmwg-virtual-net"/>)related to Benchmarks for virtual - switches. The OPNFV project is sharing its present view on these areas, - as they develop their specifications in the Level Test Design (LTD) - document.</t> - - <section title="Comparison with Physical Network Functions"> - <t>To compare the performance of virtual designs and implementations - with their physical counterparts, identical benchmarks are needed. - BMWG has developed specifications for many network functions this memo - re-uses existing benchmarks through references, and expands them - during development of new methods. A key configuration aspect is the - number of parallel cores required to achieve comparable performance - with a given physical device, or whether some limit of scale was - reached before the cores could achieve the comparable level.</t> - - <t>It's unlikely that the virtual switch will be the only application - running on the SUT, so CPU utilization, Cache utilization, and Memory - footprint should also be recorded for the virtual implementations of - internetworking functions.</t> - </section> - - <section title="Continued Emphasis on Black-Box Benchmarks"> - <t>External observations remain essential as the basis for Benchmarks. - Internal observations with fixed specification and interpretation will - be provided in parallel to assist the development of operations - procedures when the technology is deployed.</t> - </section> - - <section title="New Configuration Parameters"> - <t>A key consideration when conducting any sort of benchmark is trying - to ensure the consistency and repeatability of test results. When - benchmarking the performance of a vSwitch there are many factors that - can affect the consistency of results, one key factor is matching the - various hardware and software details of the SUT. This section lists - some of the many new parameters which this project believes are - critical to report in order to achieve repeatability.</t> - - <t>Hardware details including:</t> - - <t><list style="symbols"> - <t>Platform details</t> - - <t>Processor details</t> - - <t>Memory information (type and size)</t> - - <t>Number of enabled cores</t> - - <t>Number of cores used for the test</t> - - <t>Number of physical NICs, as well as their details - (manufacturer, versions, type and the PCI slot they are plugged - into)</t> - - <t>NIC interrupt configuration</t> - - <t>BIOS version, release date and any configurations that were - modified</t> - - <t>CPU microcode level</t> - - <t>Memory DIMM configurations (quad rank performance may not be - the same as dual rank) in size, freq and slot locations</t> - - <t>PCI configuration parameters (payload size, early ack - option...)</t> - - <t>Power management at all levels (ACPI sleep states, processor - package, OS...)</t> - </list>Software details including:</t> - - <t><list style="symbols"> - <t>OS parameters and behavior (text vs graphical no one typing at - the console on one system)</t> - - <t>OS version (for host and VNF)</t> - - <t>Kernel version (for host and VNF)</t> - - <t>GRUB boot parameters (for host and VNF)</t> - - <t>Hypervisor details (Type and version)</t> - - <t>Selected vSwitch, version number or commit id used</t> - - <t>vSwitch launch command line if it has been parameterised</t> - - <t>Memory allocation to the vSwitch</t> - - <t>which NUMA node it is using, and how many memory channels</t> - - <t>DPDK or any other SW dependency version number or commit id - used</t> - - <t>Memory allocation to a VM - if it's from Hugpages/elsewhere</t> - - <t>VM storage type: snapshot/independent persistent/independent - non-persistent</t> - - <t>Number of VMs</t> - - <t>Number of Virtual NICs (vNICs), versions, type and driver</t> - - <t>Number of virtual CPUs and their core affinity on the host</t> - - <t>Number vNIC interrupt configuration</t> - - <t>Thread affinitization for the applications (including the - vSwitch itself) on the host</t> - - <t>Details of Resource isolation, such as CPUs designated for - Host/Kernel (isolcpu) and CPUs designated for specific processes - (taskset). - Test duration. - Number of flows.</t> - </list></t> - - <t>Test Traffic Information:<list style="symbols"> - <t>Traffic type - UDP, TCP, IMIX / Other</t> - - <t>Packet Sizes</t> - - <t>Deployment Scenario</t> - </list></t> - - <t/> - </section> - - <section title="Flow classification"> - <t>Virtual switches group packets into flows by processing and - matching particular packet or frame header information, or by matching - packets based on the input ports. Thus a flow can be thought of a - sequence of packets that have the same set of header field values or - have arrived on the same port. Performance results can vary based on - the parameters the vSwitch uses to match for a flow. The recommended - flow classification parameters for any vSwitch performance tests are: - the input port, the source IP address, the destination IP address and - the Ethernet protocol type field. It is essential to increase the flow - timeout time on a vSwitch before conducting any performance tests that - do not measure the flow setup time. Normally the first packet of a - particular stream will install the flow in the virtual switch which - adds an additional latency, subsequent packets of the same flow are - not subject to this latency if the flow is already installed on the - vSwitch.</t> - </section> - - <section title="Benchmarks using Baselines with Resource Isolation"> - <t>This outline describes measurement of baseline with isolated - resources at a high level, which is the intended approach at this - time.</t> - - <t><list style="numbers"> - <t>Baselines: <list style="symbols"> - <t>Optional: Benchmark platform forwarding capability without - a vswitch or VNF for at least 72 hours (serves as a means of - platform validation and a means to obtain the base performance - for the platform in terms of its maximum forwarding rate and - latency). <figure> - <preamble>Benchmark platform forwarding - capability</preamble> - - <artwork align="right"><![CDATA[ __ - +--------------------------------------------------+ | - | +------------------------------------------+ | | - | | | | | - | | Simple Forwarding App | | Host - | | | | | - | +------------------------------------------+ | | - | | NIC | | | - +---+------------------------------------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - - <postamble/> - </figure></t> - - <t>Benchmark VNF forwarding capability with direct - connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72 - hours (serves as a means of VNF validation and a means to - obtain the base performance for the VNF in terms of its - maximum forwarding rate and latency). The metrics gathered - from this test will serve as a key comparison point for - vSwitch bypass technologies performance and vSwitch - performance. <figure align="right"> - <preamble>Benchmark VNF forwarding capability</preamble> - - <artwork><![CDATA[ __ - +--------------------------------------------------+ | - | +------------------------------------------+ | | - | | | | | - | | VNF | | | - | | | | | - | +------------------------------------------+ | | - | | Passthrough/SR-IOV | | Host - | +------------------------------------------+ | | - | | NIC | | | - +---+------------------------------------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - - <postamble/> - </figure></t> - - <t>Benchmarking with isolated resources alone, with other - resources (both HW&SW) disabled Example, vSw and VM are - SUT</t> - - <t>Benchmarking with isolated resources alone, leaving some - resources unused</t> - - <t>Benchmark with isolated resources and all resources - occupied</t> - </list></t> - - <t>Next Steps<list style="symbols"> - <t>Limited sharing</t> - - <t>Production scenarios</t> - - <t>Stressful scenarios</t> - </list></t> - </list></t> - </section> - </section> - - <section title="VSWITCHPERF Specification Summary"> - <t>The overall specification in preparation is referred to as a Level - Test Design (LTD) document, which will contain a suite of performance - tests. The base performance tests in the LTD are based on the - pre-existing specifications developed by BMWG to test the performance of - physical switches. These specifications include:</t> - - <t><list style="symbols"> - <t><xref target="RFC2544"/> Benchmarking Methodology for Network - Interconnect Devices</t> - - <t><xref target="RFC2889"/> Benchmarking Methodology for LAN - Switching</t> - - <t><xref target="RFC6201"/> Device Reset Characterization</t> - - <t><xref target="RFC5481"/> Packet Delay Variation Applicability - Statement</t> - </list></t> - - <t>Some of the above/newer RFCs are being applied in benchmarking for - the first time, and represent a development challenge for test equipment - developers. Fortunately, many members of the testing system community - have engaged on the VSPERF project, including an open source test - system.</t> - - <t>In addition to this, the LTD also re-uses the terminology defined - by:</t> - - <t><list style="symbols"> - <t><xref target="RFC2285"/> Benchmarking Terminology for LAN - Switching Devices</t> - - <t><xref target="RFC5481"/> Packet Delay Variation Applicability - Statement</t> - </list></t> - - <t/> - - <t>Specifications to be included in future updates of the LTD - include:<list style="symbols"> - <t><xref target="RFC3918"/> Methodology for IP Multicast - Benchmarking</t> - - <t><xref target="RFC4737"/> Packet Reordering Metrics</t> - </list></t> - - <t>As one might expect, the most fundamental internetworking - characteristics of Throughput and Latency remain important when the - switch is virtualized, and these benchmarks figure prominently in the - specification.</t> - - <t>When considering characteristics important to "telco" network - functions, we must begin to consider additional performance metrics. In - this case, the project specifications have referenced metrics from the - IETF IP Performance Metrics (IPPM) literature. This means that the <xref - target="RFC2544"/> test of Latency is replaced by measurement of a - metric derived from IPPM's <xref target="RFC2679"/>, where a set of - statistical summaries will be provided (mean, max, min, etc.). Further - metrics planned to be benchmarked include packet delay variation as - defined by <xref target="RFC5481"/> , reordering, burst behaviour, DUT - availability, DUT capacity and packet loss in long term testing at - Throughput level, where some low-level of background loss may be present - and characterized.</t> - - <t>Tests have been (or will be) designed to collect the metrics - below:</t> - - <t><list style="symbols"> - <t>Throughput Tests to measure the maximum forwarding rate (in - frames per second or fps) and bit rate (in Mbps) for a constant load - (as defined by <xref target="RFC1242"/>) without traffic loss.</t> - - <t>Packet and Frame Delay Distribution Tests to measure average, min - and max packet and frame delay for constant loads.</t> - - <t>Packet Delay Tests to understand latency distribution for - different packet sizes and over an extended test run to uncover - outliers.</t> - - <t>Scalability Tests to understand how the virtual switch performs - as the number of flows, active ports, complexity of the forwarding - logic’s configuration… it has to deal with - increases.</t> - - <t>Stream Performance Tests (TCP, UDP) to measure bulk data transfer - performance, i.e. how fast systems can send and receive data through - the switch.</t> - - <t>Control Path and Datapath Coupling Tests, to understand how - closely coupled the datapath and the control path are as well as the - effect of this coupling on the performance of the DUT (example: - delay of the initial packet of a flow).</t> - - <t>CPU and Memory Consumption Tests to understand the virtual - switch’s footprint on the system, usually conducted as - auxiliary measurements with benchmarks above. They include: CPU - utilization, Cache utilization and Memory footprint.</t> - - <t>The so-called "Soak" tests, where the selected test is conducted - over a long period of time (with an ideal duration of 24 hours, and - at least 6 hours). The purpose of soak tests is to capture transient - changes in performance which may occur due to infrequent processes - or the low probability coincidence of two or more processes. The - performance must be evaluated periodically during continuous - testing, and this results in use of <xref target="RFC2889"/> Frame - Rate metrics instead of <xref target="RFC2544"/> Throughput (which - requires stopping traffic to allow time for all traffic to exit - internal queues).</t> - </list></t> - - <t>Future/planned test specs include:<list style="symbols"> - <t>Request/Response Performance Tests (TCP, UDP) which measure the - transaction rate through the switch.</t> - - <t>Noisy Neighbour Tests, to understand the effects of resource - sharing on the performance of a virtual switch.</t> - - <t>Tests derived from examination of ETSI NFV Draft GS IFA003 - requirements <xref target="IFA003"/> on characterization of - acceleration technologies applied to vswitches.</t> - </list>The flexibility of deployment of a virtual switch within a - network means that the BMWG IETF existing literature needs to be used to - characterize the performance of a switch in various deployment - scenarios. The deployment scenarios under consideration include:</t> - - <t><figure> - <preamble>Physical port to virtual switch to physical - port</preamble> - - <artwork><![CDATA[ __ - +--------------------------------------------------+ | - | +--------------------+ | | - | | | | | - | | v | | Host - | +--------------+ +--------------+ | | - | | phy port | vSwitch | phy port | | | - +---+--------------+------------+--------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure></t> - - <t><figure> - <preamble>Physical port to virtual switch to VNF to virtual switch - to physical port</preamble> - - <artwork><![CDATA[ __ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | ^ : | | - | | | | | Guest - | : v | | - | +---------------+ +---------------+ | | - | | logical port 0| | logical port 1| | | - +---+---------------+-----------+---------------+---+ __| - ^ : - | | - : v __ - +---+---------------+----------+---------------+---+ | - | | logical port 0| | logical port 1| | | - | +---------------+ +---------------+ | | - | ^ : | | - | | | | | Host - | : v | | - | +--------------+ +--------------+ | | - | | phy port | vSwitch | phy port | | | - +---+--------------+------------+--------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>Physical port to virtual switch to VNF to virtual switch - to VNF to virtual switch to physical port</preamble> - - <artwork><![CDATA[ __ - +----------------------+ +----------------------+ | - | Guest 1 | | Guest 2 | | - | +---------------+ | | +---------------+ | | - | | Application | | | | Application | | | - | +---------------+ | | +---------------+ | | - | ^ | | | ^ | | | - | | v | | | v | | Guests - | +---------------+ | | +---------------+ | | - | | logical ports | | | | logical ports | | | - | | 0 1 | | | | 0 1 | | | - +---+---------------+--+ +---+---------------+--+__| - ^ : ^ : - | | | | - : v : v _ - +---+---------------+---------+---------------+--+ | - | | 0 1 | | 3 4 | | | - | | logical ports | | logical ports | | | - | +---------------+ +---------------+ | | - | ^ | ^ | | | Host - | | |-----------------| v | | - | +--------------+ +--------------+ | | - | | phy ports | vSwitch | phy ports | | | - +---+--------------+----------+--------------+---+_| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>Physical port to virtual switch to VNF</preamble> - - <artwork><![CDATA[ __ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | ^ | | - | | | | Guest - | : | | - | +---------------+ | | - | | logical port 0| | | - +---+---------------+-------------------------------+ __| - ^ - | - : __ - +---+---------------+------------------------------+ | - | | logical port 0| | | - | +---------------+ | | - | ^ | | - | | | | Host - | : | | - | +--------------+ | | - | | phy port | vSwitch | | - +---+--------------+------------ -------------- ---+ __| - ^ - | - : - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>VNF to virtual switch to physical port</preamble> - - <artwork><![CDATA[ __ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | : | | - | | | | Guest - | v | | - | +---------------+ | | - | | logical port | | | - +-------------------------------+---------------+---+ __| - : - | - v __ - +------------------------------+---------------+---+ | - | | logical port | | | - | +---------------+ | | - | : | | - | | | | Host - | v | | - | +--------------+ | | - | vSwitch | phy port | | | - +-------------------------------+--------------+---+ __| - : - | - v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>VNF to virtual switch to VNF</preamble> - - <artwork><![CDATA[ __ - +----------------------+ +----------------------+ | - | Guest 1 | | Guest 2 | | - | +---------------+ | | +---------------+ | | - | | Application | | | | Application | | | - | +---------------+ | | +---------------+ | | - | | | | ^ | | - | v | | | | | Guests - | +---------------+ | | +---------------+ | | - | | logical ports | | | | logical ports | | | - | | 0 | | | | 0 | | | - +---+---------------+--+ +---+---------------+--+__| - : ^ - | | - v : _ - +---+---------------+---------+---------------+--+ | - | | 1 | | 1 | | | - | | logical ports | | logical ports | | | - | +---------------+ +---------------+ | | - | | ^ | | Host - | L-----------------+ | | - | | | - | vSwitch | | - +------------------------------------------------+_|]]></artwork> - </figure></t> - - <t>A set of Deployment Scenario figures is available on the VSPERF Test - Methodology Wiki page <xref target="TestTopo"/>.</t> - </section> - - <section title="3x3 Matrix Coverage"> - <t>This section organizes the many existing test specifications into the - "3x3" matrix (introduced in <xref target="I-D.ietf-bmwg-virtual-net"/>). - Because the LTD specification ID names are quite long, this section is - organized into lists for each occupied cell of the matrix (not all are - occupied, also the matrix has grown to 3x4 to accommodate scale metrics - when displaying the coverage of many metrics/benchmarks). The current - version of the LTD specification is available <xref target="LTD"/>.</t> - - <t>The tests listed below assess the activation of paths in the data - plane, rather than the control plane.</t> - - <t>A complete list of tests with short summaries is available on the - VSPERF "LTD Test Spec Overview" Wiki page <xref target="LTDoverV"/>.</t> - - <section title="Speed of Activation"> - <t><list style="symbols"> - <t>Activation.RFC2889.AddressLearningRate</t> - - <t>PacketLatency.InitialPacketProcessingLatency</t> - </list></t> - </section> - - <section title="Accuracy of Activation section"> - <t><list style="symbols"> - <t>CPDP.Coupling.Flow.Addition</t> - </list></t> - </section> - - <section title="Reliability of Activation"> - <t><list style="symbols"> - <t>Throughput.RFC2544.SystemRecoveryTime</t> - - <t>Throughput.RFC2544.ResetTime</t> - </list></t> - </section> - - <section title="Scale of Activation"> - <t><list style="symbols"> - <t>Activation.RFC2889.AddressCachingCapacity</t> - </list></t> - </section> - - <section title="Speed of Operation"> - <t><list style="symbols"> - <t>Throughput.RFC2544.PacketLossRate</t> - - <t>CPU.RFC2544.0PacketLoss</t> - - <t>Throughput.RFC2544.PacketLossRateFrameModification</t> - - <t>Throughput.RFC2544.BackToBackFrames</t> - - <t>Throughput.RFC2889.MaxForwardingRate</t> - - <t>Throughput.RFC2889.ForwardPressure</t> - - <t>Throughput.RFC2889.BroadcastFrameForwarding</t> - </list></t> - </section> - - <section title="Accuracy of Operation"> - <t><list style="symbols"> - <t>Throughput.RFC2889.ErrorFramesFiltering</t> - - <t>Throughput.RFC2544.Profile</t> - </list></t> - </section> - - <section title="Reliability of Operation"> - <t><list style="symbols"> - <t>Throughput.RFC2889.Soak</t> - - <t>Throughput.RFC2889.SoakFrameModification</t> - - <t>PacketDelayVariation.RFC3393.Soak</t> - </list></t> - </section> - - <section title="Scalability of Operation"> - <t><list style="symbols"> - <t>Scalability.RFC2544.0PacketLoss</t> - - <t>MemoryBandwidth.RFC2544.0PacketLoss.Scalability</t> - </list></t> - </section> - - <section title="Summary"> - <t><figure> - <artwork><![CDATA[|------------------------------------------------------------------------| -| | | | | | -| | SPEED | ACCURACY | RELIABILITY | SCALE | -| | | | | | -|------------------------------------------------------------------------| -| | | | | | -| Activation | X | X | X | X | -| | | | | | -|------------------------------------------------------------------------| -| | | | | | -| Operation | X | X | X | X | -| | | | | | -|------------------------------------------------------------------------| -| | | | | | -| De-activation | | | | | -| | | | | | -|------------------------------------------------------------------------|]]></artwork> - </figure></t> - </section> - </section> - - <section title="Security Considerations"> - <t>Benchmarking activities as described in this memo are limited to - technology characterization of a Device Under Test/System Under Test - (DUT/SUT) using controlled stimuli in a laboratory environment, with - dedicated address space and the constraints specified in the sections - above.</t> - - <t>The benchmarking network topology will be an independent test setup - and MUST NOT be connected to devices that may forward the test traffic - into a production network, or misroute traffic to the test management - network.</t> - - <t>Further, benchmarking is performed on a "black-box" basis, relying - solely on measurements observable external to the DUT/SUT.</t> - - <t>Special capabilities SHOULD NOT exist in the DUT/SUT specifically for - benchmarking purposes. Any implications for network security arising - from the DUT/SUT SHOULD be identical in the lab and in production - networks.</t> - </section> - - <section anchor="IANA" title="IANA Considerations"> - <t>No IANA Action is requested at this time.</t> - </section> - - <section title="Acknowledgements"> - <t>The authors appreciate and acknowledge comments from Scott Bradner, - Marius Georgescu, Ramki Krishnan, and Doug Montgomery, and others for - their reviews.</t> - </section> - </middle> - - <back> - <references title="Normative References"> - <?rfc ?> - - <?rfc include="reference.RFC.2119"?> - - <?rfc ?> - - <?rfc include="reference.RFC.2330"?> - - <?rfc include='reference.RFC.2544'?> - - <?rfc include="reference.RFC.2679"?> - - <?rfc include='reference.RFC.2680'?> - - <?rfc include='reference.RFC.3393'?> - - <?rfc include='reference.RFC.3432'?> - - <?rfc include='reference.RFC.2681'?> - - <?rfc include='reference.RFC.5905'?> - - <?rfc include='reference.RFC.4689'?> - - <?rfc include='reference.RFC.4737'?> - - <?rfc include='reference.RFC.5357'?> - - <?rfc include='reference.RFC.2889'?> - - <?rfc include='reference.RFC.3918'?> - - <?rfc include='reference.RFC.6201'?> - - <?rfc include='reference.RFC.2285'?> - - <reference anchor="NFV.PER001"> - <front> - <title>Network Function Virtualization: Performance and Portability - Best Practices</title> - - <author fullname="ETSI NFV" initials="" surname=""> - <organization/> - </author> - - <date month="June" year="2014"/> - </front> - - <seriesInfo name="Group Specification" - value="ETSI GS NFV-PER 001 V1.1.1 (2014-06)"/> - - <format type="PDF"/> - </reference> - </references> - - <references title="Informative References"> - <?rfc include='reference.RFC.1242'?> - - <?rfc include='reference.RFC.5481'?> - - <?rfc include='reference.RFC.6049'?> - - <?rfc include='reference.RFC.6248'?> - - <?rfc include='reference.RFC.6390'?> - - <?rfc include='reference.I-D.ietf-bmwg-virtual-net'?> - - <?rfc include='reference.I-D.huang-bmwg-virtual-network-performance'?> - - <reference anchor="TestTopo"> - <front> - <title>Test Topologies - https://wiki.opnfv.org/vsperf/test_methodology</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="LTDoverV"> - <front> - <title>LTD Test Spec Overview - https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="LTD"> - <front> - <title>LTD Test Specification - http://artifacts.opnfv.org/vswitchperf/docs/requirements/index.html</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="BrahRel"> - <front> - <title>Brahmaputra, Second OPNFV Release - https://www.opnfv.org/brahmaputra</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="IFA003"> - <front> - <title>https://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA003_Acceleration_-_vSwitch_Spec/</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - </references> - </back> -</rfc> diff --git a/docs/testing/developer/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-02.xml b/docs/testing/developer/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-02.xml deleted file mode 100644 index 9157763e..00000000 --- a/docs/testing/developer/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-02.xml +++ /dev/null @@ -1,1016 +0,0 @@ -<?xml version="1.0" encoding="US-ASCII"?> -<!DOCTYPE rfc SYSTEM "rfc2629.dtd"> -<?rfc toc="yes"?> -<?rfc tocompact="yes"?> -<?rfc tocdepth="3"?> -<?rfc tocindent="yes"?> -<?rfc symrefs="yes"?> -<?rfc sortrefs="yes"?> -<?rfc comments="yes"?> -<?rfc inline="yes"?> -<?rfc compact="yes"?> -<?rfc subcompact="no"?> -<rfc category="info" docName="draft-vsperf-bmwg-vswitch-opnfv-02" - ipr="trust200902"> - <front> - <title abbrev="Benchmarking vSwitches">Benchmarking Virtual Switches in - OPNFV</title> - - <author fullname="Maryam Tahhan" initials="M." surname="Tahhan"> - <organization>Intel</organization> - - <address> - <postal> - <street/> - - <city/> - - <region/> - - <code/> - - <country/> - </postal> - - <phone/> - - <facsimile/> - - <email>maryam.tahhan@intel.com</email> - - <uri/> - </address> - </author> - - <author fullname="Billy O'Mahony" initials="B." surname="O'Mahony"> - <organization>Intel</organization> - - <address> - <postal> - <street/> - - <city/> - - <region/> - - <code/> - - <country/> - </postal> - - <phone/> - - <facsimile/> - - <email>billy.o.mahony@intel.com</email> - - <uri/> - </address> - </author> - - <author fullname="Al Morton" initials="A." surname="Morton"> - <organization>AT&T Labs</organization> - - <address> - <postal> - <street>200 Laurel Avenue South</street> - - <city>Middletown,</city> - - <region>NJ</region> - - <code>07748</code> - - <country>USA</country> - </postal> - - <phone>+1 732 420 1571</phone> - - <facsimile>+1 732 368 1192</facsimile> - - <email>acmorton@att.com</email> - - <uri>http://home.comcast.net/~acmacm/</uri> - </address> - </author> - - <date day="21" month="March" year="2016"/> - - <abstract> - <t>This memo describes the progress of the Open Platform for NFV (OPNFV) - project on virtual switch performance "VSWITCHPERF". This project - intends to build on the current and completed work of the Benchmarking - Methodology Working Group in IETF, by referencing existing literature. - The Benchmarking Methodology Working Group has traditionally conducted - laboratory characterization of dedicated physical implementations of - internetworking functions. Therefore, this memo begins to describe the - additional considerations when virtual switches are implemented in - general-purpose hardware. The expanded tests and benchmarks are also - influenced by the OPNFV mission to support virtualization of the "telco" - infrastructure.</t> - </abstract> - - <note title="Requirements Language"> - <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", - "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this - document are to be interpreted as described in <xref - target="RFC2119">RFC 2119</xref>.</t> - - <t/> - </note> - </front> - - <middle> - <section title="Introduction"> - <t>Benchmarking Methodology Working Group (BMWG) has traditionally - conducted laboratory characterization of dedicated physical - implementations of internetworking functions. The Black-box Benchmarks - of Throughput, Latency, Forwarding Rates and others have served our - industry for many years. Now, Network Function Virtualization (NFV) has - the goal to transform how internetwork functions are implemented, and - therefore has garnered much attention.</t> - - <t>This memo summarizes the progress of the Open Platform for NFV - (OPNFV) project on virtual switch performance characterization, - "VSWITCHPERF", through the Brahmaputra (second) release <xref - target="BrahRel"/>. This project intends to build on the current and - completed work of the Benchmarking Methodology Working Group in IETF, by - referencing existing literature. For example, currently the most often - referenced RFC is <xref target="RFC2544"/> (which depends on <xref - target="RFC1242"/>) and foundation of the benchmarking work in OPNFV is - common and strong.</t> - - <t>See - https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases - for more background, and the OPNFV website for general information: - https://www.opnfv.org/</t> - - <t>The authors note that OPNFV distinguishes itself from other open - source compute and networking projects through its emphasis on existing - "telco" services as opposed to cloud-computing. There are many ways in - which telco requirements have different emphasis on performance - dimensions when compared to cloud computing: support for and transfer of - isochronous media streams is one example.</t> - - <t>Note also that the move to NFV Infrastructure has resulted in many - new benchmarking initiatives across the industry. The authors are - currently doing their best to maintain alignment with many other - projects, and this Internet Draft is one part of the efforts. We - acknowledge the early work in <xref - target="I-D.huang-bmwg-virtual-network-performance"/>, and useful - discussion with the authors.</t> - </section> - - <section title="Scope"> - <t>The primary purpose and scope of the memo is to inform the industry - of work-in-progress that builds on the body of extensive BMWG literature - and experience, and describe the extensions needed for benchmarking - virtual switches. Inital feedback indicates that many of these - extensions may be applicable beyond the current scope (to hardware - switches in the NFV Infrastructure and to virtual routers, for example). - Additionally, this memo serves as a vehicle to include more detail and - commentary from BMWG and other Open Source communities, under BMWG's - chartered work to characterize the NFV Infrastructure (a virtual switch - is an important aspect of that infrastructure).</t> - </section> - - <section title="Benchmarking Considerations"> - <t>This section highlights some specific considerations (from <xref - target="I-D.ietf-bmwg-virtual-net"/>)related to Benchmarks for virtual - switches. The OPNFV project is sharing its present view on these areas, - as they develop their specifications in the Level Test Design (LTD) - document.</t> - - <section title="Comparison with Physical Network Functions"> - <t>To compare the performance of virtual designs and implementations - with their physical counterparts, identical benchmarks are needed. - BMWG has developed specifications for many network functions this memo - re-uses existing benchmarks through references, and expands them - during development of new methods. A key configuration aspect is the - number of parallel cores required to achieve comparable performance - with a given physical device, or whether some limit of scale was - reached before the cores could achieve the comparable level.</t> - - <t>It's unlikely that the virtual switch will be the only application - running on the SUT, so CPU utilization, Cache utilization, and Memory - footprint should also be recorded for the virtual implementations of - internetworking functions.</t> - </section> - - <section title="Continued Emphasis on Black-Box Benchmarks"> - <t>External observations remain essential as the basis for Benchmarks. - Internal observations with fixed specification and interpretation will - be provided in parallel to assist the development of operations - procedures when the technology is deployed.</t> - </section> - - <section title="New Configuration Parameters"> - <t>A key consideration when conducting any sort of benchmark is trying - to ensure the consistency and repeatability of test results. When - benchmarking the performance of a vSwitch there are many factors that - can affect the consistency of results, one key factor is matching the - various hardware and software details of the SUT. This section lists - some of the many new parameters which this project believes are - critical to report in order to achieve repeatability.</t> - - <t>Hardware details including:</t> - - <t><list style="symbols"> - <t>Platform details</t> - - <t>Processor details</t> - - <t>Memory information (type and size)</t> - - <t>Number of enabled cores</t> - - <t>Number of cores used for the test</t> - - <t>Number of physical NICs, as well as their details - (manufacturer, versions, type and the PCI slot they are plugged - into)</t> - - <t>NIC interrupt configuration</t> - - <t>BIOS version, release date and any configurations that were - modified</t> - - <t>CPU microcode level</t> - - <t>Memory DIMM configurations (quad rank performance may not be - the same as dual rank) in size, freq and slot locations</t> - - <t>PCI configuration parameters (payload size, early ack - option...)</t> - - <t>Power management at all levels (ACPI sleep states, processor - package, OS...)</t> - </list>Software details including:</t> - - <t><list style="symbols"> - <t>OS parameters and behavior (text vs graphical no one typing at - the console on one system)</t> - - <t>OS version (for host and VNF)</t> - - <t>Kernel version (for host and VNF)</t> - - <t>GRUB boot parameters (for host and VNF)</t> - - <t>Hypervisor details (Type and version)</t> - - <t>Selected vSwitch, version number or commit id used</t> - - <t>vSwitch launch command line if it has been parameterised</t> - - <t>Memory allocation to the vSwitch</t> - - <t>which NUMA node it is using, and how many memory channels</t> - - <t>DPDK or any other SW dependency version number or commit id - used</t> - - <t>Memory allocation to a VM - if it's from Hugpages/elsewhere</t> - - <t>VM storage type: snapshot/independent persistent/independent - non-persistent</t> - - <t>Number of VMs</t> - - <t>Number of Virtual NICs (vNICs), versions, type and driver</t> - - <t>Number of virtual CPUs and their core affinity on the host</t> - - <t>Number vNIC interrupt configuration</t> - - <t>Thread affinitization for the applications (including the - vSwitch itself) on the host</t> - - <t>Details of Resource isolation, such as CPUs designated for - Host/Kernel (isolcpu) and CPUs designated for specific processes - (taskset). - Test duration. - Number of flows.</t> - </list></t> - - <t>Test Traffic Information:<list style="symbols"> - <t>Traffic type - UDP, TCP, IMIX / Other</t> - - <t>Packet Sizes</t> - - <t>Deployment Scenario</t> - </list></t> - - <t/> - </section> - - <section title="Flow classification"> - <t>Virtual switches group packets into flows by processing and - matching particular packet or frame header information, or by matching - packets based on the input ports. Thus a flow can be thought of a - sequence of packets that have the same set of header field values or - have arrived on the same port. Performance results can vary based on - the parameters the vSwitch uses to match for a flow. The recommended - flow classification parameters for any vSwitch performance tests are: - the input port, the source IP address, the destination IP address and - the Ethernet protocol type field. It is essential to increase the flow - timeout time on a vSwitch before conducting any performance tests that - do not measure the flow setup time. Normally the first packet of a - particular stream will install the flow in the virtual switch which - adds an additional latency, subsequent packets of the same flow are - not subject to this latency if the flow is already installed on the - vSwitch.</t> - </section> - - <section title="Benchmarks using Baselines with Resource Isolation"> - <t>This outline describes measurement of baseline with isolated - resources at a high level, which is the intended approach at this - time.</t> - - <t><list style="numbers"> - <t>Baselines: <list style="symbols"> - <t>Optional: Benchmark platform forwarding capability without - a vswitch or VNF for at least 72 hours (serves as a means of - platform validation and a means to obtain the base performance - for the platform in terms of its maximum forwarding rate and - latency). <figure> - <preamble>Benchmark platform forwarding - capability</preamble> - - <artwork align="right"><![CDATA[ __ - +--------------------------------------------------+ | - | +------------------------------------------+ | | - | | | | | - | | Simple Forwarding App | | Host - | | | | | - | +------------------------------------------+ | | - | | NIC | | | - +---+------------------------------------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - - <postamble/> - </figure></t> - - <t>Benchmark VNF forwarding capability with direct - connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72 - hours (serves as a means of VNF validation and a means to - obtain the base performance for the VNF in terms of its - maximum forwarding rate and latency). The metrics gathered - from this test will serve as a key comparison point for - vSwitch bypass technologies performance and vSwitch - performance. <figure align="right"> - <preamble>Benchmark VNF forwarding capability</preamble> - - <artwork><![CDATA[ __ - +--------------------------------------------------+ | - | +------------------------------------------+ | | - | | | | | - | | VNF | | | - | | | | | - | +------------------------------------------+ | | - | | Passthrough/SR-IOV | | Host - | +------------------------------------------+ | | - | | NIC | | | - +---+------------------------------------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - - <postamble/> - </figure></t> - - <t>Benchmarking with isolated resources alone, with other - resources (both HW&SW) disabled Example, vSw and VM are - SUT</t> - - <t>Benchmarking with isolated resources alone, leaving some - resources unused</t> - - <t>Benchmark with isolated resources and all resources - occupied</t> - </list></t> - - <t>Next Steps<list style="symbols"> - <t>Limited sharing</t> - - <t>Production scenarios</t> - - <t>Stressful scenarios</t> - </list></t> - </list></t> - </section> - </section> - - <section title="VSWITCHPERF Specification Summary"> - <t>The overall specification in preparation is referred to as a Level - Test Design (LTD) document, which will contain a suite of performance - tests. The base performance tests in the LTD are based on the - pre-existing specifications developed by BMWG to test the performance of - physical switches. These specifications include:</t> - - <t><list style="symbols"> - <t><xref target="RFC2544"/> Benchmarking Methodology for Network - Interconnect Devices</t> - - <t><xref target="RFC2889"/> Benchmarking Methodology for LAN - Switching</t> - - <t><xref target="RFC6201"/> Device Reset Characterization</t> - - <t><xref target="RFC5481"/> Packet Delay Variation Applicability - Statement</t> - </list></t> - - <t>Some of the above/newer RFCs are being applied in benchmarking for - the first time, and represent a development challenge for test equipment - developers. Fortunately, many members of the testing system community - have engaged on the VSPERF project, including an open source test - system.</t> - - <t>In addition to this, the LTD also re-uses the terminology defined - by:</t> - - <t><list style="symbols"> - <t><xref target="RFC2285"/> Benchmarking Terminology for LAN - Switching Devices</t> - - <t><xref target="RFC5481"/> Packet Delay Variation Applicability - Statement</t> - </list></t> - - <t/> - - <t>Specifications to be included in future updates of the LTD - include:<list style="symbols"> - <t><xref target="RFC3918"/> Methodology for IP Multicast - Benchmarking</t> - - <t><xref target="RFC4737"/> Packet Reordering Metrics</t> - </list></t> - - <t>As one might expect, the most fundamental internetworking - characteristics of Throughput and Latency remain important when the - switch is virtualized, and these benchmarks figure prominently in the - specification.</t> - - <t>When considering characteristics important to "telco" network - functions, we must begin to consider additional performance metrics. In - this case, the project specifications have referenced metrics from the - IETF IP Performance Metrics (IPPM) literature. This means that the <xref - target="RFC2544"/> test of Latency is replaced by measurement of a - metric derived from IPPM's <xref target="RFC2679"/>, where a set of - statistical summaries will be provided (mean, max, min, etc.). Further - metrics planned to be benchmarked include packet delay variation as - defined by <xref target="RFC5481"/> , reordering, burst behaviour, DUT - availability, DUT capacity and packet loss in long term testing at - Throughput level, where some low-level of background loss may be present - and characterized.</t> - - <t>Tests have been (or will be) designed to collect the metrics - below:</t> - - <t><list style="symbols"> - <t>Throughput Tests to measure the maximum forwarding rate (in - frames per second or fps) and bit rate (in Mbps) for a constant load - (as defined by <xref target="RFC1242"/>) without traffic loss.</t> - - <t>Packet and Frame Delay Distribution Tests to measure average, min - and max packet and frame delay for constant loads.</t> - - <t>Packet Delay Tests to understand latency distribution for - different packet sizes and over an extended test run to uncover - outliers.</t> - - <t>Scalability Tests to understand how the virtual switch performs - as the number of flows, active ports, complexity of the forwarding - logic’s configuration… it has to deal with - increases.</t> - - <t>Stream Performance Tests (TCP, UDP) to measure bulk data transfer - performance, i.e. how fast systems can send and receive data through - the switch.</t> - - <t>Control Path and Datapath Coupling Tests, to understand how - closely coupled the datapath and the control path are as well as the - effect of this coupling on the performance of the DUT (example: - delay of the initial packet of a flow).</t> - - <t>CPU and Memory Consumption Tests to understand the virtual - switch’s footprint on the system, usually conducted as - auxiliary measurements with benchmarks above. They include: CPU - utilization, Cache utilization and Memory footprint.</t> - - <t>The so-called "Soak" tests, where the selected test is conducted - over a long period of time (with an ideal duration of 24 hours, and - at least 6 hours). The purpose of soak tests is to capture transient - changes in performance which may occur due to infrequent processes - or the low probability coincidence of two or more processes. The - performance must be evaluated periodically during continuous - testing, and this results in use of <xref target="RFC2889"/> Frame - Rate metrics instead of <xref target="RFC2544"/> Throughput (which - requires stopping traffic to allow time for all traffic to exit - internal queues).</t> - </list></t> - - <t>Future/planned test specs include:<list style="symbols"> - <t>Request/Response Performance Tests (TCP, UDP) which measure the - transaction rate through the switch.</t> - - <t>Noisy Neighbour Tests, to understand the effects of resource - sharing on the performance of a virtual switch.</t> - - <t>Tests derived from examination of ETSI NFV Draft GS IFA003 - requirements <xref target="IFA003"/> on characterization of - acceleration technologies applied to vswitches.</t> - </list>The flexibility of deployment of a virtual switch within a - network means that the BMWG IETF existing literature needs to be used to - characterize the performance of a switch in various deployment - scenarios. The deployment scenarios under consideration include:</t> - - <t><figure> - <preamble>Physical port to virtual switch to physical - port</preamble> - - <artwork><![CDATA[ __ - +--------------------------------------------------+ | - | +--------------------+ | | - | | | | | - | | v | | Host - | +--------------+ +--------------+ | | - | | phy port | vSwitch | phy port | | | - +---+--------------+------------+--------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure></t> - - <t><figure> - <preamble>Physical port to virtual switch to VNF to virtual switch - to physical port</preamble> - - <artwork><![CDATA[ __ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | ^ : | | - | | | | | Guest - | : v | | - | +---------------+ +---------------+ | | - | | logical port 0| | logical port 1| | | - +---+---------------+-----------+---------------+---+ __| - ^ : - | | - : v __ - +---+---------------+----------+---------------+---+ | - | | logical port 0| | logical port 1| | | - | +---------------+ +---------------+ | | - | ^ : | | - | | | | | Host - | : v | | - | +--------------+ +--------------+ | | - | | phy port | vSwitch | phy port | | | - +---+--------------+------------+--------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>Physical port to virtual switch to VNF to virtual switch - to VNF to virtual switch to physical port</preamble> - - <artwork><![CDATA[ __ - +----------------------+ +----------------------+ | - | Guest 1 | | Guest 2 | | - | +---------------+ | | +---------------+ | | - | | Application | | | | Application | | | - | +---------------+ | | +---------------+ | | - | ^ | | | ^ | | | - | | v | | | v | | Guests - | +---------------+ | | +---------------+ | | - | | logical ports | | | | logical ports | | | - | | 0 1 | | | | 0 1 | | | - +---+---------------+--+ +---+---------------+--+__| - ^ : ^ : - | | | | - : v : v _ - +---+---------------+---------+---------------+--+ | - | | 0 1 | | 3 4 | | | - | | logical ports | | logical ports | | | - | +---------------+ +---------------+ | | - | ^ | ^ | | | Host - | | |-----------------| v | | - | +--------------+ +--------------+ | | - | | phy ports | vSwitch | phy ports | | | - +---+--------------+----------+--------------+---+_| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>Physical port to virtual switch to VNF</preamble> - - <artwork><![CDATA[ __ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | ^ | | - | | | | Guest - | : | | - | +---------------+ | | - | | logical port 0| | | - +---+---------------+-------------------------------+ __| - ^ - | - : __ - +---+---------------+------------------------------+ | - | | logical port 0| | | - | +---------------+ | | - | ^ | | - | | | | Host - | : | | - | +--------------+ | | - | | phy port | vSwitch | | - +---+--------------+------------ -------------- ---+ __| - ^ - | - : - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>VNF to virtual switch to physical port</preamble> - - <artwork><![CDATA[ __ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | : | | - | | | | Guest - | v | | - | +---------------+ | | - | | logical port | | | - +-------------------------------+---------------+---+ __| - : - | - v __ - +------------------------------+---------------+---+ | - | | logical port | | | - | +---------------+ | | - | : | | - | | | | Host - | v | | - | +--------------+ | | - | vSwitch | phy port | | | - +-------------------------------+--------------+---+ __| - : - | - v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+]]></artwork> - </figure><figure> - <preamble>VNF to virtual switch to VNF</preamble> - - <artwork><![CDATA[ __ - +----------------------+ +----------------------+ | - | Guest 1 | | Guest 2 | | - | +---------------+ | | +---------------+ | | - | | Application | | | | Application | | | - | +---------------+ | | +---------------+ | | - | | | | ^ | | - | v | | | | | Guests - | +---------------+ | | +---------------+ | | - | | logical ports | | | | logical ports | | | - | | 0 | | | | 0 | | | - +---+---------------+--+ +---+---------------+--+__| - : ^ - | | - v : _ - +---+---------------+---------+---------------+--+ | - | | 1 | | 1 | | | - | | logical ports | | logical ports | | | - | +---------------+ +---------------+ | | - | | ^ | | Host - | L-----------------+ | | - | | | - | vSwitch | | - +------------------------------------------------+_|]]></artwork> - </figure></t> - - <t>A set of Deployment Scenario figures is available on the VSPERF Test - Methodology Wiki page <xref target="TestTopo"/>.</t> - </section> - - <section title="3x3 Matrix Coverage"> - <t>This section organizes the many existing test specifications into the - "3x3" matrix (introduced in <xref target="I-D.ietf-bmwg-virtual-net"/>). - Because the LTD specification ID names are quite long, this section is - organized into lists for each occupied cell of the matrix (not all are - occupied, also the matrix has grown to 3x4 to accommodate scale metrics - when displaying the coverage of many metrics/benchmarks). The current - version of the LTD specification is available <xref target="LTD"/>.</t> - - <t>The tests listed below assess the activation of paths in the data - plane, rather than the control plane.</t> - - <t>A complete list of tests with short summaries is available on the - VSPERF "LTD Test Spec Overview" Wiki page <xref target="LTDoverV"/>.</t> - - <section title="Speed of Activation"> - <t><list style="symbols"> - <t>Activation.RFC2889.AddressLearningRate</t> - - <t>PacketLatency.InitialPacketProcessingLatency</t> - </list></t> - </section> - - <section title="Accuracy of Activation section"> - <t><list style="symbols"> - <t>CPDP.Coupling.Flow.Addition</t> - </list></t> - </section> - - <section title="Reliability of Activation"> - <t><list style="symbols"> - <t>Throughput.RFC2544.SystemRecoveryTime</t> - - <t>Throughput.RFC2544.ResetTime</t> - </list></t> - </section> - - <section title="Scale of Activation"> - <t><list style="symbols"> - <t>Activation.RFC2889.AddressCachingCapacity</t> - </list></t> - </section> - - <section title="Speed of Operation"> - <t><list style="symbols"> - <t>Throughput.RFC2544.PacketLossRate</t> - - <t>CPU.RFC2544.0PacketLoss</t> - - <t>Throughput.RFC2544.PacketLossRateFrameModification</t> - - <t>Throughput.RFC2544.BackToBackFrames</t> - - <t>Throughput.RFC2889.MaxForwardingRate</t> - - <t>Throughput.RFC2889.ForwardPressure</t> - - <t>Throughput.RFC2889.BroadcastFrameForwarding</t> - </list></t> - </section> - - <section title="Accuracy of Operation"> - <t><list style="symbols"> - <t>Throughput.RFC2889.ErrorFramesFiltering</t> - - <t>Throughput.RFC2544.Profile</t> - </list></t> - </section> - - <section title="Reliability of Operation"> - <t><list style="symbols"> - <t>Throughput.RFC2889.Soak</t> - - <t>Throughput.RFC2889.SoakFrameModification</t> - - <t>PacketDelayVariation.RFC3393.Soak</t> - </list></t> - </section> - - <section title="Scalability of Operation"> - <t><list style="symbols"> - <t>Scalability.RFC2544.0PacketLoss</t> - - <t>MemoryBandwidth.RFC2544.0PacketLoss.Scalability</t> - </list></t> - </section> - - <section title="Summary"> - <t><figure> - <artwork><![CDATA[|------------------------------------------------------------------------| -| | | | | | -| | SPEED | ACCURACY | RELIABILITY | SCALE | -| | | | | | -|------------------------------------------------------------------------| -| | | | | | -| Activation | X | X | X | X | -| | | | | | -|------------------------------------------------------------------------| -| | | | | | -| Operation | X | X | X | X | -| | | | | | -|------------------------------------------------------------------------| -| | | | | | -| De-activation | | | | | -| | | | | | -|------------------------------------------------------------------------|]]></artwork> - </figure></t> - </section> - </section> - - <section title="Security Considerations"> - <t>Benchmarking activities as described in this memo are limited to - technology characterization of a Device Under Test/System Under Test - (DUT/SUT) using controlled stimuli in a laboratory environment, with - dedicated address space and the constraints specified in the sections - above.</t> - - <t>The benchmarking network topology will be an independent test setup - and MUST NOT be connected to devices that may forward the test traffic - into a production network, or misroute traffic to the test management - network.</t> - - <t>Further, benchmarking is performed on a "black-box" basis, relying - solely on measurements observable external to the DUT/SUT.</t> - - <t>Special capabilities SHOULD NOT exist in the DUT/SUT specifically for - benchmarking purposes. Any implications for network security arising - from the DUT/SUT SHOULD be identical in the lab and in production - networks.</t> - </section> - - <section anchor="IANA" title="IANA Considerations"> - <t>No IANA Action is requested at this time.</t> - </section> - - <section title="Acknowledgements"> - <t>The authors appreciate and acknowledge comments from Scott Bradner, - Marius Georgescu, Ramki Krishnan, Doug Montgomery, Martin Klozik, - Christian Trautman, and others for their reviews.</t> - </section> - </middle> - - <back> - <references title="Normative References"> - <?rfc ?> - - <?rfc include="reference.RFC.2119"?> - - <?rfc ?> - - <?rfc include="reference.RFC.2330"?> - - <?rfc include='reference.RFC.2544'?> - - <?rfc include="reference.RFC.2679"?> - - <?rfc include='reference.RFC.2680'?> - - <?rfc include='reference.RFC.3393'?> - - <?rfc include='reference.RFC.3432'?> - - <?rfc include='reference.RFC.2681'?> - - <?rfc include='reference.RFC.5905'?> - - <?rfc include='reference.RFC.4689'?> - - <?rfc include='reference.RFC.4737'?> - - <?rfc include='reference.RFC.5357'?> - - <?rfc include='reference.RFC.2889'?> - - <?rfc include='reference.RFC.3918'?> - - <?rfc include='reference.RFC.6201'?> - - <?rfc include='reference.RFC.2285'?> - - <reference anchor="NFV.PER001"> - <front> - <title>Network Function Virtualization: Performance and Portability - Best Practices</title> - - <author fullname="ETSI NFV" initials="" surname=""> - <organization/> - </author> - - <date month="June" year="2014"/> - </front> - - <seriesInfo name="Group Specification" - value="ETSI GS NFV-PER 001 V1.1.1 (2014-06)"/> - - <format type="PDF"/> - </reference> - </references> - - <references title="Informative References"> - <?rfc include='reference.RFC.1242'?> - - <?rfc include='reference.RFC.5481'?> - - <?rfc include='reference.RFC.6049'?> - - <?rfc include='reference.RFC.6248'?> - - <?rfc include='reference.RFC.6390'?> - - <?rfc include='reference.I-D.ietf-bmwg-virtual-net'?> - - <?rfc include='reference.I-D.huang-bmwg-virtual-network-performance'?> - - <reference anchor="TestTopo"> - <front> - <title>Test Topologies - https://wiki.opnfv.org/vsperf/test_methodology</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="LTDoverV"> - <front> - <title>LTD Test Spec Overview - https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="LTD"> - <front> - <title>LTD Test Specification - http://artifacts.opnfv.org/vswitchperf/docs/requirements/index.html</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="BrahRel"> - <front> - <title>Brahmaputra, Second OPNFV Release - https://www.opnfv.org/brahmaputra</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - - <reference anchor="IFA003"> - <front> - <title>https://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA003_Acceleration_-_vSwitch_Spec/</title> - - <author> - <organization/> - </author> - - <date/> - </front> - </reference> - </references> - </back> -</rfc> diff --git a/docs/testing/developer/requirements/vm2vm_alternative_benchmark.png b/docs/testing/developer/requirements/vm2vm_alternative_benchmark.png Binary files differdeleted file mode 100644 index d21334ba..00000000 --- a/docs/testing/developer/requirements/vm2vm_alternative_benchmark.png +++ /dev/null diff --git a/docs/testing/developer/requirements/vm2vm_benchmark.png b/docs/testing/developer/requirements/vm2vm_benchmark.png Binary files differdeleted file mode 100644 index 3a85e51f..00000000 --- a/docs/testing/developer/requirements/vm2vm_benchmark.png +++ /dev/null diff --git a/docs/testing/developer/requirements/vm2vm_hypervisor_benchmark.png b/docs/testing/developer/requirements/vm2vm_hypervisor_benchmark.png Binary files differdeleted file mode 100644 index b5b76e8a..00000000 --- a/docs/testing/developer/requirements/vm2vm_hypervisor_benchmark.png +++ /dev/null diff --git a/docs/testing/developer/requirements/vm2vm_virtual_interface_benchmark.png b/docs/testing/developer/requirements/vm2vm_virtual_interface_benchmark.png Binary files differdeleted file mode 100644 index 55294af6..00000000 --- a/docs/testing/developer/requirements/vm2vm_virtual_interface_benchmark.png +++ /dev/null diff --git a/docs/testing/developer/requirements/vswitchperf_ltd.rst b/docs/testing/developer/requirements/vswitchperf_ltd.rst deleted file mode 100644 index e1372520..00000000 --- a/docs/testing/developer/requirements/vswitchperf_ltd.rst +++ /dev/null @@ -1,1712 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Intel Corporation, AT&T and others. - -****************************** -VSPERF LEVEL TEST DESIGN (LTD) -****************************** - -.. 3.1 - -============ -Introduction -============ - -The intention of this Level Test Design (LTD) document is to specify the set of -tests to carry out in order to objectively measure the current characteristics -of a virtual switch in the Network Function Virtualization Infrastructure -(NFVI) as well as the test pass criteria. The detailed test cases will be -defined in details-of-LTD_, preceded by the doc-id-of-LTD_ and the scope-of-LTD_. - -This document is currently in draft form. - -.. 3.1.1 - - -.. _doc-id-of-LTD: - -Document identifier -=================== - -The document id will be used to uniquely -identify versions of the LTD. The format for the document id will be: -OPNFV\_vswitchperf\_LTD\_REL\_STATUS, where by the -status is one of: draft, reviewed, corrected or final. The document id -for this version of the LTD is: -OPNFV\_vswitchperf\_LTD\_Brahmaputra\_REVIEWED. - -.. 3.1.2 - -.. _scope-of-LTD: - -Scope -===== - -The main purpose of this project is to specify a suite of -performance tests in order to objectively measure the current packet -transfer characteristics of a virtual switch in the NFVI. The intent of -the project is to facilitate testing of any virtual switch. Thus, a -generic suite of tests shall be developed, with no hard dependencies to -a single implementation. In addition, the test case suite shall be -architecture independent. - -The test cases developed in this project shall not form part of a -separate test framework, all of these tests may be inserted into the -Continuous Integration Test Framework and/or the Platform Functionality -Test Framework - if a vSwitch becomes a standard component of an OPNFV -release. - -.. 3.1.3 - -References -========== - -* `RFC 1242 Benchmarking Terminology for Network Interconnection - Devices <http://www.ietf.org/rfc/rfc1242.txt>`__ -* `RFC 2544 Benchmarking Methodology for Network Interconnect - Devices <http://www.ietf.org/rfc/rfc2544.txt>`__ -* `RFC 2285 Benchmarking Terminology for LAN Switching - Devices <http://www.ietf.org/rfc/rfc2285.txt>`__ -* `RFC 2889 Benchmarking Methodology for LAN Switching - Devices <http://www.ietf.org/rfc/rfc2889.txt>`__ -* `RFC 3918 Methodology for IP Multicast - Benchmarking <http://www.ietf.org/rfc/rfc3918.txt>`__ -* `RFC 4737 Packet Reordering - Metrics <http://www.ietf.org/rfc/rfc4737.txt>`__ -* `RFC 5481 Packet Delay Variation Applicability - Statement <http://www.ietf.org/rfc/rfc5481.txt>`__ -* `RFC 6201 Device Reset - Characterization <http://tools.ietf.org/html/rfc6201>`__ - -.. 3.2 - -.. _details-of-LTD: - -================================ -Details of the Level Test Design -================================ - -This section describes the features to be tested (FeaturesToBeTested-of-LTD_), and -identifies the sets of test cases or scenarios (TestIdentification-of-LTD_). - -.. 3.2.1 - -.. _FeaturesToBeTested-of-LTD: - -Features to be tested -===================== - -Characterizing virtual switches (i.e. Device Under Test (DUT) in this document) -includes measuring the following performance metrics: - -- Throughput -- Packet delay -- Packet delay variation -- Packet loss -- Burst behaviour -- Packet re-ordering -- Packet correctness -- Availability and capacity of the DUT - -.. 3.2.2 - -.. _TestIdentification-of-LTD: - -Test identification -=================== - -.. 3.2.2.1 - -Throughput tests ----------------- - -The following tests aim to determine the maximum forwarding rate that -can be achieved with a virtual switch. The list is not exhaustive but -should indicate the type of tests that should be required. It is -expected that more will be added. - -.. 3.2.2.1.1 - -.. _PacketLossRatio: - -Test ID: LTD.Throughput.RFC2544.PacketLossRatio -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2544 X% packet loss ratio Throughput and Latency Test - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - This test determines the DUT's maximum forwarding rate with X% traffic - loss for a constant load (fixed length frames at a fixed interval time). - The default loss percentages to be tested are: - X = 0% - X = 10^-7% - - Note: Other values can be tested if required by the user. - - The selected frame sizes are those previously defined under - :ref:`default-test-parameters`. - The test can also be used to determine the average latency of the traffic. - - Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ - test methodology, the test duration will - include a number of trials; each trial should run for a minimum period - of 60 seconds. A binary search methodology must be applied for each - trial to obtain the final result. - - **Expected Result**: At the end of each trial, the presence or absence - of loss determines the modification of offered load for the next trial, - converging on a maximum rate, or - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X% - loss. - The Throughput load is re-used in related - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other - tests. - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of - the DUT for each frame size with X% packet loss. - - The average latency of the traffic flow when passing through the DUT - (if testing for latency, note that this average is different from the - test specified in Section 26.3 of - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__). - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - -.. 3.2.2.1.2 - -.. _PacketLossRatioFrameModification: - -Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2544 X% packet loss Throughput and Latency Test with - packet modification - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - This test determines the DUT's maximum forwarding rate with X% traffic - loss for a constant load (fixed length frames at a fixed interval time). - The default loss percentages to be tested are: - X = 0% - X = 10^-7% - - Note: Other values can be tested if required by the user. - - The selected frame sizes are those previously defined under - :ref:`default-test-parameters`. - The test can also be used to determine the average latency of the traffic. - - Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ - test methodology, the test duration will - include a number of trials; each trial should run for a minimum period - of 60 seconds. A binary search methodology must be applied for each - trial to obtain the final result. - - During this test, the DUT must perform the following operations on the - traffic flow: - - - Perform packet parsing on the DUT's ingress port. - - Perform any relevant address look-ups on the DUT's ingress ports. - - Modify the packet header before forwarding the packet to the DUT's - egress port. Packet modifications include: - - - Modifying the Ethernet source or destination MAC address. - - Modifying/adding a VLAN tag. (**Recommended**). - - Modifying/adding a MPLS tag. - - Modifying the source or destination ip address. - - Modifying the TOS/DSCP field. - - Modifying the source or destination ports for UDP/TCP/SCTP. - - Modifying the TTL. - - **Expected Result**: The Packet parsing/modifications require some - additional degree of processing resource, therefore the - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ - Throughput is expected to be somewhat lower than the Throughput level - measured without additional steps. The reduction is expected to be - greatest on tests with the smallest packet sizes (greatest header - processing rates). - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of - the DUT for each frame size with X% packet loss and packet - modification operations being performed by the DUT. - - The average latency of the traffic flow when passing through the DUT - (if testing for latency, note that this average is different from the - test specified in Section 26.3 of - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__). - - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__ - PDV form of delay variation on the traffic flow, - using the 99th percentile. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - -.. 3.2.2.1.3 - -Test ID: LTD.Throughput.RFC2544.Profile -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2544 Throughput and Latency Profile - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - This test reveals how throughput and latency degrades as the offered - rate varies in the region of the DUT's maximum forwarding rate as - determined by LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss). - For example it can be used to determine if the degradation of throughput - and latency as the offered rate increases is slow and graceful or sudden - and severe. - - The selected frame sizes are those previously defined under - :ref:`default-test-parameters`. - - The offered traffic rate is described as a percentage delta with respect - to the DUT's RFC 2544 Throughput as determined by - LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss case). A delta - of 0% is equivalent to an offered traffic rate equal to the RFC 2544 - Maximum Throughput; A delta of +50% indicates an offered rate half-way - between the Maximum RFC2544 Throughput and line-rate, whereas a delta of - -50% indicates an offered rate of half the RFC 2544 Maximum Throughput. - Therefore the range of the delta figure is natuarlly bounded at -100% - (zero offered traffic) and +100% (traffic offered at line rate). - - The following deltas to the maximum forwarding rate should be applied: - - - -50%, -10%, 0%, +10% & +50% - - **Expected Result**: For each packet size a profile should be produced - of how throughput and latency vary with offered rate. - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The forwarding rate in Frames Per Second (FPS) and Mbps of the DUT - for each delta to the maximum forwarding rate and for each frame - size. - - The average latency for each delta to the maximum forwarding rate and - for each frame size. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - Any failures experienced (for example if the vSwitch crashes, stops - processing packets, restarts or becomes unresponsive to commands) - when the offered load is above Maximum Throughput MUST be recorded - and reported with the results. - -.. 3.2.2.1.4 - -Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2544 System Recovery Time Test - - **Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio - - **Priority**: - - **Description**: - - The aim of this test is to determine the length of time it takes the DUT - to recover from an overload condition for a constant load (fixed length - frames at a fixed interval time). The selected frame sizes are those - previously defined under :ref:`default-test-parameters`, - traffic should be sent to the DUT under normal conditions. During the - duration of the test and while the traffic flows are passing though the - DUT, at least one situation leading to an overload condition for the DUT - should occur. The time from the end of the overload condition to when - the DUT returns to normal operations should be measured to determine - recovery time. Prior to overloading the DUT, one should record the - average latency for 10,000 packets forwarded through the DUT. - - The overload condition SHOULD be to transmit traffic at a very high - frame rate to the DUT (150% of the maximum 0% packet loss rate as - determined by LTD.Throughput.RFC2544.PacketLossRatio or line-rate - whichever is lower), for at least 60 seconds, then reduce the frame rate - to 75% of the maximum 0% packet loss rate. A number of time-stamps - should be recorded: - Record the time-stamp at which the frame rate was - reduced and record a second time-stamp at the time of the last frame - lost. The recovery time is the difference between the two timestamps. - - Record the average latency for 10,000 frames after the last frame loss - and continue to record average latency measurements for every 10,000 - frames, when latency returns to within 10% of pre-overload levels record - the time-stamp. - - **Expected Result**: - - **Metrics collected** - - The following are the metrics collected for this test: - - - The length of time it takes the DUT to recover from an overload - condition. - - The length of time it takes the DUT to recover the average latency to - pre-overload conditions. - - **Deployment scenario**: - - - Physical → virtual switch → physical. - -.. 3.2.2.1.5 - -.. _BackToBackFrames: - -Test ID: LTD.Throughput.RFC2544.BackToBackFrames -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC2544 Back To Back Frames Test - - **Prerequisite Test**: N - - **Priority**: - - **Description**: - - The aim of this test is to characterize the ability of the DUT to - process back-to-back frames. For each frame size previously defined - under :ref:`default-test-parameters`, a burst of traffic - is sent to the DUT with the minimum inter-frame gap between each frame. - If the number of received frames equals the number of frames that were - transmitted, the burst size should be increased and traffic is sent to - the DUT again. The value measured is the back-to-back value, that is the - maximum burst size the DUT can handle without any frame loss. Please note - a trial must run for a minimum of 2 seconds and should be repeated 50 - times (at a minimum). - - **Expected Result**: - - Tests of back-to-back frames with physical devices have produced - unstable results in some cases. All tests should be repeated in multiple - test sessions and results stability should be examined. - - **Metrics collected** - - The following are the metrics collected for this test: - - - The average back-to-back value across the trials, which is - the number of frames in the longest burst that the DUT will - handle without the loss of any frames. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - **Deployment scenario**: - - - Physical → virtual switch → physical. - -.. 3.2.2.1.6 - -Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2889 X% packet loss Max Forwarding Rate Soak Test - - **Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio - - **Priority**: - - **Description**: - - The aim of this test is to understand the Max Forwarding Rate stability - over an extended test duration in order to uncover any outliers. To allow - for an extended test duration, the test should ideally run for 24 hours - or, if this is not possible, for at least 6 hours. For this test, each frame - size must be sent at the highest Throughput rate with X% packet loss, as - determined in the prerequisite test. The default loss percentages to be - tested are: - X = 0% - X = 10^-7% - - Note: Other values can be tested if required by the user. - - **Expected Result**: - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - Max Forwarding Rate stability of the DUT. - - - This means reporting the number of packets lost per time interval - and reporting any time intervals with packet loss. The - `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__ - Forwarding Rate shall be measured in each interval. - An interval of 60s is suggested. - - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__ - PDV form of delay variation on the traffic flow, - using the 99th percentile. - -.. 3.2.2.1.7 - -Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2889 Max Forwarding Rate Soak Test with Frame Modification - - **Prerequisite Test**: - LTD.Throughput.RFC2544.PacketLossRatioFrameModification (0% Packet Loss) - - **Priority**: - - **Description**: - - The aim of this test is to understand the Max Forwarding Rate stability over an - extended test duration in order to uncover any outliers. To allow for an - extended test duration, the test should ideally run for 24 hours or, if - this is not possible, for at least 6 hour. For this test, each frame - size must be sent at the highest Throughput rate with 0% packet loss, as - determined in the prerequisite test. - - During this test, the DUT must perform the following operations on the - traffic flow: - - - Perform packet parsing on the DUT's ingress port. - - Perform any relevant address look-ups on the DUT's ingress ports. - - Modify the packet header before forwarding the packet to the DUT's - egress port. Packet modifications include: - - - Modifying the Ethernet source or destination MAC address. - - Modifying/adding a VLAN tag (**Recommended**). - - Modifying/adding a MPLS tag. - - Modifying the source or destination ip address. - - Modifying the TOS/DSCP field. - - Modifying the source or destination ports for UDP/TCP/SCTP. - - Modifying the TTL. - - **Expected Result**: - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - Max Forwarding Rate stability of the DUT. - - - This means reporting the number of packets lost per time interval - and reporting any time intervals with packet loss. The - `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__ - Forwarding Rate shall be measured in each interval. - An interval of 60s is suggested. - - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__ - PDV form of delay variation on the traffic flow, using the 99th - percentile. - -.. 3.2.2.1.8 - -Test ID: LTD.Throughput.RFC6201.ResetTime -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 6201 Reset Time Test - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - The aim of this test is to determine the length of time it takes the DUT - to recover from a reset. - - Two reset methods are defined - planned and unplanned. A planned reset - requires stopping and restarting the virtual switch by the usual - 'graceful' method defined by it's documentation. An unplanned reset - requires simulating a fatal internal fault in the virtual switch - for - example by using kill -SIGKILL on a Linux environment. - - Both reset methods SHOULD be exercised. - - For each frame size previously defined under :ref:`default-test-parameters`, - traffic should be sent to the DUT under - normal conditions. During the duration of the test and while the traffic - flows are passing through the DUT, the DUT should be reset and the Reset - time measured. The Reset time is the total time that a device is - determined to be out of operation and includes the time to perform the - reset and the time to recover from it (cf. `RFC6201 - <https://www.rfc-editor.org/rfc/rfc6201.txt>`__). - - `RFC6201 <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ defines two methods - to measure the Reset time: - - - Frame-Loss Method: which requires the monitoring of the number of - lost frames and calculates the Reset time based on the number of - frames lost and the offered rate according to the following - formula: - - .. code-block:: console - - Frames_lost (packets) - Reset_time = ------------------------------------- - Offered_rate (packets per second) - - - Timestamp Method: which measures the time from which the last frame - is forwarded from the DUT to the time the first frame is forwarded - after the reset. This involves time-stamping all transmitted frames - and recording the timestamp of the last frame that was received prior - to the reset and also measuring the timestamp of the first frame that - is received after the reset. The Reset time is the difference between - these two timestamps. - - According to `RFC6201 <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ the - choice of method depends on the test tool's capability; the Frame-Loss - method SHOULD be used if the test tool supports: - - * Counting the number of lost frames per stream. - * Transmitting test frame despite the physical link status. - - whereas the Timestamp method SHOULD be used if the test tool supports: - - * Timestamping each frame. - * Monitoring received frame's timestamp. - * Transmitting frames only if the physical link status is up. - - **Expected Result**: - - **Metrics collected** - - The following are the metrics collected for this test: - - * Average Reset Time over the number of trials performed. - - Results of this test should include the following information: - - * The reset method used. - * Throughput in Fps and Mbps. - * Average Frame Loss over the number of trials performed. - * Average Reset Time in milliseconds over the number of trials performed. - * Number of trials performed. - * Protocol: IPv4, IPv6, MPLS, etc. - * Frame Size in Octets - * Port Media: Ethernet, Gigabit Ethernet (GbE), etc. - * Port Speed: 10 Gbps, 40 Gbps etc. - * Interface Encapsulation: Ethernet, Ethernet VLAN, etc. - - **Deployment scenario**: - - * Physical → virtual switch → physical. - -.. 3.2.2.1.9 - -Test ID: LTD.Throughput.RFC2889.MaxForwardingRate -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC2889 Forwarding Rate Test - - **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio - - **Priority**: - - **Description**: - - This test measures the DUT's Max Forwarding Rate when the Offered Load - is varied between the throughput and the Maximum Offered Load for fixed - length frames at a fixed time interval. The selected frame sizes are - those previously defined under :ref:`default-test-parameters`. - The throughput is the maximum offered - load with 0% frame loss (measured by the prerequisite test), and the - Maximum Offered Load (as defined by - `RFC2285 <https://www.rfc-editor.org/rfc/rfc2285.txt>`__) is *"the highest - number of frames per second that an external source can transmit to a - DUT/SUT for forwarding to a specified output interface or interfaces"*. - - Traffic should be sent to the DUT at a particular rate (TX rate) - starting with TX rate equal to the throughput rate. The rate of - successfully received frames at the destination counted (in FPS). If the - RX rate is equal to the TX rate, the TX rate should be increased by a - fixed step size and the RX rate measured again until the Max Forwarding - Rate is found. - - The trial duration for each iteration should last for the period of time - needed for the system to reach steady state for the frame size being - tested. Under `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__ - (Sec. 5.6.3.1) test methodology, the test - duration should run for a minimum period of 30 seconds, regardless - whether the system reaches steady state before the minimum duration - ends. - - **Expected Result**: According to - `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__ The Max Forwarding - Rate is the highest forwarding rate of a DUT taken from an iterative set of - forwarding rate measurements. The iterative set of forwarding rate measurements - are made by setting the intended load transmitted from an external source and - measuring the offered load (i.e what the DUT is capable of forwarding). If the - Throughput == the Maximum Offered Load, it follows that Max Forwarding Rate is - equal to the Maximum Offered Load. - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The Max Forwarding Rate for the DUT for each packet size. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - **Deployment scenario**: - - - Physical → virtual switch → physical. Note: Full mesh tests with - multiple ingress and egress ports are a key aspect of RFC 2889 - benchmarks, and scenarios with both 2 and 4 ports should be tested. - In any case, the number of ports used must be reported. - -.. 3.2.2.1.10 - -Test ID: LTD.Throughput.RFC2889.ForwardPressure -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC2889 Forward Pressure Test - - **Prerequisite Test**: LTD.Throughput.RFC2889.MaxForwardingRate - - **Priority**: - - **Description**: - - The aim of this test is to determine if the DUT transmits frames with an - inter-frame gap that is less than 12 bytes. This test overloads the DUT - and measures the output for forward pressure. Traffic should be - transmitted to the DUT with an inter-frame gap of 11 bytes, this will - overload the DUT by 1 byte per frame. The forwarding rate of the DUT - should be measured. - - **Expected Result**: The forwarding rate should not exceed the maximum - forwarding rate of the DUT collected by - LTD.Throughput.RFC2889.MaxForwardingRate. - - **Metrics collected** - - The following are the metrics collected for this test: - - - Forwarding rate of the DUT in FPS or Mbps. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - **Deployment scenario**: - - - Physical → virtual switch → physical. - -.. 3.2.2.1.11 - -Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC2889 Error Frames Filtering Test - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - The aim of this test is to determine whether the DUT will propagate any - erroneous frames it receives or whether it is capable of filtering out - the erroneous frames. Traffic should be sent with erroneous frames - included within the flow at random intervals. Illegal frames that must - be tested include: - Oversize Frames. - Undersize Frames. - CRC Errored - Frames. - Dribble Bit Errored Frames - Alignment Errored Frames - - The traffic flow exiting the DUT should be recorded and checked to - determine if the erroneous frames where passed through the DUT. - - **Expected Result**: Broken frames are not passed! - - **Metrics collected** - - No Metrics are collected in this test, instead it determines: - - - Whether the DUT will propagate erroneous frames. - - Or whether the DUT will correctly filter out any erroneous frames - from traffic flow with out removing correct frames. - - **Deployment scenario**: - - - Physical → virtual switch → physical. - -.. 3.2.2.1.12 - -Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC2889 Broadcast Frame Forwarding Test - - **Prerequisite Test**: N - - **Priority**: - - **Description**: - - The aim of this test is to determine the maximum forwarding rate of the - DUT when forwarding broadcast traffic. For each frame previously defined - under :ref:`default-test-parameters`, the traffic should - be set up as broadcast traffic. The traffic throughput of the DUT should - be measured. - - The test should be conducted with at least 4 physical ports on the DUT. - The number of ports used MUST be recorded. - - As broadcast involves forwarding a single incoming packet to several - destinations, the latency of a single packet is defined as the average - of the latencies for each of the broadcast destinations. - - The incoming packet is transmitted on each of the other physical ports, - it is not transmitted on the port on which it was received. The test MAY - be conducted using different broadcasting ports to uncover any - performance differences. - - **Expected Result**: - - **Metrics collected**: - - The following are the metrics collected for this test: - - - The forwarding rate of the DUT when forwarding broadcast traffic. - - The minimum, average & maximum packets latencies observed. - - **Deployment scenario**: - - - Physical → virtual switch 3x physical. In the Broadcast rate testing, - four test ports are required. One of the ports is connected to the test - device, so it can send broadcast frames and listen for miss-routed frames. - -.. 3.2.2.1.13 - -Test ID: LTD.Throughput.RFC2544.WorstN-BestN -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: Modified RFC 2544 X% packet loss ratio Throughput and Latency Test - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - This test determines the DUT's maximum forwarding rate with X% traffic - loss for a constant load (fixed length frames at a fixed interval time). - The default loss percentages to be tested are: X = 0%, X = 10^-7% - - Modified RFC 2544 throughput benchmarking methodology aims to quantify - the throughput measurement variations observed during standard RFC 2544 - benchmarking measurements of virtual switches and VNFs. The RFC2544 - binary search algorithm is modified to use more samples per test trial - to drive the binary search and yield statistically more meaningful - results. This keeps the heart of the RFC2544 methodology, still relying - on the binary search of throughput at specified loss tolerance, while - providing more useful information about the range of results seen in - testing. Instead of using a single traffic trial per iteration step, - each traffic trial is repeated N times and the success/failure of the - iteration step is based on these N traffic trials. Two types of revised - tests are defined - *Worst-of-N* and *Best-of-N*. - - **Worst-of-N** - - *Worst-of-N* indicates the lowest expected maximum throughput for ( - packet size, loss tolerance) when repeating the test. - - 1. Repeat the same test run N times at a set packet rate, record each - result. - 2. Take the WORST result (highest packet loss) out of N result samples, - called the Worst-of-N sample. - 3. If Worst-of-N sample has loss less than the set loss tolerance, then - the step is successful - increase the test traffic rate. - 4. If Worst-of-N sample has loss greater than the set loss tolerance - then the step failed - decrease the test traffic rate. - 5. Go to step 1. - - **Best-of-N** - - *Best-of-N* indicates the highest expected maximum throughput for ( - packet size, loss tolerance) when repeating the test. - - 1. Repeat the same traffic run N times at a set packet rate, record - each result. - 2. Take the BEST result (least packet loss) out of N result samples, - called the Best-of-N sample. - 3. If Best-of-N sample has loss less than the set loss tolerance, then - the step is successful - increase the test traffic rate. - 4. If Best-of-N sample has loss greater than the set loss tolerance, - then the step failed - decrease the test traffic rate. - 5. Go to step 1. - - Performing both Worst-of-N and Best-of-N benchmark tests yields lower - and upper bounds of expected maximum throughput under the operating - conditions, giving a very good indication to the user of the - deterministic performance range for the tested setup. - - **Expected Result**: At the end of each trial series, the presence or - absence of loss determines the modification of offered load for the - next trial series, converging on a maximum rate, or - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput - with X% loss. - The Throughput load is re-used in related - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other - tests. - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of - the DUT for each frame size with X% packet loss. - - The average latency of the traffic flow when passing through the DUT - (if testing for latency, note that this average is different from the - test specified in Section 26.3 of - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__). - - Following may also be collected as part of this test, to determine - the vSwitch's performance footprint on the system: - - - CPU core utilization. - - CPU cache utilization. - - Memory footprint. - - System bus (QPI, PCI, ...) utilization. - - CPU cycles consumed per packet. - -.. 3.2.2.1.14 - -Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: <tech> Overlay Network RFC 2544 X% packet loss ratio Throughput and Latency Test - - - NOTE: Throughout this test, four interchangeable overlay technologies are covered by the - same test description. They are: VXLAN, GRE, NVGRE and GENEVE. - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - This test evaluates standard switch performance benchmarks for the scenario where an - Overlay Network is deployed for all paths through the vSwitch. Overlay Technologies covered - (replacing <tech> in the test name) include: - - - VXLAN - - GRE - - NVGRE - - GENEVE - - Performance will be assessed for each of the following overlay network functions: - - - Encapsulation only - - De-encapsulation only - - Both Encapsulation and De-encapsulation - - For each native packet, the DUT must perform the following operations: - - - Examine the packet and classify its correct overlay net (tunnel) assignment - - Encapsulate the packet - - Switch the packet to the correct port - - For each encapsulated packet, the DUT must perform the following operations: - - - Examine the packet and classify its correct native network assignment - - De-encapsulate the packet, if required - - Switch the packet to the correct port - - The selected frame sizes are those previously defined under - :ref:`default-test-parameters`. - - Thus, each test comprises an overlay technology, a network function, - and a packet size *with* overlay network overhead included - (but see also the discussion at - https://etherpad.opnfv.org/p/vSwitchTestsDrafts ). - - The test can also be used to determine the average latency of the traffic. - - Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ - test methodology, the test duration will - include a number of trials; each trial should run for a minimum period - of 60 seconds. A binary search methodology must be applied for each - trial to obtain the final result for Throughput. - - **Expected Result**: At the end of each trial, the presence or absence - of loss determines the modification of offered load for the next trial, - converging on a maximum rate, or - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X% - loss (where the value of X is typically equal to zero). - The Throughput load is re-used in related - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other - tests. - - **Metrics Collected**: - The following are the metrics collected for this test: - - - The maximum Throughput in Frames Per Second (FPS) and Mbps of - the DUT for each frame size with X% packet loss. - - The average latency of the traffic flow when passing through the DUT - and VNFs (if testing for latency, note that this average is different from the - test specified in Section 26.3 of - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__). - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - -.. 3.2.3.1.15 - -Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2544 X% packet loss ratio match action Throughput and Latency Test - - **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio - - **Priority**: - - **Description**: - - The aim of this test is to determine the cost of carrying out match - action(s) on the DUT’s RFC2544 Throughput with X% traffic loss for - a constant load (fixed length frames at a fixed interval time). - - Each test case requires: - - * selection of a specific match action(s), - * specifying a percentage of total traffic that is elligible - for the match action, - * determination of the specific test configuration (number - of flows, number of test ports, presence of an external - controller, etc.), and - * measurement of the RFC 2544 Throughput level with X% packet - loss: Traffic shall be bi-directional and symmetric. - - Note: It would be ideal to verify that all match action-elligible - traffic was forwarded to the correct port, and if forwarded to - an unintended port it should be considered lost. - - A match action is an action that is typically carried on a frame - or packet that matches a set of flow classification parameters - (typically frame/packet header fields). A match action may or may - not modify a packet/frame. Match actions include [1]: - - * output : outputs a packet to a particular port. - * normal: Subjects the packet to traditional L2/L3 processing - (MAC learning). - * flood: Outputs the packet on all switch physical ports - other than the port on which it was received and any ports - on which flooding is disabled. - * all: Outputs the packet on all switch physical ports other - than the port on which it was received. - * local: Outputs the packet on the ``local port``, which - corresponds to the network device that has the same name as - the bridge. - * in_port: Outputs the packet on the port from which it was - received. - * Controller: Sends the packet and its metadata to the - OpenFlow controller as a ``packet in`` message. - * enqueue: Enqueues the packet on the specified queue - within port. - * drop: discard the packet. - - Modifications include [1]: - - * mod vlan: covered by LTD.Throughput.RFC2544.PacketLossRatioFrameModification - * mod_dl_src: Sets the source Ethernet address. - * mod_dl_dst: Sets the destination Ethernet address. - * mod_nw_src: Sets the IPv4 source address. - * mod_nw_dst: Sets the IPv4 destination address. - * mod_tp_src: Sets the TCP or UDP or SCTP source port. - * mod_tp_dst: Sets the TCP or UDP or SCTP destination port. - * mod_nw_tos: Sets the DSCP bits in the IPv4 ToS/DSCP or - IPv6 traffic class field. - * mod_nw_ecn: Sets the ECN bits in the appropriate IPv4 or - IPv6 field. - * mod_nw_ttl: Sets the IPv4 TTL or IPv6 hop limit field. - - Note: This comprehensive list requires extensive traffic generator - capabilities. - - The match action(s) that were applied as part of the test should be - reported in the final test report. - - During this test, the DUT must perform the following operations on - the traffic flow: - - * Perform packet parsing on the DUT’s ingress port. - * Perform any relevant address look-ups on the DUT’s ingress - ports. - * Carry out one or more of the match actions specified above. - - The default loss percentages to be tested are: - X = 0% - X = 10^-7% - Other values can be tested if required by the user. The selected - frame sizes are those previously defined under - :ref:`default-test-parameters`. - - The test can also be used to determine the average latency of the - traffic when a match action is applied to packets in a flow. Under - the RFC2544 test methodology, the test duration will include a - number of trials; each trial should run for a minimum period of 60 - seconds. A binary search methodology must be applied for each - trial to obtain the final result. - - **Expected Result:** - - At the end of each trial, the presence or absence of loss - determines the modification of offered load for the next trial, - converging on a maximum rate, or RFC2544Throughput with X% loss. - The Throughput load is re-used in related RFC2544 tests and other - tests. - - **Metrics Collected:** - - The following are the metrics collected for this test: - - * The RFC 2544 Throughput in Frames Per Second (FPS) and Mbps - of the DUT for each frame size with X% packet loss. - * The average latency of the traffic flow when passing through - the DUT (if testing for latency, note that this average is - different from the test specified in Section 26.3 ofRFC2544). - * CPU and memory utilization may also be collected as part of - this test, to determine the vSwitch’s performance footprint - on the system. - - The metrics collected can be compared to that of the prerequisite - test to determine the cost of the match action(s) in the pipeline. - - **Deployment scenario**: - - - Physical → virtual switch → physical (and others are possible) - - [1] ovs-ofctl - administer OpenFlow switches - [http://openvswitch.org/support/dist-docs/ovs-ofctl.8.txt ] - - -.. 3.2.2.2 - -Packet Latency tests --------------------- - -These tests will measure the store and forward latency as well as the packet -delay variation for various packet types through the virtual switch. The -following list is not exhaustive but should indicate the type of tests -that should be required. It is expected that more will be added. - -.. 3.2.2.2.1 - -Test ID: LTD.PacketLatency.InitialPacketProcessingLatency -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: Initial Packet Processing Latency - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - In some virtual switch architectures, the first packets of a flow will - take the system longer to process than subsequent packets in the flow. - This test determines the latency for these packets. The test will - measure the latency of the packets as they are processed by the - flow-setup-path of the DUT. There are two methods for this test, a - recommended method and a nalternative method that can be used if it is - possible to disable the fastpath of the virtual switch. - - Recommended method: This test will send 64,000 packets to the DUT, each - belonging to a different flow. Average packet latency will be determined - over the 64,000 packets. - - Alternative method: This test will send a single packet to the DUT after - a fixed interval of time. The time interval will be equivalent to the - amount of time it takes for a flow to time out in the virtual switch - plus 10%. Average packet latency will be determined over 1,000,000 - packets. - - This test is intended only for non-learning virtual switches; For learning - virtual switches use RFC2889. - - For this test, only unidirectional traffic is required. - - **Expected Result**: The average latency for the initial packet of all - flows should be greater than the latency of subsequent traffic. - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - Average latency of the initial packets of all flows that are - processed by the DUT. - - **Deployment scenario**: - - - Physical → Virtual Switch → Physical. - -.. 3.2.2.2.2 - -Test ID: LTD.PacketDelayVariation.RFC3393.Soak -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: Packet Delay Variation Soak Test - - **Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss) - - **Priority**: - - **Description**: - - The aim of this test is to understand the distribution of packet delay - variation for different frame sizes over an extended test duration and - to determine if there are any outliers. To allow for an extended test - duration, the test should ideally run for 24 hours or, if this is not - possible, for at least 6 hour. For this test, each frame size must be - sent at the highest possible throughput with 0% packet loss, as - determined in the prerequisite test. - - **Expected Result**: - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The packet delay variation value for traffic passing through the DUT. - - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__ - PDV form of delay variation on the traffic flow, - using the 99th percentile, for each 60s interval during the test. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - -.. 3.2.2.3 - -Scalability tests ------------------ - -The general aim of these tests is to understand the impact of large flow -table size and flow lookups on throughput. The following list is not -exhaustive but should indicate the type of tests that should be required. -It is expected that more will be added. - -.. 3.2.2.3.1 - -.. _Scalability0PacketLoss: - -Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2544 0% loss Flow Scalability throughput test - - **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio, IF the - delta Throughput between the single-flow RFC2544 test and this test with - a variable number of flows is desired. - - **Priority**: - - **Description**: - - The aim of this test is to measure how throughput changes as the number - of flows in the DUT increases. The test will measure the throughput - through the fastpath, as such the flows need to be installed on the DUT - before passing traffic. - - For each frame size previously defined under :ref:`default-test-parameters` - and for each of the following number of flows: - - - 1,000 - - 2,000 - - 4,000 - - 8,000 - - 16,000 - - 32,000 - - 64,000 - - Max supported number of flows. - - This test will be conducted under two conditions following the - establishment of all flows as required by RFC 2544, regarding the flow - expiration time-out: - - 1) The time-out never expires during each trial. - - 2) The time-out expires for all flows periodically. This would require a - short time-out compared with flow re-appearance for a small number of - flows, and may not be possible for all flow conditions. - - The maximum 0% packet loss Throughput should be determined in a manner - identical to LTD.Throughput.RFC2544.PacketLossRatio. - - **Expected Result**: - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The maximum number of frames per second that can be forwarded at the - specified number of flows and the specified frame size, with zero - packet loss. - -.. 3.2.2.3.2 - -Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2544 0% loss Memory Bandwidth Scalability test - - **Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio, IF the - delta Throughput between an undisturbed RFC2544 test and this test with - the Throughput affected by cache and memory bandwidth contention is desired. - - **Priority**: - - **Description**: - - The aim of this test is to understand how the DUT's performance is - affected by cache sharing and memory bandwidth between processes. - - During the test all cores not used by the vSwitch should be running a - memory intensive application. This application should read and write - random data to random addresses in unused physical memory. The random - nature of the data and addresses is intended to consume cache, exercise - main memory access (as opposed to cache) and exercise all memory buses - equally. Furthermore: - - - the ratio of reads to writes should be recorded. A ratio of 1:1 - SHOULD be used. - - the reads and writes MUST be of cache-line size and be cache-line aligned. - - in NUMA architectures memory access SHOULD be local to the core's node. - Whether only local memory or a mix of local and remote memory is used - MUST be recorded. - - the memory bandwidth (reads plus writes) used per-core MUST be recorded; - the test MUST be run with a per-core memory bandwidth equal to half the - maximum system memory bandwidth divided by the number of cores. The test - MAY be run with other values for the per-core memory bandwidth. - - the test MAY also be run with the memory intensive application running - on all cores. - - Under these conditions the DUT's 0% packet loss throughput is determined - as per LTD.Throughput.RFC2544.PacketLossRatio. - - **Expected Result**: - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The DUT's 0% packet loss throughput in the presence of cache sharing and - memory bandwidth between processes. - -.. 3.2.2.3.3 - -Test ID: LTD.Scalability.VNF.RFC2544.PacketLossRatio -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: VNF Scalability RFC 2544 X% packet loss ratio Throughput and - Latency Test - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - This test determines the DUT's throughput rate with X% traffic loss for - a constant load (fixed length frames at a fixed interval time) when the - number of VNFs on the DUT increases. The default loss percentages - to be tested are: - X = 0% - X = 10^-7% . The minimum number of - VNFs to be tested are 3. - - Flow classification should be conducted with L2, L3 and L4 matching - to understand the matching and scaling capability of the vSwitch. The - matching fields which were used as part of the test should be reported - as part of the benchmark report. - - The vSwitch is responsible for forwarding frames between the VNFs - - The SUT (vSwitch and VNF daisy chain) operation should be validated - before running the test. This may be completed by running a burst or - continuous stream of traffic through the SUT to ensure proper operation - before a test. - - **Note**: The traffic rate used to validate SUT operation should be low - enough not to stress the SUT. - - **Note**: Other values can be tested if required by the user. - - **Note**: The same VNF should be used in the "daisy chain" formation. - Each addition of a VNF should be conducted in a new test setup (The DUT - is brought down, then the DUT is brought up again). An atlernative approach - would be to continue to add VNFs without bringing down the DUT. The - approach used needs to be documented as part of the test report. - - The selected frame sizes are those previously defined under - :ref:`default-test-parameters`. - The test can also be used to determine the average latency of the traffic. - - Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ - test methodology, the test duration will - include a number of trials; each trial should run for a minimum period - of 60 seconds. A binary search methodology must be applied for each - trial to obtain the final result for Throughput. - - **Expected Result**: At the end of each trial, the presence or absence - of loss determines the modification of offered load for the next trial, - converging on a maximum rate, or - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X% - loss. - The Throughput load is re-used in related - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other - tests. - - If the test VNFs are rather light-weight in terms of processing, the test - provides a view of multiple passes through the vswitch on logical - interfaces. In other words, the test produces an optimistic count of - daisy-chained VNFs, but the cumulative effect of traffic on the vSwitch is - "real" (assuming that the vSwitch has some dedicated resources, and the - effects on shared resources is understood). - - - **Metrics Collected**: - The following are the metrics collected for this test: - - - The maximum Throughput in Frames Per Second (FPS) and Mbps of - the DUT for each frame size with X% packet loss. - - The average latency of the traffic flow when passing through the DUT - and VNFs (if testing for latency, note that this average is different from the - test specified in Section 26.3 of - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__). - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - -.. 3.2.2.3.4 - -Test ID: LTD.Scalability.VNF.RFC2544.PacketLossProfile -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: VNF Scalability RFC 2544 Throughput and Latency Profile - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - This test reveals how throughput and latency degrades as the number - of VNFs increases and offered rate varies in the region of the DUT's - maximum forwarding rate as determined by - LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss). - For example it can be used to determine if the degradation of throughput - and latency as the number of VNFs and offered rate increases is slow - and graceful, or sudden and severe. The minimum number of VNFs to - be tested is 3. - - The selected frame sizes are those previously defined under - :ref:`default-test-parameters`. - - The offered traffic rate is described as a percentage delta with respect - to the DUT's RFC 2544 Throughput as determined by - LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss case). A delta - of 0% is equivalent to an offered traffic rate equal to the RFC 2544 - Throughput; A delta of +50% indicates an offered rate half-way - between the Throughput and line-rate, whereas a delta of - -50% indicates an offered rate of half the maximum rate. Therefore the - range of the delta figure is natuarlly bounded at -100% (zero offered - traffic) and +100% (traffic offered at line rate). - - The following deltas to the maximum forwarding rate should be applied: - - - -50%, -10%, 0%, +10% & +50% - - **Note**: Other values can be tested if required by the user. - - **Note**: The same VNF should be used in the "daisy chain" formation. - Each addition of a VNF should be conducted in a new test setup (The DUT - is brought down, then the DUT is brought up again). An atlernative approach - would be to continue to add VNFs without bringing down the DUT. The - approach used needs to be documented as part of the test report. - - Flow classification should be conducted with L2, L3 and L4 matching - to understand the matching and scaling capability of the vSwitch. The - matching fields which were used as part of the test should be reported - as part of the benchmark report. - - The SUT (vSwitch and VNF daisy chain) operation should be validated - before running the test. This may be completed by running a burst or - continuous stream of traffic through the SUT to ensure proper operation - before a test. - - **Note**: the traffic rate used to validate SUT operation should be low - enough not to stress the SUT - - **Expected Result**: For each packet size a profile should be produced - of how throughput and latency vary with offered rate. - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The forwarding rate in Frames Per Second (FPS) and Mbps of the DUT - for each delta to the maximum forwarding rate and for each frame - size. - - The average latency for each delta to the maximum forwarding rate and - for each frame size. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - Any failures experienced (for example if the vSwitch crashes, stops - processing packets, restarts or becomes unresponsive to commands) - when the offered load is above Maximum Throughput MUST be recorded - and reported with the results. - -.. 3.2.2.4 - -Activation tests ----------------- - -The general aim of these tests is to understand the capacity of the -and speed with which the vswitch can accommodate new flows. - -.. 3.2.2.4.1 - -Test ID: LTD.Activation.RFC2889.AddressCachingCapacity -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC2889 Address Caching Capacity Test - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - Please note this test is only applicable to virtual switches that are capable of - MAC learning. The aim of this test is to determine the address caching - capacity of the DUT for a constant load (fixed length frames at a fixed - interval time). The selected frame sizes are those previously defined - under :ref:`default-test-parameters`. - - In order to run this test the aging time, that is the maximum time the - DUT will keep a learned address in its flow table, and a set of initial - addresses, whose value should be >= 1 and <= the max number supported by - the implementation must be known. Please note that if the aging time is - configurable it must be longer than the time necessary to produce frames - from the external source at the specified rate. If the aging time is - fixed the frame rate must be brought down to a value that the external - source can produce in a time that is less than the aging time. - - Learning Frames should be sent from an external source to the DUT to - install a number of flows. The Learning Frames must have a fixed - destination address and must vary the source address of the frames. The - DUT should install flows in its flow table based on the varying source - addresses. Frames should then be transmitted from an external source at - a suitable frame rate to see if the DUT has properly learned all of the - addresses. If there is no frame loss and no flooding, the number of - addresses sent to the DUT should be increased and the test is repeated - until the max number of cached addresses supported by the DUT - determined. - - **Expected Result**: - - **Metrics collected**: - - The following are the metrics collected for this test: - - - Number of cached addresses supported by the DUT. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - **Deployment scenario**: - - - Physical → virtual switch → 2 x physical (one receiving, one listening). - -.. 3.2.2.4.2 - -Test ID: LTD.Activation.RFC2889.AddressLearningRate -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC2889 Address Learning Rate Test - - **Prerequisite Test**: LTD.Memory.RFC2889.AddressCachingCapacity - - **Priority**: - - **Description**: - - Please note this test is only applicable to virtual switches that are capable of - MAC learning. The aim of this test is to determine the rate of address - learning of the DUT for a constant load (fixed length frames at a fixed - interval time). The selected frame sizes are those previously defined - under :ref:`default-test-parameters`, traffic should be - sent with each IPv4/IPv6 address incremented by one. The rate at which - the DUT learns a new address should be measured. The maximum caching - capacity from LTD.Memory.RFC2889.AddressCachingCapacity should be taken - into consideration as the maximum number of addresses for which the - learning rate can be obtained. - - **Expected Result**: It may be worthwhile to report the behaviour when - operating beyond address capacity - some DUTs may be more friendly to - new addresses than others. - - **Metrics collected**: - - The following are the metrics collected for this test: - - - The address learning rate of the DUT. - - **Deployment scenario**: - - - Physical → virtual switch → 2 x physical (one receiving, one listening). - -.. 3.2.2.5 - -Coupling between control path and datapath Tests ------------------------------------------------- - -The following tests aim to determine how tightly coupled the datapath -and the control path are within a virtual switch. The following list -is not exhaustive but should indicate the type of tests that should be -required. It is expected that more will be added. - -.. 3.2.2.5.1 - -Test ID: LTD.CPDPCouplingFlowAddition -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: Control Path and Datapath Coupling - - **Prerequisite Test**: - - **Priority**: - - **Description**: - - The aim of this test is to understand how exercising the DUT's control - path affects datapath performance. - - Initially a certain number of flow table entries are installed in the - vSwitch. Then over the duration of an RFC2544 throughput test - flow-entries are added and removed at the rates specified below. No - traffic is 'hitting' these flow-entries, they are simply added and - removed. - - The test MUST be repeated with the following initial number of - flow-entries installed: - < 10 - 1000 - 100,000 - 10,000,000 (or the - maximum supported number of flow-entries) - - The test MUST be repeated with the following rates of flow-entry - addition and deletion per second: - 0 - 1 (i.e. 1 addition plus 1 - deletion) - 100 - 10,000 - - **Expected Result**: - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of - the DUT. - - The average latency of the traffic flow when passing through the DUT - (if testing for latency, note that this average is different from the - test specified in Section 26.3 of - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__). - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - **Deployment scenario**: - - - Physical → virtual switch → physical. - -.. 3.2.2.6 - -CPU and memory consumption --------------------------- - -The following tests will profile a virtual switch's CPU and memory -utilization under various loads and circumstances. The following -list is not exhaustive but should indicate the type of tests that -should be required. It is expected that more will be added. - -.. 3.2.2.6.1 - -.. _CPU0PacketLoss: - -Test ID: LTD.Stress.RFC2544.0PacketLoss -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2544 0% Loss CPU OR Memory Stress Test - - **Prerequisite Test**: - - **Priority**: - - **Description**: - - The aim of this test is to understand the overall performance of the - system when a CPU or Memory intensive application is run on the same DUT as - the Virtual Switch. For each frame size, an - LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss) test should be - performed. Throughout the entire test a CPU or Memory intensive application - should be run on all cores on the system not in use by the Virtual Switch. - For NUMA system only cores on the same NUMA node are loaded. - - It is recommended that stress-ng be used for loading the non-Virtual - Switch cores but any stress tool MAY be used. - - **Expected Result**: - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - Memory and CPU utilization of the cores running the Virtual Switch. - - The number of identity of the cores allocated to the Virtual Switch. - - The configuration of the stress tool (for example the command line - parameters used to start it.) - - **Note:** Stress in the test ID can be replaced with the name of the - component being stressed, when reporting the results: - LTD.CPU.RFC2544.0PacketLoss or LTD.Memory.RFC2544.0PacketLoss - -.. 3.2.2.7 - -Summary List of Tests ---------------------- - -1. Throughput tests - - - Test ID: LTD.Throughput.RFC2544.PacketLossRatio - - Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification - - Test ID: LTD.Throughput.RFC2544.Profile - - Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime - - Test ID: LTD.Throughput.RFC2544.BackToBackFrames - - Test ID: LTD.Throughput.RFC2889.Soak - - Test ID: LTD.Throughput.RFC2889.SoakFrameModification - - Test ID: LTD.Throughput.RFC6201.ResetTime - - Test ID: LTD.Throughput.RFC2889.MaxForwardingRate - - Test ID: LTD.Throughput.RFC2889.ForwardPressure - - Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering - - Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding - - Test ID: LTD.Throughput.RFC2544.WorstN-BestN - - Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio - -2. Packet Latency tests - - - Test ID: LTD.PacketLatency.InitialPacketProcessingLatency - - Test ID: LTD.PacketDelayVariation.RFC3393.Soak - -3. Scalability tests - - - Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss - - Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability - - LTD.Scalability.VNF.RFC2544.PacketLossProfile - - LTD.Scalability.VNF.RFC2544.PacketLossRatio - -4. Activation tests - - - Test ID: LTD.Activation.RFC2889.AddressCachingCapacity - - Test ID: LTD.Activation.RFC2889.AddressLearningRate - -5. Coupling between control path and datapath Tests - - - Test ID: LTD.CPDPCouplingFlowAddition - -6. CPU and memory consumption - - - Test ID: LTD.Stress.RFC2544.0PacketLoss diff --git a/docs/testing/developer/requirements/vswitchperf_ltp.rst b/docs/testing/developer/requirements/vswitchperf_ltp.rst deleted file mode 100644 index 2b74d676..00000000 --- a/docs/testing/developer/requirements/vswitchperf_ltp.rst +++ /dev/null @@ -1,1348 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Intel Corporation, AT&T and others. - -.. 3.1 - -***************************** -VSPERF LEVEL TEST PLAN (LTP) -***************************** - -=============== -Introduction -=============== - -The objective of the OPNFV project titled -**Characterize vSwitch Performance for Telco NFV Use Cases**, is to -evaluate the performance of virtual switches to identify its suitability for a -Telco Network Function Virtualization (NFV) environment. The intention of this -Level Test Plan (LTP) document is to specify the scope, approach, resources, -and schedule of the virtual switch performance benchmarking activities in -OPNFV. The test cases will be identified in a separate document called the -Level Test Design (LTD) document. - -This document is currently in draft form. - -.. 3.1.1 - - -.. _doc-id: - -Document identifier -========================= - -The document id will be used to uniquely identify versions of the LTP. The -format for the document id will be: OPNFV\_vswitchperf\_LTP\_REL\_STATUS, where -by the status is one of: draft, reviewed, corrected or final. The document id -for this version of the LTP is: OPNFV\_vswitchperf\_LTP\_Colorado\_REVIEWED. - -.. 3.1.2 - -.. _scope: - -Scope -========== - -The main purpose of this project is to specify a suite of -performance tests in order to objectively measure the current packet -transfer characteristics of a virtual switch in the NFVI. The intent of -the project is to facilitate the performance testing of any virtual switch. -Thus, a generic suite of tests shall be developed, with no hard dependencies to -a single implementation. In addition, the test case suite shall be -architecture independent. - -The test cases developed in this project shall not form part of a -separate test framework, all of these tests may be inserted into the -Continuous Integration Test Framework and/or the Platform Functionality -Test Framework - if a vSwitch becomes a standard component of an OPNFV -release. - -.. 3.1.3 - -References -=============== - -* `RFC 1242 Benchmarking Terminology for Network Interconnection - Devices <http://www.ietf.org/rfc/rfc1242.txt>`__ -* `RFC 2544 Benchmarking Methodology for Network Interconnect - Devices <http://www.ietf.org/rfc/rfc2544.txt>`__ -* `RFC 2285 Benchmarking Terminology for LAN Switching - Devices <http://www.ietf.org/rfc/rfc2285.txt>`__ -* `RFC 2889 Benchmarking Methodology for LAN Switching - Devices <http://www.ietf.org/rfc/rfc2889.txt>`__ -* `RFC 3918 Methodology for IP Multicast - Benchmarking <http://www.ietf.org/rfc/rfc3918.txt>`__ -* `RFC 4737 Packet Reordering - Metrics <http://www.ietf.org/rfc/rfc4737.txt>`__ -* `RFC 5481 Packet Delay Variation Applicability - Statement <http://www.ietf.org/rfc/rfc5481.txt>`__ -* `RFC 6201 Device Reset - Characterization <http://tools.ietf.org/html/rfc6201>`__ - -.. 3.1.4 - -Level in the overall sequence -=============================== -The level of testing conducted by vswitchperf in the overall testing sequence (among -all the testing projects in OPNFV) is the performance benchmarking of a -specific component (the vswitch) in the OPNFV platfrom. It's expected that this -testing will follow on from the functional and integration testing conducted by -other testing projects in OPNFV, namely Functest and Yardstick. - -.. 3.1.5 - -Test classes and overall test conditions -========================================= -A benchmark is defined by the IETF as: A standardized test that serves as a -basis for performance evaluation and comparison. It's important to note that -benchmarks are not Functional tests. They do not provide PASS/FAIL criteria, -and most importantly ARE NOT performed on live networks, or performed with live -network traffic. - -In order to determine the packet transfer characteristics of a virtual switch, -the benchmarking tests will be broken down into the following categories: - -- **Throughput Tests** to measure the maximum forwarding rate (in - frames per second or fps) and bit rate (in Mbps) for a constant load - (as defined by `RFC1242 <https://www.rfc-editor.org/rfc/rfc1242.txt>`__) - without traffic loss. -- **Packet and Frame Delay Tests** to measure average, min and max - packet and frame delay for constant loads. -- **Stream Performance Tests** (TCP, UDP) to measure bulk data transfer - performance, i.e. how fast systems can send and receive data through - the virtual switch. -- **Request/Response Performance** Tests (TCP, UDP) the measure the - transaction rate through the virtual switch. -- **Packet Delay Tests** to understand latency distribution for - different packet sizes and over an extended test run to uncover - outliers. -- **Scalability Tests** to understand how the virtual switch performs - as the number of flows, active ports, complexity of the forwarding - logic's configuration... it has to deal with increases. -- **Control Path and Datapath Coupling** Tests, to understand how - closely coupled the datapath and the control path are as well as the - effect of this coupling on the performance of the DUT. -- **CPU and Memory Consumption Tests** to understand the virtual - switch’s footprint on the system, this includes: - - * CPU core utilization. - * CPU cache utilization. - * Memory footprint. - * System bus (QPI, PCI, ..) utilization. - * Memory lanes utilization. - * CPU cycles consumed per packet. - * Time To Establish Flows Tests. - -- **Noisy Neighbour Tests**, to understand the effects of resource - sharing on the performance of a virtual switch. - -**Note:** some of the tests above can be conducted simultaneously where -the combined results would be insightful, for example Packet/Frame Delay -and Scalability. - - - -.. 3.2 - -.. _details-of-LTP: - -=================================== -Details of the Level Test Plan -=================================== - -This section describes the following items: -* Test items and their identifiers (TestItems_) -* Test Traceability Matrix (TestMatrix_) -* Features to be tested (FeaturesToBeTested_) -* Features not to be tested (FeaturesNotToBeTested_) -* Approach (Approach_) -* Item pass/fail criteria (PassFailCriteria_) -* Suspension criteria and resumption requirements (SuspensionResumptionReqs_) - -.. 3.2.1 - -.. _TestItems: - -Test items and their identifiers -================================== -The test item/application vsperf is trying to test are virtual switches and in -particular their performance in an nfv environment. vsperf will first try to -measure the maximum achievable performance by a virtual switch and then it will -focus in on usecases that are as close to real life deployment scenarios as -possible. - -.. 3.2.2 - -.. _TestMatrix: - -Test Traceability Matrix -========================== -vswitchperf leverages the "3x3" matrix (introduced in -https://tools.ietf.org/html/draft-ietf-bmwg-virtual-net-02) to achieve test -traceability. The matrix was expanded to 3x4 to accommodate scale metrics when -displaying the coverage of many metrics/benchmarks). Test case covreage in the -LTD is tracked using the following catagories: - - -+---------------+-------------+------------+---------------+-------------+ -| | | | | | -| | SPEED | ACCURACY | RELIABILITY | SCALE | -| | | | | | -+---------------+-------------+------------+---------------+-------------+ -| | | | | | -| Activation | X | X | X | X | -| | | | | | -+---------------+-------------+------------+---------------+-------------+ -| | | | | | -| Operation | X | X | X | X | -| | | | | | -+---------------+-------------+------------+---------------+-------------+ -| | | | | | -| De-activation | | | | | -| | | | | | -+---------------+-------------+------------+---------------+-------------+ - -X = denotes a test catagory that has 1 or more test cases defined. - -.. 3.2.3 - -.. _FeaturesToBeTested: - -Features to be tested -========================== - -Characterizing virtual switches (i.e. Device Under Test (DUT) in this document) -includes measuring the following performance metrics: - -- **Throughput** as defined by `RFC1242 - <https://www.rfc-editor.org/rfc/rfc1242.txt>`__: The maximum rate at which - **none** of the offered frames are dropped by the DUT. The maximum frame - rate and bit rate that can be transmitted by the DUT without any error - should be recorded. Note there is an equivalent bit rate and a specific - layer at which the payloads contribute to the bits. Errors and - improperly formed frames or packets are dropped. -- **Packet delay** introduced by the DUT and its cumulative effect on - E2E networks. Frame delay can be measured equivalently. -- **Packet delay variation**: measured from the perspective of the - VNF/application. Packet delay variation is sometimes called "jitter". - However, we will avoid the term "jitter" as the term holds different - meaning to different groups of people. In this document we will - simply use the term packet delay variation. The preferred form for this - metric is the PDV form of delay variation defined in `RFC5481 - <https://www.rfc-editor.org/rfc/rfc5481.txt>`__. The most relevant - measurement of PDV considers the delay variation of a single user flow, - as this will be relevant to the size of end-system buffers to compensate - for delay variation. The measurement system's ability to store the - delays of individual packets in the flow of interest is a key factor - that determines the specific measurement method. At the outset, it is - ideal to view the complete PDV distribution. Systems that can capture - and store packets and their delays have the freedom to calculate the - reference minimum delay and to determine various quantiles of the PDV - distribution accurately (in post-measurement processing routines). - Systems without storage must apply algorithms to calculate delay and - statistical measurements on the fly. For example, a system may store - temporary estimates of the mimimum delay and the set of (100) packets - with the longest delays during measurement (to calculate a high quantile, - and update these sets with new values periodically. - In some cases, a limited number of delay histogram bins will be - available, and the bin limits will need to be set using results from - repeated experiments. See section 8 of `RFC5481 - <https://www.rfc-editor.org/rfc/rfc5481.txt>`__. -- **Packet loss** (within a configured waiting time at the receiver): All - packets sent to the DUT should be accounted for. -- **Burst behaviour**: measures the ability of the DUT to buffer packets. -- **Packet re-ordering**: measures the ability of the device under test to - maintain sending order throughout transfer to the destination. -- **Packet correctness**: packets or Frames must be well-formed, in that - they include all required fields, conform to length requirements, pass - integrity checks, etc. -- **Availability and capacity** of the DUT i.e. when the DUT is fully “up” - and connected, following measurements should be captured for - DUT without any network packet load: - - - Includes average power consumption of the CPUs (in various power states) and - system over specified period of time. Time period should not be less - than 60 seconds. - - Includes average per core CPU utilization over specified period of time. - Time period should not be less than 60 seconds. - - Includes the number of NIC interfaces supported. - - Includes headroom of VM workload processing cores (i.e. available - for applications). - -.. 3.2.4 - -.. _FeaturesNotToBeTested: - -Features not to be tested -========================== -vsperf doesn't intend to define or perform any functional tests. The aim is to -focus on performance. - -.. 3.2.5 - -.. _Approach: - -Approach -============== -The testing approach adoped by the vswitchperf project is black box testing, -meaning the test inputs can be generated and the outputs captured and -completely evaluated from the outside of the System Under Test. Some metrics -can be collected on the SUT, such as cpu or memory utilization if the -collection has no/minimal impact on benchmark. -This section will look at the deployment scenarios and the general methodology -used by vswitchperf. In addition, this section will also specify the details of -the Test Report that must be collected for each of the test cases. - -.. 3.2.5.1 - -Deployment Scenarios --------------------------- -The following represents possible deployment test scenarios which can -help to determine the performance of both the virtual switch and the -datapaths to physical ports (to NICs) and to logical ports (to VNFs): - -.. 3.2.5.1.1 - -.. _Phy2Phy: - -Physical port → vSwitch → physical port -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. code-block:: console - - _ - +--------------------------------------------------+ | - | +--------------------+ | | - | | | | | - | | v | | Host - | +--------------+ +--------------+ | | - | | phy port | vSwitch | phy port | | | - +---+--------------+------------+--------------+---+ _| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - -.. 3.2.5.1.2 - -.. _PVP: - -Physical port → vSwitch → VNF → vSwitch → physical port -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. code-block:: console - - _ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | ^ : | | - | | | | | Guest - | : v | | - | +---------------+ +---------------+ | | - | | logical port 0| | logical port 1| | | - +---+---------------+-----------+---------------+---+ _| - ^ : - | | - : v _ - +---+---------------+----------+---------------+---+ | - | | logical port 0| | logical port 1| | | - | +---------------+ +---------------+ | | - | ^ : | | - | | | | | Host - | : v | | - | +--------------+ +--------------+ | | - | | phy port | vSwitch | phy port | | | - +---+--------------+------------+--------------+---+ _| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - -.. 3.2.5.1.3 - -.. _PVVP: - -Physical port → vSwitch → VNF → vSwitch → VNF → vSwitch → physical port -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: console - - _ - +----------------------+ +----------------------+ | - | Guest 1 | | Guest 2 | | - | +---------------+ | | +---------------+ | | - | | Application | | | | Application | | | - | +---------------+ | | +---------------+ | | - | ^ | | | ^ | | | - | | v | | | v | | Guests - | +---------------+ | | +---------------+ | | - | | logical ports | | | | logical ports | | | - | | 0 1 | | | | 0 1 | | | - +---+---------------+--+ +---+---------------+--+ _| - ^ : ^ : - | | | | - : v : v _ - +---+---------------+---------+---------------+--+ | - | | 0 1 | | 3 4 | | | - | | logical ports | | logical ports | | | - | +---------------+ +---------------+ | | - | ^ | ^ | | | Host - | | L-----------------+ v | | - | +--------------+ +--------------+ | | - | | phy ports | vSwitch | phy ports | | | - +---+--------------+----------+--------------+---+ _| - ^ ^ : : - | | | | - : : v v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - -.. 3.2.5.1.4 - -Physical port → VNF → vSwitch → VNF → physical port -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: console - - _ - +----------------------+ +----------------------+ | - | Guest 1 | | Guest 2 | | - |+-------------------+ | | +-------------------+| | - || Application | | | | Application || | - |+-------------------+ | | +-------------------+| | - | ^ | | | ^ | | | Guests - | | v | | | v | | - |+-------------------+ | | +-------------------+| | - || logical ports | | | | logical ports || | - || 0 1 | | | | 0 1 || | - ++--------------------++ ++--------------------++ _| - ^ : ^ : - (PCI passthrough) | | (PCI passthrough) - | v : | _ - +--------++------------+-+------------++---------+ | - | | || 0 | | 1 || | | | - | | ||logical port| |logical port|| | | | - | | |+------------+ +------------+| | | | - | | | | ^ | | | | - | | | L-----------------+ | | | | - | | | | | | | Host - | | | vSwitch | | | | - | | +-----------------------------+ | | | - | | | | | - | | v | | - | +--------------+ +--------------+ | | - | | phy port/VF | | phy port/VF | | | - +-+--------------+--------------+--------------+-+ _| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - -.. 3.2.5.1.5 - -Physical port → vSwitch → VNF -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: console - - _ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | ^ | | - | | | | Guest - | : | | - | +---------------+ | | - | | logical port 0| | | - +---+---------------+-------------------------------+ _| - ^ - | - : _ - +---+---------------+------------------------------+ | - | | logical port 0| | | - | +---------------+ | | - | ^ | | - | | | | Host - | : | | - | +--------------+ | | - | | phy port | vSwitch | | - +---+--------------+------------ -------------- ---+ _| - ^ - | - : - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - -.. 3.2.5.1.6 - -VNF → vSwitch → physical port -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: console - - _ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | : | | - | | | | Guest - | v | | - | +---------------+ | | - | | logical port | | | - +-------------------------------+---------------+---+ _| - : - | - v _ - +------------------------------+---------------+---+ | - | | logical port | | | - | +---------------+ | | - | : | | - | | | | Host - | v | | - | +--------------+ | | - | vSwitch | phy port | | | - +-------------------------------+--------------+---+ _| - : - | - v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - -.. 3.2.5.1.7 - -VNF → vSwitch → VNF → vSwitch -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: console - - _ - +-------------------------+ +-------------------------+ | - | Guest 1 | | Guest 2 | | - | +-----------------+ | | +-----------------+ | | - | | Application | | | | Application | | | - | +-----------------+ | | +-----------------+ | | - | : | | ^ | | - | | | | | | | Guest - | v | | : | | - | +---------------+ | | +---------------+ | | - | | logical port 0| | | | logical port 0| | | - +-----+---------------+---+ +---+---------------+-----+ _| - : ^ - | | - v : _ - +----+---------------+------------+---------------+-----+ | - | | port 0 | | port 1 | | | - | +---------------+ +---------------+ | | - | : ^ | | - | | | | | Host - | +--------------------+ | | - | | | - | vswitch | | - +-------------------------------------------------------+ _| - -.. 3.2.5.1.8 - -HOST 1(Physical port → virtual switch → VNF → virtual switch → Physical port) -→ HOST 2(Physical port → virtual switch → VNF → virtual switch → Physical port) - -HOST 1 (PVP) → HOST 2 (PVP) -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: console - - _ - +----------------------+ +----------------------+ | - | Guest 1 | | Guest 2 | | - | +---------------+ | | +---------------+ | | - | | Application | | | | Application | | | - | +---------------+ | | +---------------+ | | - | ^ | | | ^ | | | - | | v | | | v | | Guests - | +---------------+ | | +---------------+ | | - | | logical ports | | | | logical ports | | | - | | 0 1 | | | | 0 1 | | | - +---+---------------+--+ +---+---------------+--+ _| - ^ : ^ : - | | | | - : v : v _ - +---+---------------+--+ +---+---------------+--+ | - | | 0 1 | | | | 3 4 | | | - | | logical ports | | | | logical ports | | | - | +---------------+ | | +---------------+ | | - | ^ | | | ^ | | | Hosts - | | v | | | v | | - | +--------------+ | | +--------------+ | | - | | phy ports | | | | phy ports | | | - +---+--------------+---+ +---+--------------+---+ _| - ^ : : : - | +-----------------+ | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - - - -**Note:** For tests where the traffic generator and/or measurement -receiver are implemented on VM and connected to the virtual switch -through vNIC, the issues of shared resources and interactions between -the measurement devices and the device under test must be considered. - -**Note:** Some RFC 2889 tests require a full-mesh sending and receiving -pattern involving more than two ports. This possibility is illustrated in the -Physical port → vSwitch → VNF → vSwitch → VNF → vSwitch → physical port -diagram above (with 2 sending and 2 receiving ports, though all ports -could be used bi-directionally). - -**Note:** When Deployment Scenarios are used in RFC 2889 address learning -or cache capacity testing, an additional port from the vSwitch must be -connected to the test device. This port is used to listen for flooded -frames. - -.. 3.2.5.2 - -General Methodology: --------------------------- -To establish the baseline performance of the virtual switch, tests would -initially be run with a simple workload in the VNF (the recommended -simple workload VNF would be `DPDK <http://www.dpdk.org/>`__'s testpmd -application forwarding packets in a VM or vloop\_vnf a simple kernel -module that forwards traffic between two network interfaces inside the -virtualized environment while bypassing the networking stack). -Subsequently, the tests would also be executed with a real Telco -workload running in the VNF, which would exercise the virtual switch in -the context of higher level Telco NFV use cases, and prove that its -underlying characteristics and behaviour can be measured and validated. -Suitable real Telco workload VNFs are yet to be identified. - -.. 3.2.5.2.1 - -.. _default-test-parameters: - -Default Test Parameters -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The following list identifies the default parameters for suite of -tests: - -- Reference application: Simple forwarding or Open Source VNF. -- Frame size (bytes): 64, 128, 256, 512, 1024, 1280, 1518, 2K, 4k OR - Packet size based on use-case (e.g. RTP 64B, 256B) OR Mix of packet sizes as - maintained by the Functest project <https://wiki.opnfv.org/traffic_profile_management>. -- Reordering check: Tests should confirm that packets within a flow are - not reordered. -- Duplex: Unidirectional / Bidirectional. Default: Full duplex with - traffic transmitting in both directions, as network traffic generally - does not flow in a single direction. By default the data rate of - transmitted traffic should be the same in both directions, please - note that asymmetric traffic (e.g. downlink-heavy) tests will be - mentioned explicitly for the relevant test cases. -- Number of Flows: Default for non scalability tests is a single flow. - For scalability tests the goal is to test with maximum supported - flows but where possible will test up to 10 Million flows. Start with - a single flow and scale up. By default flows should be added - sequentially, tests that add flows simultaneously will explicitly - call out their flow addition behaviour. Packets are generated across - the flows uniformly with no burstiness. For multi-core tests should - consider the number of packet flows based on vSwitch/VNF multi-thread - implementation and behavior. - -- Traffic Types: UDP, SCTP, RTP, GTP and UDP traffic. -- Deployment scenarios are: -- Physical → virtual switch → physical. -- Physical → virtual switch → VNF → virtual switch → physical. -- Physical → virtual switch → VNF → virtual switch → VNF → virtual - switch → physical. -- Physical → VNF → virtual switch → VNF → physical. -- Physical → virtual switch → VNF. -- VNF → virtual switch → Physical. -- VNF → virtual switch → VNF. - -Tests MUST have these parameters unless otherwise stated. **Test cases -with non default parameters will be stated explicitly**. - -**Note**: For throughput tests unless stated otherwise, test -configurations should ensure that traffic traverses the installed flows -through the virtual switch, i.e. flows are installed and have an appropriate -time out that doesn't expire before packet transmission starts. - -.. 3.2.5.2.2 - -Flow Classification -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Virtual switches classify packets into flows by processing and matching -particular header fields in the packet/frame and/or the input port where -the packets/frames arrived. The vSwitch then carries out an action on -the group of packets that match the classification parameters. Thus a -flow is considered to be a sequence of packets that have a shared set of -header field values or have arrived on the same port and have the same -action applied to them. Performance results can vary based on the -parameters the vSwitch uses to match for a flow. The recommended flow -classification parameters for L3 vSwitch performance tests are: the -input port, the source IP address, the destination IP address and the -Ethernet protocol type field. It is essential to increase the flow -time-out time on a vSwitch before conducting any performance tests that -do not measure the flow set-up time. Normally the first packet of a -particular flow will install the flow in the vSwitch which adds an -additional latency, subsequent packets of the same flow are not subject -to this latency if the flow is already installed on the vSwitch. - -.. 3.2.5.2.3 - -Test Priority -~~~~~~~~~~~~~~~~~~~~~ - -Tests will be assigned a priority in order to determine which tests -should be implemented immediately and which tests implementations -can be deferred. - -Priority can be of following types: - Urgent: Must be implemented -immediately. - High: Must be implemented in the next release. - Medium: -May be implemented after the release. - Low: May or may not be -implemented at all. - -.. 3.2.5.2.4 - -SUT Setup -~~~~~~~~~~~~~~~~~~ - -The SUT should be configured to its "default" state. The -SUT's configuration or set-up must not change between tests in any way -other than what is required to do the test. All supported protocols must -be configured and enabled for each test set up. - -.. 3.2.5.2.5 - -Port Configuration -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The DUT should be configured with n ports where -n is a multiple of 2. Half of the ports on the DUT should be used as -ingress ports and the other half of the ports on the DUT should be used -as egress ports. Where a DUT has more than 2 ports, the ingress data -streams should be set-up so that they transmit packets to the egress -ports in sequence so that there is an even distribution of traffic -across ports. For example, if a DUT has 4 ports 0(ingress), 1(ingress), -2(egress) and 3(egress), the traffic stream directed at port 0 should -output a packet to port 2 followed by a packet to port 3. The traffic -stream directed at port 1 should also output a packet to port 2 followed -by a packet to port 3. - -.. 3.2.5.2.6 - -Frame Formats -~~~~~~~~~~~~~~~~~~~~~ - -**Frame formats Layer 2 (data link layer) protocols** - -- Ethernet II - -.. code-block:: console - - +---------------------------+-----------+ - | Ethernet Header | Payload | Check Sum | - +-----------------+---------+-----------+ - |_________________|_________|___________| - 14 Bytes 46 - 1500 4 Bytes - Bytes - - -**Layer 3 (network layer) protocols** - -- IPv4 - -.. code-block:: console - - +-----------------+-----------+---------+-----------+ - | Ethernet Header | IP Header | Payload | Checksum | - +-----------------+-----------+---------+-----------+ - |_________________|___________|_________|___________| - 14 Bytes 20 bytes 26 - 1480 4 Bytes - Bytes - -- IPv6 - -.. code-block:: console - - +-----------------+-----------+---------+-----------+ - | Ethernet Header | IP Header | Payload | Checksum | - +-----------------+-----------+---------+-----------+ - |_________________|___________|_________|___________| - 14 Bytes 40 bytes 26 - 1460 4 Bytes - Bytes - -**Layer 4 (transport layer) protocols** - - - TCP - - UDP - - SCTP - -.. code-block:: console - - +-----------------+-----------+-----------------+---------+-----------+ - | Ethernet Header | IP Header | Layer 4 Header | Payload | Checksum | - +-----------------+-----------+-----------------+---------+-----------+ - |_________________|___________|_________________|_________|___________| - 14 Bytes 40 bytes 20 Bytes 6 - 1460 4 Bytes - Bytes - - -**Layer 5 (application layer) protocols** - - - RTP - - GTP - -.. code-block:: console - - +-----------------+-----------+-----------------+---------+-----------+ - | Ethernet Header | IP Header | Layer 4 Header | Payload | Checksum | - +-----------------+-----------+-----------------+---------+-----------+ - |_________________|___________|_________________|_________|___________| - 14 Bytes 20 bytes 20 Bytes >= 6 Bytes 4 Bytes - -.. 3.2.5.2.7 - -Packet Throughput -~~~~~~~~~~~~~~~~~~~~~~~~~ -There is a difference between an Ethernet frame, -an IP packet, and a UDP datagram. In the seven-layer OSI model of -computer networking, packet refers to a data unit at layer 3 (network -layer). The correct term for a data unit at layer 2 (data link layer) is -a frame, and at layer 4 (transport layer) is a segment or datagram. - -Important concepts related to 10GbE performance are frame rate and -throughput. The MAC bit rate of 10GbE, defined in the IEEE standard 802 -.3ae, is 10 billion bits per second. Frame rate is based on the bit rate -and frame format definitions. Throughput, defined in IETF RFC 1242, is -the highest rate at which the system under test can forward the offered -load, without loss. - -The frame rate for 10GbE is determined by a formula that divides the 10 -billion bits per second by the preamble + frame length + inter-frame -gap. - -The maximum frame rate is calculated using the minimum values of the -following parameters, as described in the IEEE 802 .3ae standard: - -- Preamble: 8 bytes \* 8 = 64 bits -- Frame Length: 64 bytes (minimum) \* 8 = 512 bits -- Inter-frame Gap: 12 bytes (minimum) \* 8 = 96 bits - -Therefore, Maximum Frame Rate (64B Frames) -= MAC Transmit Bit Rate / (Preamble + Frame Length + Inter-frame Gap) -= 10,000,000,000 / (64 + 512 + 96) -= 10,000,000,000 / 672 -= 14,880,952.38 frame per second (fps) - -.. 3.2.5.3 - -RFCs for testing virtual switch performance --------------------------------------------------- - -The starting point for defining the suite of tests for benchmarking the -performance of a virtual switch is to take existing RFCs and standards -that were designed to test their physical counterparts and adapting them -for testing virtual switches. The rationale behind this is to establish -a fair comparison between the performance of virtual and physical -switches. This section outlines the RFCs that are used by this -specification. - -.. 3.2.5.3.1 - -RFC 1242 Benchmarking Terminology for Network Interconnection -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Devices RFC 1242 defines the terminology that is used in describing -performance benchmarking tests and their results. Definitions and -discussions covered include: Back-to-back, bridge, bridge/router, -constant load, data link frame size, frame loss rate, inter frame gap, -latency, and many more. - -.. 3.2.5.3.2 - -RFC 2544 Benchmarking Methodology for Network Interconnect Devices -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -RFC 2544 outlines a benchmarking methodology for network Interconnect -Devices. The methodology results in performance metrics such as latency, -frame loss percentage, and maximum data throughput. - -In this document network “throughput” (measured in millions of frames -per second) is based on RFC 2544, unless otherwise noted. Frame size -refers to Ethernet frames ranging from smallest frames of 64 bytes to -largest frames of 9K bytes. - -Types of tests are: - -1. Throughput test defines the maximum number of frames per second - that can be transmitted without any error, or 0% loss ratio. - In some Throughput tests (and those tests with long duration), - evaluation of an additional frame loss ratio is suggested. The - current ratio (10^-7 %) is based on understanding the typical - user-to-user packet loss ratio needed for good application - performance and recognizing that a single transfer through a - vswitch must contribute a tiny fraction of user-to-user loss. - Further, the ratio 10^-7 % also recognizes practical limitations - when measuring loss ratio. - -2. Latency test measures the time required for a frame to travel from - the originating device through the network to the destination device. - Please note that RFC2544 Latency measurement will be superseded with - a measurement of average latency over all successfully transferred - packets or frames. - -3. Frame loss test measures the network’s - response in overload conditions - a critical indicator of the - network’s ability to support real-time applications in which a - large amount of frame loss will rapidly degrade service quality. - -4. Burst test assesses the buffering capability of a virtual switch. It - measures the maximum number of frames received at full line rate - before a frame is lost. In carrier Ethernet networks, this - measurement validates the excess information rate (EIR) as defined in - many SLAs. - -5. System recovery to characterize speed of recovery from an overload - condition. - -6. Reset to characterize speed of recovery from device or software - reset. This type of test has been updated by `RFC6201 - <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ as such, - the methodology defined by this specification will be that of RFC 6201. - -Although not included in the defined RFC 2544 standard, another crucial -measurement in Ethernet networking is packet delay variation. The -definition set out by this specification comes from -`RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__. - -.. 3.2.5.3.3 - -RFC 2285 Benchmarking Terminology for LAN Switching Devices -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -RFC 2285 defines the terminology that is used to describe the -terminology for benchmarking a LAN switching device. It extends RFC -1242 and defines: DUTs, SUTs, Traffic orientation and distribution, -bursts, loads, forwarding rates, etc. - -.. 3.2.5.3.4 - -RFC 2889 Benchmarking Methodology for LAN Switching -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -RFC 2889 outlines a benchmarking methodology for LAN switching, it -extends RFC 2544. The outlined methodology gathers performance -metrics for forwarding, congestion control, latency, address handling -and finally filtering. - -.. 3.2.5.3.5 - -RFC 3918 Methodology for IP Multicast Benchmarking -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -RFC 3918 outlines a methodology for IP Multicast benchmarking. - -.. 3.2.5.3.6 - -RFC 4737 Packet Reordering Metrics -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -RFC 4737 describes metrics for identifying and counting re-ordered -packets within a stream, and metrics to measure the extent each -packet has been re-ordered. - -.. 3.2.5.3.7 - -RFC 5481 Packet Delay Variation Applicability Statement -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -RFC 5481 defined two common, but different forms of delay variation -metrics, and compares the metrics over a range of networking -circumstances and tasks. The most suitable form for vSwitch -benchmarking is the "PDV" form. - -.. 3.2.5.3.8 - -RFC 6201 Device Reset Characterization -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -RFC 6201 extends the methodology for characterizing the speed of -recovery of the DUT from device or software reset described in RFC -2544. - -.. 3.2.6: - -.. _PassFailCriteria: - -Item pass/fail criteria -========================= - -vswitchperf does not specify Pass/Fail criteria for the tests in terms of a -threshold, as benchmarks do not (and should not do this). The results/metrics -for a test are simply reported. If it had to be defined, a test is considered -to have passed if it succesfully completed and a relavent metric was -recorded/reported for the SUT. - -.. 3.2.7: - -.. _SuspensionResumptionReqs: - -Suspension criteria and resumption requirements -================================================ -In the case of a throughput test, a test should be suspended if a virtual -switch is failing to forward any traffic. A test should be restarted from a -clean state if the intention is to carry out the test again. - -.. 3.2.8: - -.. _TestDelierables: - -Test deliverables -================== -Each test should produce a test report that details SUT information as well as -the test results. There are a number of parameters related to the system, DUT -and tests that can affect the repeatability of a test results and should be -recorded. In order to minimise the variation in the results of a test, -it is recommended that the test report includes the following information: - -- Hardware details including: - - - Platform details. - - Processor details. - - Memory information (see below) - - Number of enabled cores. - - Number of cores used for the test. - - Number of physical NICs, as well as their details (manufacturer, - versions, type and the PCI slot they are plugged into). - - NIC interrupt configuration. - - BIOS version, release date and any configurations that were - modified. - -- Software details including: - - - OS version (for host and VNF) - - Kernel version (for host and VNF) - - GRUB boot parameters (for host and VNF). - - Hypervisor details (Type and version). - - Selected vSwitch, version number or commit id used. - - vSwitch launch command line if it has been parameterised. - - Memory allocation to the vSwitch – which NUMA node it is using, - and how many memory channels. - - Where the vswitch is built from source: compiler details including - versions and the flags that were used to compile the vSwitch. - - DPDK or any other SW dependency version number or commit id used. - - Memory allocation to a VM - if it's from Hugpages/elsewhere. - - VM storage type: snapshot/independent persistent/independent - non-persistent. - - Number of VMs. - - Number of Virtual NICs (vNICs), versions, type and driver. - - Number of virtual CPUs and their core affinity on the host. - - Number vNIC interrupt configuration. - - Thread affinitization for the applications (including the vSwitch - itself) on the host. - - Details of Resource isolation, such as CPUs designated for - Host/Kernel (isolcpu) and CPUs designated for specific processes - (taskset). - -- Memory Details - - - Total memory - - Type of memory - - Used memory - - Active memory - - Inactive memory - - Free memory - - Buffer memory - - Swap cache - - Total swap - - Used swap - - Free swap - -- Test duration. -- Number of flows. -- Traffic Information: - - - Traffic type - UDP, TCP, IMIX / Other. - - Packet Sizes. - -- Deployment Scenario. - -**Note**: Tests that require additional parameters to be recorded will -explicitly specify this. - - -.. 3.3: - -.. _TestManagement: - -Test management -================= -This section will detail the test activities that will be conducted by vsperf -as well as the infrastructure that will be used to complete the tests in OPNFV. - -.. 3.3.1: - -Planned activities and tasks; test progression -================================================= -A key consideration when conducting any sort of benchmark is trying to -ensure the consistency and repeatability of test results between runs. -When benchmarking the performance of a virtual switch there are many -factors that can affect the consistency of results. This section -describes these factors and the measures that can be taken to limit -their effects. In addition, this section will outline some system tests -to validate the platform and the VNF before conducting any vSwitch -benchmarking tests. - -**System Isolation:** - -When conducting a benchmarking test on any SUT, it is essential to limit -(and if reasonable, eliminate) any noise that may interfere with the -accuracy of the metrics collected by the test. This noise may be -introduced by other hardware or software (OS, other applications), and -can result in significantly varying performance metrics being collected -between consecutive runs of the same test. In the case of characterizing -the performance of a virtual switch, there are a number of configuration -parameters that can help increase the repeatability and stability of -test results, including: - -- OS/GRUB configuration: - - - maxcpus = n where n >= 0; limits the kernel to using 'n' - processors. Only use exactly what you need. - - isolcpus: Isolate CPUs from the general scheduler. Isolate all - CPUs bar one which will be used by the OS. - - use taskset to affinitize the forwarding application and the VNFs - onto isolated cores. VNFs and the vSwitch should be allocated - their own cores, i.e. must not share the same cores. vCPUs for the - VNF should be affinitized to individual cores also. - - Limit the amount of background applications that are running and - set OS to boot to runlevel 3. Make sure to kill any unnecessary - system processes/daemons. - - Only enable hardware that you need to use for your test – to - ensure there are no other interrupts on the system. - - Configure NIC interrupts to only use the cores that are not - allocated to any other process (VNF/vSwitch). - -- NUMA configuration: Any unused sockets in a multi-socket system - should be disabled. -- CPU pinning: The vSwitch and the VNF should each be affinitized to - separate logical cores using a combination of maxcpus, isolcpus and - taskset. -- BIOS configuration: BIOS should be configured for performance where - an explicit option exists, sleep states should be disabled, any - virtualization optimization technologies should be enabled, and - hyperthreading should also be enabled, turbo boost and overclocking - should be disabled. - -**System Validation:** - -System validation is broken down into two sub-categories: Platform -validation and VNF validation. The validation test itself involves -verifying the forwarding capability and stability for the sub-system -under test. The rationale behind system validation is two fold. Firstly -to give a tester confidence in the stability of the platform or VNF that -is being tested; and secondly to provide base performance comparison -points to understand the overhead introduced by the virtual switch. - -* Benchmark platform forwarding capability: This is an OPTIONAL test - used to verify the platform and measure the base performance (maximum - forwarding rate in fps and latency) that can be achieved by the - platform without a vSwitch or a VNF. The following diagram outlines - the set-up for benchmarking Platform forwarding capability: - - .. code-block:: console - - __ - +--------------------------------------------------+ | - | +------------------------------------------+ | | - | | | | | - | | l2fw or DPDK L2FWD app | | Host - | | | | | - | +------------------------------------------+ | | - | | NIC | | | - +---+------------------------------------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - -* Benchmark VNF forwarding capability: This test is used to verify - the VNF and measure the base performance (maximum forwarding rate in - fps and latency) that can be achieved by the VNF without a vSwitch. - The performance metrics collected by this test will serve as a key - comparison point for NIC passthrough technologies and vSwitches. VNF - in this context refers to the hypervisor and the VM. The following - diagram outlines the set-up for benchmarking VNF forwarding - capability: - - .. code-block:: console - - __ - +--------------------------------------------------+ | - | +------------------------------------------+ | | - | | | | | - | | VNF | | | - | | | | | - | +------------------------------------------+ | | - | | Passthrough/SR-IOV | | Host - | +------------------------------------------+ | | - | | NIC | | | - +---+------------------------------------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - - -**Methodology to benchmark Platform/VNF forwarding capability** - - -The recommended methodology for the platform/VNF validation and -benchmark is: - Run `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__ -Maximum Forwarding Rate test, this test will produce maximum -forwarding rate and latency results that will serve as the -expected values. These expected values can be used in -subsequent steps or compared with in subsequent validation tests. - -Transmit bidirectional traffic at line rate/max forwarding rate -(whichever is higher) for at least 72 hours, measure throughput (fps) -and latency. - Note: Traffic should be bidirectional. - Establish a -baseline forwarding rate for what the platform can achieve. - Additional -validation: After the test has completed for 72 hours run bidirectional -traffic at the maximum forwarding rate once more to see if the system is -still functional and measure throughput (fps) and latency. Compare the -measure the new obtained values with the expected values. - -**NOTE 1**: How the Platform is configured for its forwarding capability -test (BIOS settings, GRUB configuration, runlevel...) is how the -platform should be configured for every test after this - -**NOTE 2**: How the VNF is configured for its forwarding capability test -(# of vCPUs, vNICs, Memory, affinitization…) is how it should be -configured for every test that uses a VNF after this. - -**Methodology to benchmark the VNF to vSwitch to VNF deployment scenario** - -vsperf has identified the following concerns when benchmarking the VNF to -vSwitch to VNF deployment scenario: - -* The accuracy of the timing synchronization between VNFs/VMs. -* The clock accuracy of a VNF/VM if they were to be used as traffic generators. -* VNF traffic generator/receiver may be using resources of the system under - test, causing at least three forms of workload to increase as the traffic - load increases (generation, switching, receiving). - -The recommendation from vsperf is that tests for this sceanario must -include an external HW traffic generator to act as the tester/traffic transmitter -and receiver. The perscribed methodology to benchmark this deployment scanrio with -an external tester involves the following three steps: - -#. Determine the forwarding capability and latency through the virtual interface -connected to the VNF/VM. - -.. Figure:: vm2vm_virtual_interface_benchmark.png - - Virtual interfaces performance benchmark - -#. Determine the forwarding capability and latency through the VNF/hypervisor. - -.. Figure:: vm2vm_hypervisor_benchmark.png - - Hypervisor performance benchmark - -#. Determine the forwarding capability and latency for the VNF to vSwitch to VNF - taking the information from the previous two steps into account. - -.. Figure:: vm2vm_benchmark.png - - VNF to vSwitch to VNF performance benchmark - -vsperf also identified an alternative configuration for the final step: - -.. Figure:: vm2vm_alternative_benchmark.png - - VNF to vSwitch to VNF alternative performance benchmark - -.. 3.3.2: - -Environment/infrastructure -============================ -Intel is providing a hosted test-bed with nine bare-metal environments -allocated to different OPNFV projects. Currently a number of servers in -`Intel POD 3 <https://wiki.opnfv.org/display/pharos/Intel+Pod3>`__ are -allocated to vsperf: - - * pod3-wcp-node3 and pod3-wcp-node4 which are used for CI jobs. - * pod3-node6 which is used as a vsperf sandbox environment. - -vsperf CI ---------- -vsperf CI jobs are broken down into: - - * Daily job: - - * Runs everyday takes about 10 hours to complete. - * TESTCASES_DAILY='phy2phy_tput back2back phy2phy_tput_mod_vlan - phy2phy_scalability pvp_tput pvp_back2back pvvp_tput pvvp_back2back'. - * TESTPARAM_DAILY='--test-params TRAFFICGEN_PKT_SIZES=(64,128,512,1024,1518)'. - - * Merge job: - - * Runs whenever patches are merged to master. - * Runs a basic Sanity test. - - * Verify job: - - * Runs every time a patch is pushed to gerrit. - * Builds documentation. - -Scripts: --------- -There are 2 scripts that are part of VSPERFs CI: - - * build-vsperf.sh: Lives in the VSPERF repository in the ci/ directory and is - used to run vsperf with the appropriate cli parameters. - * vswitchperf.yml: YAML description of our jenkins job. lives in the RELENG - repository. - -More info on vsperf CI can be found here: -https://wiki.opnfv.org/display/vsperf/VSPERF+CI - -.. 3.3.3: - -Responsibilities and authority -=============================== -The group responsible for managing, designing, preparing and executing the -tests listed in the LTD are the vsperf committers and contributors. The vsperf -committers and contributors should work with the relavent OPNFV projects to -ensure that the infrastructure is in place for testing vswitches, and that the -results are published to common end point (a results database). - diff --git a/docs/testing/developer/results/results.rst b/docs/testing/developer/results/results.rst deleted file mode 100644 index 42df9611..00000000 --- a/docs/testing/developer/results/results.rst +++ /dev/null @@ -1,38 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Intel Corporation, AT&T and others. - -OPNFV VSPERF CI Results -========================= -The VSPERF CI jobs are run daily through Jenkins. Result samples can be found at -https://wiki.opnfv.org/display/vsperf/Vsperf+Results - -The following table maps the results in the test dashboard to the appropriate -test case in the VSPERF Framework and specifies the metric the vertical/Y axis -is plotting. **Please note**, the presence of dpdk within a test name signifies -that the vswitch under test was OVS with DPDK, while its absence indicates that -the vswitch under test was stock OVS. - -===================== ===================== ================== =============== - Dashboard Test Framework Test Metric Guest Interface -===================== ===================== ================== =============== -tput_ovsdpdk phy2phy_tput Throughput (FPS) N/A -tput_ovs phy2phy_tput Throughput (FPS) N/A -b2b_ovsdpdk back2back Back-to-back value N/A -b2b_ovs back2back Back-to-back value N/A -tput_mod_vlan_ovs phy2phy_tput_mod_vlan Throughput (FPS) N/A -tput_mod_vlan_ovsdpdk phy2phy_tput_mod_vlan Throughput (FPS) N/A -scalability_ovs phy2phy_scalability Throughput (FPS) N/A -scalability_ovsdpdk phy2phy_scalability Throughput (FPS) N/A -pvp_tput_ovsdpdkuser pvp_tput Throughput (FPS) vhost-user -pvp_tput_ovsvirtio pvp_tput Throughput (FPS) virtio-net -pvp_b2b_ovsdpdkuser pvp_back2back Back-to-back value vhost-user -pvp_b2b_ovsvirtio pvp_back2back Back-to-back value virtio-net -pvvp_tput_ovsdpdkuser pvvp_tput Throughput (FPS) vhost-user -pvvp_tput_ovsvirtio pvvp_tput Throughput (FPS) virtio-net -pvvp_b2b_ovsdpdkuser pvvp_back2back Throughput (FPS) vhost-user -pvvp_b2b_ovsvirtio pvvp_back2back Throughput (FPS) virtio-net -===================== ===================== ================== =============== - -The loopback application in the VNF used for PVP and PVVP scenarios was DPDK -testpmd. diff --git a/docs/testing/developer/results/scenario.rst b/docs/testing/developer/results/scenario.rst deleted file mode 100644 index 2967fc60..00000000 --- a/docs/testing/developer/results/scenario.rst +++ /dev/null @@ -1,36 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Intel Corporation, AT&T and others. - -OPNFV VSPERF Scenarios -=========================== - -Predefined network topologies: - -* :ref:`Phy2Phy <Phy2Phy>`: Physical port -> vSwitch -> Physical port. -* :ref:`PVP <PVP>`: Physical port -> vSwitch -> VNF -> vSwitch -> Physical port. -* :ref:`PVVP <PVVP>`: Physical port -> vSwitch -> VNF -> vSwitch -> VNF -> vSwitch -> - Physical port. - -Loopback applications in the Guest can be: -* `DPDK testpmd <http://dpdk.org/doc/guides/testpmd_app_ug/index.html>`_. -* Linux Bridge. -* :ref:`l2fwd-module` - -===================== =========================================================== - Testcase Definition -===================== =========================================================== -phy2phy_tput :ref:`PacketLossRatio <PacketLossRatio>` for :ref:`Phy2Phy <Phy2Phy>` -back2back :ref:`BackToBackFrames <BackToBackFrames>` for :ref:`Phy2Phy <Phy2Phy>` -phy2phy_tput_mod_vlan :ref:`PacketLossRatioFrameModification <PacketLossRatioFrameModification>` for :ref:`Phy2Phy <Phy2Phy>` -phy2phy_cont :ref:`Phy2Phy <Phy2Phy>` blast vswitch at x% TX rate and measure throughput -pvp_cont :ref:`PVP <PVP>` blast vswitch at x% TX rate and measure throughput -pvvp_cont :ref:`PVVP <PVVP>` blast vswitch at x% TX rate and measure throughput -phy2phy_scalability :ref:`Scalability0PacketLoss <Scalability0PacketLoss>` for :ref:`Phy2Phy <Phy2Phy>` -pvp_tput :ref:`PacketLossRatio <PacketLossRatio>` for :ref:`PVP <PVP>` -pvp_back2back :ref:`BackToBackFrames <BackToBackFrames>` for :ref:`PVP <PVP>` -pvvp_tput :ref:`PacketLossRatio <PacketLossRatio>` for :ref:`PVVP <PVVP>` -pvvp_back2back :ref:`BackToBackFrames <BackToBackFrames>` for :ref:`PVVP <PVVP>` -phy2phy_cpu_load :ref:`CPU0PacketLoss <CPU0PacketLoss>` for :ref:`Phy2Phy <Phy2Phy>` -phy2phy_mem_load Same as :ref:`CPU0PacketLoss <CPU0PacketLoss>` but using a memory intensive app -===================== =========================================================== |
