diff options
Diffstat (limited to 'docs/testing/user')
| -rw-r--r-- | docs/testing/user/configguide/index.rst | 1 | ||||
| -rw-r--r-- | docs/testing/user/configguide/installation.rst | 2 | ||||
| -rw-r--r-- | docs/testing/user/configguide/tools.rst | 177 | ||||
| -rw-r--r-- | docs/testing/user/configguide/trafficgen.rst | 135 | ||||
| -rw-r--r-- | docs/testing/user/userguide/integration.rst | 12 | ||||
| -rw-r--r-- | docs/testing/user/userguide/testlist.rst | 58 | ||||
| -rw-r--r-- | docs/testing/user/userguide/teststeps.rst | 37 | ||||
| -rw-r--r-- | docs/testing/user/userguide/testusage.rst | 174 | ||||
| -rw-r--r-- | docs/testing/user/userguide/trafficcapture.rst | 8 |
9 files changed, 537 insertions, 67 deletions
diff --git a/docs/testing/user/configguide/index.rst b/docs/testing/user/configguide/index.rst index 83908a97..75a2082d 100644 --- a/docs/testing/user/configguide/index.rst +++ b/docs/testing/user/configguide/index.rst @@ -48,6 +48,7 @@ VSPERF Install and Configuration ./installation.rst ./upgrade.rst ./trafficgen.rst + ./tools.rst ================= VSPERF Test Guide diff --git a/docs/testing/user/configguide/installation.rst b/docs/testing/user/configguide/installation.rst index 7f4d640b..51588007 100644 --- a/docs/testing/user/configguide/installation.rst +++ b/docs/testing/user/configguide/installation.rst @@ -202,7 +202,7 @@ new shell session. Its activation is specific to your OS: .. code:: bash - $ scl enable python33 bash + $ scl enable rh-python34 bash $ source $HOME/vsperfenv/bin/activate * Fedora and Ubuntu diff --git a/docs/testing/user/configguide/tools.rst b/docs/testing/user/configguide/tools.rst new file mode 100644 index 00000000..907e86d2 --- /dev/null +++ b/docs/testing/user/configguide/tools.rst @@ -0,0 +1,177 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Intel Corporation, Spirent, AT&T and others. + +.. _additional-tools-configuration: + +============================================= +'vsperf' Additional Tools Configuration Guide +============================================= + +Overview +-------- + +VSPERF supports the following categories additional tools: + + * `Infrastructure Metrics Collectors`_ + * `Load Generators`_ + * `L3 Cache Management`_ + +Under each category, there are one or more tools supported by VSPERF. +This guide provides the details of how to install (if required) +and configure the above mentioned tools. + +.. _`Infrastructure Metrics Collectors`: + +Infrastructure Metrics Collection +--------------------------------- + +VSPERF supports following two tools for collecting and reporting the metrics: + +* pidstat +* collectd + +*pidstat* is a command in linux systems, which is used for monitoring individual +tasks currently being managed by Linux kernel. In VSPERF this command is used to +monitor *ovs-vswitchd*, *ovsdb-server* and *kvm* processes. + +*collectd* is linux application that collects, stores and transfers various system +metrics. For every category of metrics, there is a separate plugin in collectd. For +example, CPU plugin and Interface plugin provides all the cpu metrics and interface +metrics, respectively. CPU metrics may include user-time, system-time, etc., whereas +interface metrics may include received-packets, dropped-packets, etc. + +Installation +^^^^^^^^^^^^ + +No installation is required for *pidstat*, whereas, collectd has to be installed +separately. For installation of collectd, we recommend to follow the process described +in *OPNFV-Barometer* project, which can be found here `Barometer-Euphrates <http://docs.opnfv.org/en/stable-euphrates/submodules/barometer/docs/release/userguide/feature.userguide.html#building-all-barometer-upstreamed-plugins-from-scratch>`_ or the most +recent release. + +VSPERF assumes that collectd is installed and configured to send metrics over localhost. +The metrics sent should be for the following categories: CPU, Processes, Interface, +OVS, DPDK, Intel-RDT. + +Configuration +^^^^^^^^^^^^^ + +The configuration file for the collectors can be found in **conf/05_collector.conf**. +*pidstat* specific configuration includes: + +* ``PIDSTAT_MONITOR`` - processes to be monitored by pidstat +* ``PIDSTAT_OPTIONS`` - options which will be passed to pidstat command +* ``PIDSTAT_SAMPLE_INTERVAL`` - sampling interval used by pidstat to collect statistics +* ``LOG_FILE_PIDSTAT`` - prefix of pidstat's log file + +The *collectd* configuration option includes: + +* ``COLLECTD_IP`` - IP address where collectd is running +* ``COLLECTD_PORT`` - Port number over which collectd is sending the metrics +* ``COLLECTD_SECURITY_LEVEL`` - Security level for receiving metrics +* ``COLLECTD_AUTH_FILE`` - Authentication file for receiving metrics +* ``LOG_FILE_COLLECTD`` - Prefix for collectd's log file. +* ``COLLECTD_CPU_KEYS`` - Interesting metrics from CPU +* ``COLLECTD_PROCESSES_KEYS`` - Interesting metrics from processes +* ``COLLECTD_INTERFACE_KEYS`` - Interesting metrics from interface +* ``COLLECTD_OVSSTAT_KEYS`` - Interesting metrics from OVS +* ``COLLECTD_DPDKSTAT_KEYS`` - Interesting metrics from DPDK. +* ``COLLECTD_INTELRDT_KEYS`` - Interesting metrics from Intel-RDT +* ``COLLECTD_INTERFACE_XKEYS`` - Metrics to exclude from Interface +* ``COLLECTD_INTELRDT_XKEYS`` - Metrics to exclude from Intel-RDT + + +.. _`Load Generators`: + + +Load Generation +--------------- + +In VSPERF, load generation refers to creating background cpu and memory loads to +study the impact of these loads on system under test. There are two options to +create loads in VSPERF. These options are used for different use-cases. The options are: + +* stress or stress-ng +* Stressor-VMs + +*stress and stress-ng* are linux tools to stress the system in various ways. +It can stress different subsystems such as CPU and memory. *stress-ng* is the +improvised version of *stress*. StressorVMs are custom build virtual-machines +for the noisy-neighbor use-cases. + +Installation +^^^^^^^^^^^^ + +stress and stress-ng can be installed through standard linux installation process. +Information about stress-ng, including the steps for installing can be found +here: `stress-ng <https://github.com/ColinIanKing/stress-ng>`_ + +There are two options for StressorVMs - one is VMs based on stress-ng and second +is VM based on Spirent's cloudstress. VMs based on stress-ng can be found in this +`link <https://github.com/opensource-tnbt/stressng-images>`_ . Spirent's cloudstress +based VM can be downloaded from this `site <https://github.com/spirent/cloudstress>`_ + +These stressorVMs are of OSV based VMs, which are very small in size. Download +these VMs and place it in appropriate location, and this location will used in +the configuration - as mentioned below. + +Configuration +^^^^^^^^^^^^^ + +The configuration file for loadgens can be found in **conf/07_loadgen.conf**. +There are no specific configurations for stress and stress-ng commands based +load-generation. However, for StressorVMs, following configurations apply: + +* ``NN_COUNT`` - Number of stressor VMs required. +* ``NN_MEMORY`` - Comma separated memory configuration for each VM +* ``NN_SMP`` - Comma separated configuration for each VM +* ``NN_IMAGE`` - Comma separated list of Paths for each VM image +* ``NN_SHARED_DRIVE_TYPE`` - Comma separated list of shaed drive type for each VM +* ``NN_BOOT_DRIVE_TYPE`` - Comma separated list of boot drive type for each VM +* ``NN_CORE_BINDING`` - Comma separated lists of list specifying the cores associated with each VM. +* ``NN_NICS_NR`` - Comma seprated list of number of NICS for each VM +* ``NN_BASE_VNC_PORT`` - Base VNC port Index. +* ``NN_LOG_FILE`` - Name of the log file + +.. _`L3 Cache Management`: + +Last Level Cache Management +--------------------------- + +VSPERF support last-level cache management using Intel's RDT tool(s) - the +relavant ones are `Intel CAT-CMT <https://github.com/intel/intel-cmt-cat>`_ and +`Intel RMD <https://github.com/intel/rmd>`_. RMD is a linux daemon that runs on +individual hosts, and provides a REST API for control/orchestration layer to +request LLC for the VMs/Containers/Applications. RDT receives resource policy +form orchestration layer - in this case, from VSPERF - and enforce it on the host. +It achieves this enforcement via kernel interfaces such as resctrlfs and libpqos. +The resource here refer to the last-level cache. User can configure policies to +define how much of cache a CPU can get. The policy configuration is described below. + +Installation +^^^^^^^^^^^^ + +For installation of RMD tool, please install CAT-CMT first and then install RMD. +The details of installation can be found here: `Intel CAT-CMT <https://github.com/intel/intel-cmt-cat>`_ +and `Intel RMD <https://github.com/intel/rmd>`_ + +Configuration +^^^^^^^^^^^^^ + +The configuration file for cache management can be found in **conf/08_llcmanagement.conf**. + +VSPERF provides following configuration options, for user to define and enforce policies via RMD. + +* ``LLC_ALLOCATION`` - Enable or Disable LLC management. +* ``RMD_PORT`` - RMD port (port number on which API server is listening) +* ``RMD_SERVER_IP`` - IP address where RMD is running. Currently only localhost. +* ``RMD_API_VERSION`` - RMD version. Currently it is 'v1' +* ``POLICY_TYPE`` - Specify how the policy is defined - either COS or CUSTOM +* ``VSWITCH_COS`` - Class of service (CoS for Vswitch. CoS can be gold, silver-bf or bronze-shared. +* ``VNF_COS`` - Class of service for VNF +* ``PMD_COS`` - Class of service for PMD +* ``NOISEVM_COS`` - Class of service of Noisy VM. +* ``VSWITCH_CA`` - [min-cache-value, maxi-cache-value] for vswitch +* ``VNF_CA`` - [min-cache-value, max-cache-value] for VNF +* ``PMD_CA`` - [min-cache-value, max-cache-value] for PMD +* ``NOISEVM_CA`` - [min-cache-value, max-cache-value] for Noisy VM diff --git a/docs/testing/user/configguide/trafficgen.rst b/docs/testing/user/configguide/trafficgen.rst index 33824486..2636626a 100644 --- a/docs/testing/user/configguide/trafficgen.rst +++ b/docs/testing/user/configguide/trafficgen.rst @@ -39,6 +39,7 @@ and is configured as follows: TRAFFIC = { 'traffic_type' : 'rfc2544_throughput', 'frame_rate' : 100, + 'burst_size' : 100, 'bidir' : 'True', # will be passed as string in title format to tgen 'multistream' : 0, 'stream_type' : 'L4', @@ -75,8 +76,22 @@ and is configured as follows: 'count': 1, 'filter': '', }, + 'scapy': { + 'enabled': False, + '0' : 'Ether(src={Ether_src}, dst={Ether_dst})/' + 'Dot1Q(prio={Dot1Q_prio}, id={Dot1Q_id}, vlan={Dot1Q_vlan})/' + 'IP(proto={IP_proto}, src={IP_src}, dst={IP_dst})/' + '{IP_PROTO}(sport={IP_PROTO_sport}, dport={IP_PROTO_dport})', + '1' : 'Ether(src={Ether_dst}, dst={Ether_src})/' + 'Dot1Q(prio={Dot1Q_prio}, id={Dot1Q_id}, vlan={Dot1Q_vlan})/' + 'IP(proto={IP_proto}, src={IP_dst}, dst={IP_src})/' + '{IP_PROTO}(sport={IP_PROTO_dport}, dport={IP_PROTO_sport})', + } } +A detailed description of the ``TRAFFIC`` dictionary can be found at +:ref:`configuration-of-traffic-dictionary`. + The framesize parameter can be overridden from the configuration files by adding the following to your custom configuration file ``10_custom.conf``: @@ -577,7 +592,7 @@ http://www.mono-project.com/docs/getting-started/install/linux/ rpm --import "http://keyserver.ubuntu.com/pks/lookup?op=get&search=0x3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF" yum-config-manager --add-repo http://download.mono-project.com/repo/centos/ - yum -y install mono-complete + yum -y install mono-complete-5.8.0.127-0.xamarin.3.epel7.x86_64 To prevent gpg errors on future yum installation of packages the mono-project repo should be disabled once installed. @@ -667,6 +682,14 @@ or modify the length of the learning by modifying the following settings. TRAFFICGEN_XENA_CONT_PORT_LEARNING_ENABLED = False TRAFFICGEN_XENA_CONT_PORT_LEARNING_DURATION = 3 +Multistream Modifier +~~~~~~~~~~~~~~~~~~~~ + +Xena has a modifier maximum value or 64k in size. For this reason when specifying +Multistream values of greater than 64k for Layer 2 or Layer 3 it will use two +modifiers that may be modified to a value that can be square rooted to create the +two modifiers. You will see a log notification for the new value that was calculated. + MoonGen ------- @@ -745,11 +768,14 @@ You can directly download from GitHub: git clone https://github.com/cisco-system-traffic-generator/trex-core -and use the master branch: +and use the same Trex version for both server and client API. + +**NOTE:** The Trex API version used by VSPERF is defined by variable ``TREX_TAG`` +in file ``src/package-list.mk``. .. code-block:: console - git checkout master + git checkout v2.38 or Trex latest release you can download from here: @@ -795,6 +821,13 @@ It is neccesary for proper connection between Trex server and VSPERF. Firewall must allow a connection from DUT (VSPERF) to the T-Rex server running at TCP port 4501. +**NOTE:** For high speed cards it may be advantageous to start T-Rex with more transmit queues/cores. + +.. code-block:: console + + cd trex-cores/scripts/ + ./t-rex-64 -i -c 10 + For additional information about Trex stateless mode see Trex stateless documentation: https://trex-tgn.cisco.com/trex/doc/trex_stateless.html @@ -847,6 +880,21 @@ place. This can be adjusted with the following configurations: TRAFFICGEN_TREX_LEARNING_MODE=True TRAFFICGEN_TREX_LEARNING_DURATION=5 +Latency measurements have impact on T-Rex performance. Thus vswitchperf uses a separate +latency stream for each direction with limited speed. This workaround is used for RFC2544 +**Throughput** and **Continuous** traffic types. In case of **Burst** traffic type, +the latency statistics are measured for all frames in the burst. Collection of latency +statistics is driven by configuration option ``TRAFFICGEN_TREX_LATENCY_PPS`` as follows: + + * value ``0`` - disables latency measurements + * non zero integer value - enables latency measurements; In case of Throughput + and Continuous traffic types, it specifies a speed of latency specific stream + in PPS. In case of burst traffic type, it enables latency measurements for all frames. + +.. code-block:: console + + TRAFFICGEN_TREX_LATENCY_PPS = 1000 + SR-IOV and Multistream layer 2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ T-Rex by default only accepts packets on the receive side if the destination mac matches the @@ -862,6 +910,22 @@ modified. Enable Promiscuous mode when doing multistream at layer 2 testing with TRAFFICGEN_TREX_PROMISCUOUS=True +Card Bandwidth Options +~~~~~~~~~~~~~~~~~~~~~~ + +T-Rex API will attempt to retrieve the highest possible speed from the card using internal +calls to port information. If you are using two separate cards then it will take the lowest +of the two cards as the max speed. If necessary you can try to force the API to use a +specific maximum speed per port. The below configurations can be adjusted to enable this. + +.. code-block:: console + + TRAFFICGEN_TREX_FORCE_PORT_SPEED = True + TRAFFICGEN_TREX_PORT_SPEED = 40000 # 40 gig + +**Note::** Setting higher than possible speeds will result in unpredictable behavior when running +tests such as duration inaccuracy and/or complete test failure. + RFC2544 Validation ~~~~~~~~~~~~~~~~~~ @@ -881,3 +945,68 @@ The duration and maximum number of attempted verification trials can be set to c behavior of this step. If the verification step fails, it will resume the binary search with new values where the maximum output will be the last attempted frame rate minus the current set thresh hold. + +Scapy frame definition +~~~~~~~~~~~~~~~~~~~~~~ + +It is possible to use a SCAPY frame definition to generate various network protocols +by the **T-Rex** traffic generator. In case that particular network protocol layer +is disabled by the TRAFFIC dictionary (e.g. TRAFFIC['vlan']['enabled'] = False), +then disabled layer will be removed from the scapy format definition by VSPERF. + +The scapy frame definition can refer to values defined by the TRAFFIC dictionary +by following keywords. These keywords are used in next examples. + +* ``Ether_src`` - refers to ``TRAFFIC['l2']['srcmac']`` +* ``Ether_dst`` - refers to ``TRAFFIC['l2']['dstmac']`` +* ``IP_proto`` - refers to ``TRAFFIC['l3']['proto']`` +* ``IP_PROTO`` - refers to upper case version of ``TRAFFIC['l3']['proto']`` +* ``IP_src`` - refers to ``TRAFFIC['l3']['srcip']`` +* ``IP_dst`` - refers to ``TRAFFIC['l3']['dstip']`` +* ``IP_PROTO_sport`` - refers to ``TRAFFIC['l4']['srcport']`` +* ``IP_PROTO_dport`` - refers to ``TRAFFIC['l4']['dstport']`` +* ``Dot1Q_prio`` - refers to ``TRAFFIC['vlan']['priority']`` +* ``Dot1Q_id`` - refers to ``TRAFFIC['vlan']['cfi']`` +* ``Dot1Q_vlan`` - refers to ``TRAFFIC['vlan']['id']`` + +In following examples of SCAPY frame definition only relevant parts of TRAFFIC +dictionary are shown. The rest of the TRAFFIC dictionary is set to default values +as they are defined in ``conf/03_traffic.conf``. + +Please check official documentation of SCAPY project for details about SCAPY frame +definition and supported network layers at: http://www.secdev.org/projects/scapy + +#. Generate ICMP frames: + + .. code-block:: console + + 'scapy': { + 'enabled': True, + '0' : 'Ether(src={Ether_src}, dst={Ether_dst})/IP(proto="icmp", src={IP_src}, dst={IP_dst})/ICMP()', + '1' : 'Ether(src={Ether_dst}, dst={Ether_src})/IP(proto="icmp", src={IP_dst}, dst={IP_src})/ICMP()', + } + +#. Generate IPv6 ICMP Echo Request + + .. code-block:: console + + 'l3' : { + 'srcip': 'feed::01', + 'dstip': 'feed::02', + }, + 'scapy': { + 'enabled': True, + '0' : 'Ether(src={Ether_src}, dst={Ether_dst})/IPv6(src={IP_src}, dst={IP_dst})/ICMPv6EchoRequest()', + '1' : 'Ether(src={Ether_dst}, dst={Ether_src})/IPv6(src={IP_dst}, dst={IP_src})/ICMPv6EchoRequest()', + } + +#. Generate TCP frames: + + Example uses default SCAPY frame definition, which can reflect ``TRAFFIC['l3']['proto']`` settings. + + .. code-block:: console + + 'l3' : { + 'proto' : 'tcp', + }, + diff --git a/docs/testing/user/userguide/integration.rst b/docs/testing/user/userguide/integration.rst index 66808400..9d847fd8 100644 --- a/docs/testing/user/userguide/integration.rst +++ b/docs/testing/user/userguide/integration.rst @@ -1,6 +1,6 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Intel Corporation, AT&T and others. +.. (c) OPNFV, Intel Corporation, AT&T, Tieto and others. .. _integration-tests: @@ -22,6 +22,12 @@ P2P (Physical to Physical scenarios). NOTE: The configuration for overlay tests provided in this guide is for unidirectional traffic only. +NOTE: The overlay tests require an IxNet traffic generator. The tunneled traffic +is configured by ``ixnetrfc2544v2.tcl`` script. This script can be used +with all supported deployment scenarios for generation of frames with VXLAN, GRE +or GENEVE protocols. In that case options "Tunnel Operation" and +"TRAFFICGEN_IXNET_TCL_SCRIPT" must be properly configured at testcase definition. + Executing Integration Tests --------------------------- @@ -63,8 +69,8 @@ the following variables in you user_settings.py file: VTEP_IP2_SUBNET = '192.168.240.0/24' # Bridge names - TUNNEL_INTEGRATION_BRIDGE = 'br0' - TUNNEL_EXTERNAL_BRIDGE = 'br-ext' + TUNNEL_INTEGRATION_BRIDGE = 'vsperf-br0' + TUNNEL_EXTERNAL_BRIDGE = 'vsperf-br-ext' # IP of br-ext TUNNEL_EXTERNAL_BRIDGE_IP = '192.168.240.1/24' diff --git a/docs/testing/user/userguide/testlist.rst b/docs/testing/user/userguide/testlist.rst index 2b0e9d7f..fe8c840a 100644 --- a/docs/testing/user/userguide/testlist.rst +++ b/docs/testing/user/userguide/testlist.rst @@ -68,14 +68,13 @@ vswitch_pvvp_tput vSwitch - configure switch, two chained v vswitch_pvvp_back2back vSwitch - configure switch, two chained vnfs and execute RFC2544 back2back test vswitch_pvvp_cont vSwitch - configure switch, two chained vnfs and execute RFC2544 continuous stream test vswitch_pvvp_all vSwitch - configure switch, two chained vnfs and execute all test types -vswitch_p4vp Just configure 4 chained vnfs -vswitch_p4vp_tput 4 chained vnfs, execute RFC2544 throughput test -vswitch_p4vp_back2back 4 chained vnfs, execute RFC2544 back2back test -vswitch_p4vp_cont 4 chained vnfs, execute RFC2544 continuous stream test -vswitch_p4vp_all 4 chained vnfs, execute RFC2544 throughput test -2pvp_udp_dest_flows RFC2544 Continuous TC with 2 Parallel VMs, flows on UDP Dest Port -4pvp_udp_dest_flows RFC2544 Continuous TC with 4 Parallel VMs, flows on UDP Dest Port -6pvp_udp_dest_flows RFC2544 Continuous TC with 6 Parallel VMs, flows on UDP Dest Port +vswitch_p4vp_tput 4 chained vnfs, execute RFC2544 throughput test, deployment pvvp4 +vswitch_p4vp_back2back 4 chained vnfs, execute RFC2544 back2back test, deployment pvvp4 +vswitch_p4vp_cont 4 chained vnfs, execute RFC2544 continuous stream test, deployment pvvp4 +vswitch_p4vp_all 4 chained vnfs, execute RFC2544 throughput tests, deployment pvvp4 +2pvp_udp_dest_flows RFC2544 Continuous TC with 2 Parallel VMs, flows on UDP Dest Port, deployment pvpv2 +4pvp_udp_dest_flows RFC2544 Continuous TC with 4 Parallel VMs, flows on UDP Dest Port, deployment pvpv4 +6pvp_udp_dest_flows RFC2544 Continuous TC with 6 Parallel VMs, flows on UDP Dest Port, deployment pvpv6 vhost_numa_awareness vSwitch DPDK - verify that PMD threads are served by the same NUMA slot as QEMU instances ixnet_pvp_tput_1nic PVP Scenario with 1 port towards IXIA vswitch_vports_add_del_connection_vpp VPP: vSwitch - configure switch with vports, add and delete connection @@ -388,3 +387,46 @@ ovsdpdk_qos_p2p In a p2p setup, ensure when a QoS egres ovsdpdk_qos_pvp In a pvp setup, ensure when a QoS egress policer is created that the traffic is limited to the specified rate. ======================================== ====================================================================================== + +Custom Statistics ++++++++++++++++++ + +A set of functional testcases for validation of Custom Statistics support by OVS. +This feature allows Custom Statistics to be accessed by VSPERF. + +These testcases require DPDK v17.11, the latest Open vSwitch(v2.9.90) +and the IxNet traffic-generator. + +======================================== ====================================================================================== +ovsdpdk_custstat_check Test if custom statistics are supported. +ovsdpdk_custstat_rx_error Test bad ethernet CRC counter 'rx_crc_errors' exposed by custom + statistics. + +======================================== ====================================================================================== + +T-Rex in VM TestCases +^^^^^^^^^^^^^^^^^^^^^ + +A set of functional testcases, which use T-Rex running in VM as a traffic generator. +These testcases require a VM image with T-Rex server installed. An example of such +image is a vloop-vnf image with T-Rex available for download at: + +http://artifacts.opnfv.org/vswitchperf/vnf/vloop-vnf-ubuntu-16.04_trex_20180209.qcow2 + +This image can be used for both T-Rex VM and loopback VM in ``vm2vm`` testcases. + +**NOTE:** The performance of T-Rex running inside the VM is lower if compared to T-Rex +execution on bare-metal. The user should perform a calibration of the VM maximum FPS +capability, to ensure this limitation is understood. + +======================================== ====================================================================================== +trex_vm_cont T-Rex VM - execute RFC2544 Continuous Stream from T-Rex VM and loop + it back through Open vSwitch. +trex_vm_tput T-Rex VM - execute RFC2544 Throughput from T-Rex VM and loop it back + through Open vSwitch. +trex_vm2vm_cont T-Rex VM2VM - execute RFC2544 Continuous Stream from T-Rex VM and + loop it back through 2nd VM. +trex_vm2vm_tput T-Rex VM2VM - execute RFC2544 Throughput from T-Rex VM and loop it back + through 2nd VM. + +======================================== ====================================================================================== diff --git a/docs/testing/user/userguide/teststeps.rst b/docs/testing/user/userguide/teststeps.rst index 08c95311..cb627bc5 100644 --- a/docs/testing/user/userguide/teststeps.rst +++ b/docs/testing/user/userguide/teststeps.rst @@ -23,6 +23,13 @@ the step number by one which is indicated in the log. (testcases.integration) - Step 0 'vswitch add_vport ['br0']' start +Test steps are defined as a list of steps within a ``TestSteps`` item of test +case definition. Each step is a list with following structure: + +.. code-block:: python + + '[' [ optional-alias ',' ] test-object ',' test-function [ ',' optional-function-params ] '],' + Step driven tests can be used for both performance and integration testing. In case of integration test, each step in the test case is validated. If a step does not pass validation the test will fail and terminate. The test will continue @@ -57,8 +64,14 @@ Step driven testcases can be used in two different ways: Test objects and their functions -------------------------------- -Every test step can call a function of one of the supported test objects. The list -of supported objects and their most common functions follows: +Every test step can call a function of one of the supported test objects. In general +any existing function of supported test object can be called by test step. In case +that step validation is required (valid for integration test steps, which are not +suppressed), then appropriate ``validate_`` method must be implemented. + +The list of supported objects and their most common functions is listed below. Please +check implementation of test objects for full list of implemented functions and their +parameters. * ``vswitch`` - provides functions for vSwitch configuration @@ -176,6 +189,8 @@ of supported objects and their most common functions follows: * ``getValue param`` - returns value of given ``param`` * ``setValue param value`` - sets value of ``param`` to given ``value`` + * ``resetValue param`` - if ``param`` was overridden by ``TEST_PARAMS`` (e.g. by "Parameters" + section of the test case definition), then it will be set to its original value. Examples: @@ -185,6 +200,8 @@ of supported objects and their most common functions follows: ['settings', 'setValue', 'GUEST_USERNAME', ['root']] + ['settings', 'resetValue', 'WHITELIST_NICS'], + It is possible and more convenient to access any VSPERF configuration option directly via ``$NAME`` notation. Option evaluation is done during runtime and vsperf will automatically translate it to the appropriate call of ``settings.getValue``. @@ -747,6 +764,8 @@ destination UDP port. ] }, +The same test can be written in a shorter form using "Deployment" : "pvpv". + To run the test: .. code-block:: console @@ -779,20 +798,20 @@ and available in both csv and rst report files. }, }, "TestSteps": [ - ['vswitch', 'add_vport', 'br0'], - ['vswitch', 'add_vport', 'br0'], + ['vswitch', 'add_vport', '$VSWITCH_BRIDGE_NAME'], + ['vswitch', 'add_vport', '$VSWITCH_BRIDGE_NAME'], # priority must be higher than default 32768, otherwise flows won't match - ['vswitch', 'add_flow', 'br0', + ['vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '1', 'actions': ['output:#STEP[-2][1]'], 'idle_timeout': '0', 'dl_type':'0x0800', 'nw_proto':'17', 'tp_dst':'0', 'priority': '33000'}], - ['vswitch', 'add_flow', 'br0', + ['vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '2', 'actions': ['output:#STEP[-2][1]'], 'idle_timeout': '0', 'dl_type':'0x0800', 'nw_proto':'17', 'tp_dst':'0', 'priority': '33000'}], - ['vswitch', 'add_flow', 'br0', {'in_port': '#STEP[-4][1]', 'actions': ['output:1'], + ['vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '#STEP[-4][1]', 'actions': ['output:1'], 'idle_timeout': '0'}], - ['vswitch', 'add_flow', 'br0', {'in_port': '#STEP[-4][1]', 'actions': ['output:2'], + ['vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '#STEP[-4][1]', 'actions': ['output:2'], 'idle_timeout': '0'}], - ['vswitch', 'dump_flows', 'br0'], + ['vswitch', 'dump_flows', '$VSWITCH_BRIDGE_NAME'], ['vnf1', 'start'], ] }, diff --git a/docs/testing/user/userguide/testusage.rst b/docs/testing/user/userguide/testusage.rst index 20c30a40..c7cc1484 100644 --- a/docs/testing/user/userguide/testusage.rst +++ b/docs/testing/user/userguide/testusage.rst @@ -91,55 +91,41 @@ Using a custom settings file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If your ``10_custom.conf`` doesn't reside in the ``./conf`` directory -of if you want to use an alternative configuration file, the file can +or if you want to use an alternative configuration file, the file can be passed to ``vsperf`` via the ``--conf-file`` argument. .. code-block:: console $ ./vsperf --conf-file <path_to_custom_conf> ... -Note that configuration passed in via the environment (``--load-env``) -or via another command line argument will override both the default and -your custom configuration files. This "priority hierarchy" can be -described like so (1 = max priority): - -1. Testcase definition section ``Parameters`` -2. Command line arguments -3. Environment variables -4. Configuration file(s) - -Further details about configuration files evaluation and special behaviour +Evaluation of configuration parameters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The value of configuration parameter can be specified at various places, +e.g. at the test case definition, inside configuration files, by the command +line argument, etc. Thus it is important to understand the order of configuration +parameter evaluation. This "priority hierarchy" can be described like so +(1 = max priority): + +1. Testcase definition keywords ``vSwitch``, ``Trafficgen``, ``VNF`` and ``Tunnel Type`` +2. Parameters inside testcase definition section ``Parameters`` +3. Command line arguments (e.g. ``--test-params``, ``--vswitch``, ``--trafficgen``, etc.) +4. Environment variables (see ``--load-env`` argument) +5. Custom configuration file specified via ``--conf-file`` argument +6. Standard configuration files, where higher prefix number means higher + priority. + +For example, if the same configuration parameter is defined in custom configuration +file (specified via ``--conf-file`` argument), via ``--test-params`` argument +and also inside ``Parameters`` section of the testcase definition, then parameter +value from the ``Parameters`` section will be used. + +Further details about order of configuration files evaluation and special behaviour of options with ``GUEST_`` prefix could be found at :ref:`design document <design-configuration>`. .. _overriding-parameters-documentation: -Referencing parameter values -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -It is possible to use a special macro ``#PARAM()`` to refer to the value of -another configuration parameter. This reference is evaluated during -access of the parameter value (by ``settings.getValue()`` call), so it -can refer to parameters created during VSPERF runtime, e.g. NICS dictionary. -It can be used to reflect DUT HW details in the testcase definition. - -Example: - -.. code:: python - - { - ... - "Name": "testcase", - "Parameters" : { - "TRAFFIC" : { - 'l2': { - # set destination MAC to the MAC of the first - # interface from WHITELIST_NICS list - 'dstmac' : '#PARAM(NICS[0]["mac"])', - }, - }, - ... - Overriding values defined in configuration files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -155,6 +141,17 @@ Example: $ ./vsperf --test-params "TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,);" \ "GUEST_LOOPBACK=['testpmd','l2fwd']" pvvp_tput +The ``--test-params`` command line argument can also be used to override default +configuration values for multiple tests. Providing a list of parameters will apply each +element of the list to the test with the same index. If more tests are run than +parameters provided the last element of the list will repeat. + +.. code:: console + + $ ./vsperf --test-params "['TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)'," + "'TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(64,)']" \ + pvvp_tput pvvp_tput + The second option is to override configuration items by ``Parameters`` section of the test case definition. The configuration items can be added into ``Parameters`` dictionary with their new values. These values will override values defined in @@ -186,6 +183,36 @@ parameter name is passed via ``--test-params`` CLI argument or defined in ``Para section of test case definition. It is also forbidden to redefine a value of ``TEST_PARAMS`` configuration item via CLI or ``Parameters`` section. +**NOTE:** The new definition of the dictionary parameter, specified via ``--test-params`` +or inside ``Parameters`` section, will not override original dictionary values. Instead +the original dictionary will be updated with values from the new dictionary definition. + +Referencing parameter values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It is possible to use a special macro ``#PARAM()`` to refer to the value of +another configuration parameter. This reference is evaluated during +access of the parameter value (by ``settings.getValue()`` call), so it +can refer to parameters created during VSPERF runtime, e.g. NICS dictionary. +It can be used to reflect DUT HW details in the testcase definition. + +Example: + +.. code:: python + + { + ... + "Name": "testcase", + "Parameters" : { + "TRAFFIC" : { + 'l2': { + # set destination MAC to the MAC of the first + # interface from WHITELIST_NICS list + 'dstmac' : '#PARAM(NICS[0]["mac"])', + }, + }, + ... + vloop_vnf ^^^^^^^^^ @@ -205,6 +232,12 @@ A Kernel Module that provides OSI Layer 2 Ipv4 termination or forwarding with support for Destination Network Address Translation (DNAT) for both the MAC and IP addresses. l2fwd can be found in <vswitchperf_dir>/src/l2fwd +Additional Tools Setup +^^^^^^^^^^^^^^^^^^^^^^ + +Follow the `Additional tools instructions <additional-tools-configuration>` to +install and configure additional tools such as collectors and loadgens. + Executing tests ^^^^^^^^^^^^^^^ @@ -234,6 +267,12 @@ To run a single test: Where $TESTNAME is the name of the vsperf test you would like to run. +To run a test multiple times, repeat it: + +.. code-block:: console + + $ ./vsperf $TESTNAME $TESTNAME $TESTNAME + To run a group of tests, for example all tests with a name containing 'RFC2544': @@ -256,6 +295,30 @@ Some tests allow for configurable parameters, including test duration --tests RFC2544Tput \ --test-params "TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)" +To specify configurable parameters for multiple tests, use a list of +parameters. One element for each test. + +.. code:: console + + $ ./vsperf --conf-file user_settings.py \ + --test-params "['TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)',"\ + "'TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(64,)']" \ + phy2phy_cont phy2phy_cont + +If the ``CUMULATIVE_PARAMS`` setting is set to True and there are different parameters +provided for each test using ``--test-params``, each test will take the parameters of +the previous test before appyling it's own. +With ``CUMULATIVE_PARAMS`` set to True the following command will be equivalent to the +previous example: + +.. code:: console + + $ ./vsperf --conf-file user_settings.py \ + --test-params "['TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)',"\ + "'TRAFFICGEN_PKT_SIZES=(64,)']" \ + phy2phy_cont phy2phy_cont + " + For all available options, check out the help dialog: .. code-block:: console @@ -584,7 +647,7 @@ The supported dpdk guest bind drivers are: .. code-block:: console - 'uio_pci_generic' - Use uio_pci_generic driver + 'uio_pci_generic' - Use uio_pci_generic driver 'igb_uio_from_src' - Build and use the igb_uio driver from the dpdk src files 'vfio_no_iommu' - Use vfio with no iommu option. This requires custom @@ -915,6 +978,39 @@ Example of execution of VSPERF in "trafficgen" mode: $ ./vsperf -m trafficgen --trafficgen IxNet --conf-file vsperf.conf \ --test-params "TRAFFIC={'traffic_type':'rfc2544_continuous','bidir':'False','framerate':60}" +Performance Matrix +^^^^^^^^^^^^^^^^^^ + +The ``--matrix`` command line argument analyses and displays the performance of +all the tests run. Using the metric specified by ``MATRIX_METRIC`` in the conf-file, +the first test is set as the baseline and all the other tests are compared to it. +The ``MATRIX_METRIC`` must always refer to a numeric value to enable comparision. +A table, with the test ID, metric value, the change of the metric in %, testname +and the test parameters used for each test, is printed out as well as saved into the +results directory. + +Example of 2 tests being compared using Performance Matrix: + +.. code-block:: console + + $ ./vsperf --conf-file user_settings.py \ + --test-params "['TRAFFICGEN_PKT_SIZES=(64,)',"\ + "'TRAFFICGEN_PKT_SIZES=(128,)']" \ + phy2phy_cont phy2phy_cont --matrix + +Example output: + +.. code-block:: console + + +------+--------------+---------------------+----------+---------------------------------------+ + | ID | Name | throughput_rx_fps | Change | Parameters, CUMULATIVE_PARAMS = False | + +======+==============+=====================+==========+=======================================+ + | 0 | phy2phy_cont | 23749000.000 | 0 | 'TRAFFICGEN_PKT_SIZES': [64] | + +------+--------------+---------------------+----------+---------------------------------------+ + | 1 | phy2phy_cont | 16850500.000 | -29.048 | 'TRAFFICGEN_PKT_SIZES': [128] | + +------+--------------+---------------------+----------+---------------------------------------+ + + Code change verification by pylint ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/testing/user/userguide/trafficcapture.rst b/docs/testing/user/userguide/trafficcapture.rst index fa09bfed..8a224dcb 100644 --- a/docs/testing/user/userguide/trafficcapture.rst +++ b/docs/testing/user/userguide/trafficcapture.rst @@ -92,9 +92,9 @@ An example of Traffic Capture in VM test: }, TestSteps: [ # replace original flows with vlan ID modification - ['!vswitch', 'add_flow', 'br0', {'in_port': '1', 'actions': ['mod_vlan_vid:4','output:3']}], - ['!vswitch', 'add_flow', 'br0', {'in_port': '2', 'actions': ['mod_vlan_vid:4','output:4']}], - ['vswitch', 'dump_flows', 'br0'], + ['!vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '1', 'actions': ['mod_vlan_vid:4','output:3']}], + ['!vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '2', 'actions': ['mod_vlan_vid:4','output:4']}], + ['vswitch', 'dump_flows', '$VSWITCH_BRIDGE_NAME'], # verify that received frames have modified vlan ID ['VNF0', 'execute_and_wait', 'tcpdump -i eth0 -c 5 -w dump.pcap vlan 4 &'], ['trafficgen', 'send_traffic',{}], @@ -199,7 +199,7 @@ An example of Traffic Capture for testing NICs with HW offloading test: ['tools', 'exec_shell_background', 'tcpdump -i [2][device] -c 5 -w capture.pcap ' 'ether src [l2][srcmac]'], ['trafficgen', 'send_traffic', {}], - ['vswitch', 'dump_flows', 'br0'], + ['vswitch', 'dump_flows', '$VSWITCH_BRIDGE_NAME'], ['vswitch', 'dump_flows', 'br1'], # there must be 5 captured frames... ['tools', 'exec_shell', 'tcpdump -r capture.pcap | wc -l', '|^(\d+)$'], |