diff options
Diffstat (limited to 'docs')
20 files changed, 791 insertions, 172 deletions
diff --git a/docs/release/release-notes/release-notes.rst b/docs/release/release-notes/release-notes.rst index 4b3f12bcf..914daa3a4 100644 --- a/docs/release/release-notes/release-notes.rst +++ b/docs/release/release-notes/release-notes.rst @@ -33,6 +33,9 @@ Version History | November 9, 2018 | 7.0.0 | Yardstick for Gambia release | | | | | +-------------------+-----------+---------------------------------+ +| December 14, 2018 | 7.1.0 | Yardstick for Gambia release | +| | | | ++-------------------+-----------+---------------------------------+ Important Notes @@ -111,19 +114,19 @@ Release Data | **Project** | Yardstick | | | | +--------------------------------+-----------------------+ -| **Repo/tag** | yardstick/opnfv-7.0.0 | +| **Repo/tag** | yardstick/opnfv-7.1.0 | | | | +--------------------------------+-----------------------+ -| **Yardstick Docker image tag** | opnfv-7.0.0 | +| **Yardstick Docker image tag** | opnfv-7.1.0 | | | | +--------------------------------+-----------------------+ -| **Release designation** | Gambia 7.0 | +| **Release designation** | Gambia 7.1 | | | | +--------------------------------+-----------------------+ -| **Release date** | November 9, 2018 | +| **Release date** | December 14, 2018 | | | | +--------------------------------+-----------------------+ -| **Purpose of the delivery** | OPNFV Gambia 7.0.0 | +| **Purpose of the delivery** | OPNFV Gambia 7.1.0 | | | | +--------------------------------+-----------------------+ @@ -269,7 +272,7 @@ List of Scenarios New Test cases -------------- -.. note:: Yardstick Gambia 7.0.0 adds no new test cases. +.. note:: Yardstick Gambia 7.1.0 adds no new test cases. * Generic NFVI test cases @@ -326,7 +329,7 @@ Feature additions Scenario Matrix =============== -For Gambia 7.0.0, Yardstick was tested on the following scenarios: +For Gambia 7.1.0, Yardstick was tested on the following scenarios: +-------------------------+------+---------+----------+------+ | Scenario | Apex | Compass | Fuel-arm | Fuel | @@ -370,58 +373,35 @@ Known Issues/Faults Corrected Faults ---------------- -Gambia 7.0.0: +Gambia 7.1.0: +--------------------+--------------------------------------------------------------------------+ | **JIRA REFERENCE** | **DESCRIPTION** | +====================+==========================================================================+ -| YARDSTICK-1137 | Fix CLI argument handling in nsb_setup.sh | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1220 | Get stats for multiple port simultaneously | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1260 | Added missing functionality to start VM and access it using SSH keys | -| | in Standalone contexts. | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1298 | Allows for in-line overriding/modification of traffic profile variables | -| | from the testcase file. | +| YARDSTICK-1241 | Update NSB PROX devguide. | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1368 | Updated existing test cases in Yardstick to minimize changes done | -| | manually to run standalone tests for Trex. | +| YARDSTICK-1458 | NSB NFVi PROX Should report realtime port activity not historical data. | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1389 | Add status filed for RFC2544 TC iterations | +| YARDSTICK-1471 | Add Testcase Prox Standalone SRIOV. | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1395 | Update 'configure_uwsgi' role to work in baremetal/container modes. | +| YARDSTICK-1475 | Adding Testcase for Prox Stanalone OvS-DPDK. | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1402 | Change IP assignment for VM to static for standalone context | +| YARDSTICK-1500 | Adding Testcase for Prox L2FWD PktTouch Stanalone OvS-DPDK. | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1404 | CPU Utilization for VNF and traffic generator are now graphed on Grafana | +| YARDSTICK-1517 | Missing opnfv "os-ovn-nofeature-ha" scenario test suite. | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1411 | Fix Yardstick Docker image ARM build | +| YARDSTICK-l526 | Run testcase 074 result overridden by job status. | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1414 | Update the pinned sampleVNF version to use a commit instead of a branch | +| YARDSTICK-1547 | Adding scale up test case for l3fwd OvS-DPDK. | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1418 | NSB PROX NFVi test now stops after reaching expected precision | +| YARDSTICK-1560 | Fix pip environment. | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1457 | Fix influxdb "field type conflict" error | +| YARDSTICK-1561 | L3FWD Gradana Dashboards Out-of-date and incorrect. | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1458 | Update Grafana to display "real-time" data instead of historical data. | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1462 | NSB: Add OvS 2.8.1 support in SA context | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1492 | Make OvS server to listen on TCP | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1493 | The RX queues number is hard-codded and cannot be changed | -+--------------------+--------------------------------------------------------------------------+ - -Gambia 7.0.0 known restrictions/issues +Gambia 7.1.0 known restrictions/issues ====================================== -+-----------+-----------------------+------------------+ -| Installer | Scenario | Issue | -+===========+=======================+==================+ -| apex | os-ovn-nofeature-ha | YARDSTICK-1517 | -+-----------+-----------------------+------------------+ Useful links ============ diff --git a/docs/testing/developer/devguide/devguide_nsb_prox.rst b/docs/testing/developer/devguide/devguide_nsb_prox.rst index be2b5be61..44a216b75 100755 --- a/docs/testing/developer/devguide/devguide_nsb_prox.rst +++ b/docs/testing/developer/devguide/devguide_nsb_prox.rst @@ -8,6 +8,8 @@ for Telco customers, investigate the impact of new Intel compute, network and storage technologies, characterize performance, and develop optimal system architectures and configurations. +NSB PROX Supports Baremetal, Openstack and standalone configuration. + .. contents:: Prerequisites @@ -162,11 +164,11 @@ A NSB Prox test is composed of the following components :- server descriptions, in the case of baremetal the hardware description location. It also contains the name of the Traffic Generator, the SUT config file and the traffic profile description, all described below. - See nsb-test-description-label_ + See `Test Description File`_ * Traffic Profile file. Example ``prox_binsearch.yaml``. This describes the packet size, tolerated loss, initial line rate to start traffic at, test - interval etc See nsb-traffic-profile-label_ + interval etc See `Traffic Profile File`_ * Traffic Generator Config file. Usually called ``gen_<test>-<ports>.cfg``. @@ -179,7 +181,7 @@ A NSB Prox test is composed of the following components :- under test. Example traffic generator config file ``gen_l2fwd-4.cfg`` - See nsb-traffic-generator-label_ + See `Traffic Generator Config file`_ * SUT Config file. Usually called ``handle_<test>-<ports>.cfg``. @@ -193,7 +195,7 @@ A NSB Prox test is composed of the following components :- the ports to the Traffic Verifier tasks of the Traffic Generator. Example traffic generator config file ``handle_l2fwd-4.cfg`` - See nsb-sut-generator-label_ + See `SUT Config File`_ * NSB PROX Baremetal Configuration file. Usually called ``prox-baremetal-<ports>.yaml`` @@ -202,12 +204,12 @@ A NSB Prox test is composed of the following components :- This is required for baremetal only. This describes hardware, NICs, IP addresses, Network drivers, usernames and passwords. - See baremetal-config-label_ + See `Baremetal Configuration File`_ * Grafana Dashboard. Usually called ``Prox_<context>_<test>-<port>-<DateAndTime>.json`` where - * <context> Is either ``BM`` or ``heat`` + * <context> Is ``BM``,``heat``,``ovs_dpdk`` or ``sriov`` * <test> Is the a one or 2 word description of the test. * <port> is the number of ports used express as ``2Port`` or ``4Port`` * <DateAndTime> is the Date and Time expressed as a string. @@ -217,15 +219,15 @@ A NSB Prox test is composed of the following components :- Other files may be required. These are test specific files and will be covered later. -.. _nsb-test-description-label: -**Test Description File** +Test Description File +--------------------- -Here we will discuss the test description for both -baremetal and openstack. +Here we will discuss the test description for +baremetal, openstack and standalone. -*Test Description File for Baremetal* -------------------------------------- +Test Description File for Baremetal +----------------------------------- This section will introduce the meaning of the Test case description file. We will use ``tc_prox_baremetal_l2fwd-2.yaml`` as an example to @@ -239,7 +241,7 @@ Now let's examine the components of the file in detail 1. ``traffic_profile`` - This specifies the traffic profile for the test. In this case ``prox_binsearch.yaml`` is used. See - nsb-traffic-profile-label_ + `Traffic Profile File`_ 2. ``topology`` - This is either ``prox-tg-topology-1.yaml`` or ``prox-tg-topology-2.yaml`` or ``prox-tg-topology-4.yaml`` @@ -251,10 +253,13 @@ Now let's examine the components of the file in detail 4. ``interface_speed_gbps`` - This is an optional parameter. If not present the system defaults to 10Gbps. This defines the speed of the interfaces. -5. ``prox_path`` - Location of the Prox executable on the traffic +5. ``collectd`` - (Optional) This specifies we want to collect NFVI statistics + like CPU Utilization, + +6. ``prox_path`` - Location of the Prox executable on the traffic generator (Either baremetal or Openstack Virtual Machine) -6. ``prox_config`` - This is the ``SUT Config File``. +7. ``prox_config`` - This is the ``SUT Config File``. In this case it is ``handle_l2fwd-2.cfg`` A number of additional parameters can be added. This example @@ -263,6 +268,14 @@ Now let's examine the components of the file in detail options: interface_speed_gbps: 10 + traffic_config: + tolerated_loss: 0.01 + test_precision: 0.01 + packet_sizes: [64] + duration: 30 + lower_bound: 0.0 + upper_bound: 100.0 + vnf__0: prox_path: /opt/nsb_bin/prox prox_config: ``configs/handle_vpe-4.cfg`` @@ -276,55 +289,60 @@ Now let's examine the components of the file in detail ``configs/vpe_rules.lua`` : ```` prox_generate_parameter: True - ``interface_speed_gbps`` - this specifies the speed of the interface - in Gigabits Per Second. This is used to calculate pps(packets per second). - If the interfaces are of different speeds, then this specifies the speed - of the slowest interface. This parameter is optional. If omitted the - interface speed defaults to 10Gbps. + ``interface_speed_gbps`` - this specifies the speed of the interface + in Gigabits Per Second. This is used to calculate pps(packets per second). + If the interfaces are of different speeds, then this specifies the speed + of the slowest interface. This parameter is optional. If omitted the + interface speed defaults to 10Gbps. - ``prox_files`` - this specified that a number of addition files - need to be provided for the test to run correctly. This files - could provide routing information,hashing information or a - hashing algorithm and ip/mac information. + ``traffic_config`` - This allows the values here to override the values in + in the traffic_profile file. e.g. "prox_binsearch.yaml". Values provided + here override values provided in the "traffic_profile" section of the + traffic_profile file. Some, all or none of the values can be provided here. - ``prox_generate_parameter`` - this specifies that the NSB application - is required to provide information to the nsb Prox in the form - of a file called ``parameters.lua``, which contains information - retrieved from either the hardware or the openstack configuration. + The values describes the packet size, tolerated loss, initial line rate + to start traffic at, test interval etc See `Traffic Profile File`_ -7. ``prox_args`` - this specifies the command line arguments to start - prox. See `prox command line`_. + ``prox_files`` - this specified that a number of addition files + need to be provided for the test to run correctly. This files + could provide routing information,hashing information or a + hashing algorithm and ip/mac information. + + ``prox_generate_parameter`` - this specifies that the NSB application + is required to provide information to the nsb Prox in the form + of a file called ``parameters.lua``, which contains information + retrieved from either the hardware or the openstack configuration. -8. ``prox_config`` - This specifies the Traffic Generator config file. +8. ``prox_args`` - this specifies the command line arguments to start + prox. See `prox command line`_. -9. ``runner`` - This is set to ``ProxDuration`` - This specifies that the - test runs for a set duration. Other runner types are available - but it is recommend to use ``ProxDuration`` +9. ``prox_config`` - This specifies the Traffic Generator config file. - The following parrameters are supported +10. ``runner`` - This is set to ``ProxDuration`` - This specifies that the + test runs for a set duration. Other runner types are available + but it is recommend to use ``ProxDuration``. The following parameters + are supported - ``interval`` - (optional) - This specifies the sampling interval. - Default is 1 sec + ``interval`` - (optional) - This specifies the sampling interval. + Default is 1 sec - ``sampled`` - (optional) - This specifies if sampling information is - required. Default ``no`` + ``sampled`` - (optional) - This specifies if sampling information is + required. Default ``no`` - ``duration`` - This is the length of the test in seconds. Default - is 60 seconds. + ``duration`` - This is the length of the test in seconds. Default + is 60 seconds. - ``confirmation`` - This specifies the number of confirmation retests to - be made before deciding to increase or decrease line speed. Default 0. + ``confirmation`` - This specifies the number of confirmation retests to + be made before deciding to increase or decrease line speed. Default 0. -10. ``context`` - This is ``context`` for a 2 port Baremetal configuration. +11. ``context`` - This is ``context`` for a 2 port Baremetal configuration. If a 4 port configuration was required then file ``prox-baremetal-4.yaml`` would be used. This is the NSB Prox baremetal configuration file. -.. _nsb-traffic-profile-label: - -*Traffic Profile file* ----------------------- +Traffic Profile File +-------------------- This describes the details of the traffic flow. In this case ``prox_binsearch.yaml`` is used. @@ -334,8 +352,9 @@ This describes the details of the traffic flow. In this case :alt: NSB PROX Traffic Profile -1. ``name`` - The name of the traffic profile. This name should match the name - specified in the ``traffic_profile`` field in the Test Description File. +1. ``name`` - The name of the traffic profile. This name should match the + name specified in the ``traffic_profile`` field in the Test + Description File. 2. ``traffic_type`` - This specifies the type of traffic pattern generated, This name matches class name of the traffic generator. See:: @@ -396,19 +415,20 @@ See this prox_vpe.yaml as example:: lower_bound: 0.0 upper_bound: 100.0 -*Test Description File for Openstack* -------------------------------------- +Test Description File for Openstack +----------------------------------- We will use ``tc_prox_heat_context_l2fwd-2.yaml`` as a example to show you how to understand the test description file. -.. image:: images/PROX_Test_HEAT_Script1.png - :width: 800px - :alt: NSB PROX Test Description File - Part 1 + .. image:: images/PROX_Test_HEAT_Script1.png + :width: 800px + :alt: NSB PROX Test Description File - Part 1 -.. image:: images/PROX_Test_HEAT_Script2.png - :width: 800px - :alt: NSB PROX Test Description File - Part 2 + + .. image:: images/PROX_Test_HEAT_Script2.png + :width: 800px + :alt: NSB PROX Test Description File - Part 2 Now lets examine the components of the file in detail @@ -464,10 +484,64 @@ F. ``networks`` - is composed of a management network labeled ``mgmt`` port_security_enabled: False enable_dhcp: 'false' -.. _nsb-traffic-generator-label: +Test Description File for Standalone +------------------------------------ + +We will use ``tc_prox_ovs-dpdk_l2fwd-2.yaml`` as a example to show +you how to understand the test description file. + + .. image:: images/PROX_Test_ovs_dpdk_Script_1.png + :width: 800px + :alt: NSB PROX Test Standalone Description File - Part 1 + + .. image:: images/PROX_Test_ovs_dpdk_Script_2.png + :width: 800px + :alt: NSB PROX Test Standalone Description File - Part 2 + +Now lets examine the components of the file in detail + +Sections 1 to 9 are exactly the same in Baremetal and in Heat. Section +``10`` is replaced with sections A to F. Section 10 was for a baremetal +configuration file. This has no place in a heat configuration. + +A. ``file`` - Pod file for Baremetal Traffic Generator configuration: + IP Address, User/Password & Interfaces + +B. ``type`` - This defines the type of standalone configuration. + Possible values are ``StandaloneOvsDpdk`` or ``StandaloneSriov`` + +C. ``file`` - Pod file for Standalone host configuration: + IP Address, User/Password & Interfaces + +D. ``vm_deploy`` - Deploy a new VM or use an existing VM -*Traffic Generator Config file* -------------------------------- +E. ``ovs_properties`` - OVS Version, DPDK Version and configuration + to use. + +F. ``flavor``- NSB image generated when installing NSB using ansible-playbook:: + + ram- Configurable RAM for SUT VM + extra_specs + hw:cpu_sockets - Configurable number of Sockets for SUT VM + hw:cpu_cores - Configurable number of Cores for SUT VM + hw:cpu_threads- Configurable number of Threads for SUT VM + +G. ``mgmt`` - Management port of the SUT VM. Preconfig needed on TG & SUT host machines. + is the system under test. + + +H. ``xe0`` - Upline Network port + +I. ``xe1`` - Downline Network port + +J. ``uplink_0`` - Uplink Phy port of the NIC on the host. This will be used to create + the Virtual Functions. + +K. ``downlink_0`` - Downlink Phy port of the NIC on the host. This will be used to + create the Virtual Functions. + +Traffic Generator Config file +----------------------------- This section will describe the traffic generator config file. This is the same for both baremetal and heat. See this example @@ -725,10 +799,8 @@ Now let's examine the components of the file in detail to be read from. Note that the SUT workload might cause the position of the timestamp to change (i.e. due to encapsulation). -.. _nsb-sut-generator-label: - -*SUT Config file* -------------------------------- +SUT Config File +--------------- This section will describes the SUT(VNF) config file. This is the same for both baremetal and heat. See this example of ``handle_l2fwd_multiflow-2.cfg`` to @@ -892,10 +964,8 @@ Now let's examine the components of the file in detail :width: 1000px :alt: NSB PROX Config File for BNG_QOS -*Baremetal Configuration file* ------------------------------- - -.. _baremetal-config-label: +Baremetal Configuration File +---------------------------- This is required for baremetal testing. It describes the IP address of the various ports, the Network devices drivers and MAC addresses and the network @@ -933,8 +1003,8 @@ Now let's describe the sections of the file. 9. ``routing_table`` - All parameters should remain unchanged. 10. ``nd_route_tbl`` - All parameters should remain unchanged. -*Grafana Dashboard* -------------------- +Grafana Dashboard +----------------- The grafana dashboard visually displays the results of the tests. The steps required to produce a grafana dashboard are described here. @@ -987,7 +1057,7 @@ In order to run the NSB PROX test. cd /home/opnfv/repos/yardstick/samples/vnf_samples/nsut/prox b. Install prox-baremetal-2.yam and prox-baremetal-4.yaml for that - topology into this directory as per baremetal-config-label_ + topology into this directory as per `Baremetal Configuration File`_ c. Install and configure ``yardstick.conf`` :: @@ -1032,14 +1102,12 @@ Frequently Asked Questions Here is a list of frequently asked questions. -*NSB Prox does not work on Baremetal, How do I resolve this?* -------------------------------------------------------------- +NSB Prox does not work on Baremetal, How do I resolve this? +----------------------------------------------------------- If PROX NSB does not work on baremetal, problem is either in network configuration or test file. -*Solution* - 1. Verify network configuration. Execute existing baremetal test.:: yardstick --debug task start ./tc_prox_baremetal_l2fwd-4.yaml @@ -1079,10 +1147,8 @@ configuration or test file. 2. If existing baremetal works then issue is with your test. Check the traffic generator gen_<test>-<ports>.cfg to ensure it is producing a valid packet. -*How do I debug NSB Prox on Baremetal?* ---------------------------------------- - -*Solution* +How do I debug NSB Prox on Baremetal? +------------------------------------- 1. Execute the test as follows:: @@ -1148,8 +1214,8 @@ configuration or test file. dump_tx 1 0 1 -*NSB Prox works on Baremetal but not in Openstack. How do I resolve this?* --------------------------------------------------------------------------- +NSB Prox works on Baremetal but not in Openstack. How do I resolve this? +------------------------------------------------------------------------ NSB Prox on Baremetal is a lot more forgiving than NSB Prox on Openstack. A badly formed packed may still work with PROX on Baremetal. However on @@ -1157,7 +1223,6 @@ Openstack the packet must be correct and all fields of the header correct. E.g. A packet with an invalid Protocol ID would still work in Baremetal but this packet would be rejected by openstack. -*Solution* 1. Check the validity of the packet. 2. Use a known good packet in your test @@ -1165,10 +1230,8 @@ this packet would be rejected by openstack. retry. -*How do I debug NSB Prox on Openstack?* ---------------------------------------- - -*Solution* +How do I debug NSB Prox on Openstack? +------------------------------------- 1. Execute the test as follows:: @@ -1239,10 +1302,8 @@ this packet would be rejected by openstack. Now continue as baremetal. -*How do I resolve "Quota exceeded for resources"* -------------------------------------------------- - -*Solution* +How do I resolve "Quota exceeded for resources" +----------------------------------------------- This usually occurs due to 2 reasons when executing an openstack test. @@ -1276,10 +1337,8 @@ This usually occurs due to 2 reasons when executing an openstack test. openstack quota set <resource> -*Openstack Cli fails or hangs. How do I resolve this?* ------------------------------------------------------- - -*Solution* +Openstack CLI fails or hangs. How do I resolve this? +---------------------------------------------------- If it fails due to :: @@ -1310,7 +1369,7 @@ Result :: and visible. -If the Openstack ClI appears to hang, then verify the proxys and ``no_proxy`` +If the Openstack CLI appears to hang, then verify the proxys and ``no_proxy`` are set correctly. They should be similar to :: FTP_PROXY="http://<your_proxy>:<port>/" @@ -1328,8 +1387,8 @@ Where 2) 10.237.223.80 = IP Address of Controller node 3) 10.237.222.134 = IP Address of Compute Node -*How to Understand the Grafana output?* ---------------------------------------- +How to Understand the Grafana output? +------------------------------------- .. image:: images/PROX_Grafana_1.png :width: 1000px @@ -1347,50 +1406,75 @@ Where :width: 1000px :alt: NSB PROX Grafana_4 -A. Test Parameters - Test interval, Duartion, Tolerated Loss and Test Precision + .. image:: images/PROX_Grafana_5.png + :width: 1000px + :alt: NSB PROX Grafana_5 + + .. image:: images/PROX_Grafana_6.png + :width: 1000px + :alt: NSB PROX Grafana_6 + +A. Test Parameters - Test interval, Duration, Tolerated Loss and Test Precision B. No. of packets send and received during test -C. Generator Stats - packets sent, received and attempted by Generator +C. Generator Stats - Average Throughput per step (Step Duration is specified by + "Duration" field in A above) D. Packet size -E. No. of packets received by SUT +E. No. of packets sent by the generator per second per interface in millions + of packets per second. + +F. No. of packets recieved by the generator per second per interface in millions + of packets per second. + +G. No. of packets received by the SUT from the generator in millions of packets + per second. -F. No. of packets forwarded by SUT +H. No. of packets sent by the the SUT to the generator in millions of packets + per second. -G. No. of packets sent by the generator per port, for each interval. +I. No. of packets sent by the Generator to the SUT per step per interface + in millions of packets per second. -H. No. of packets received by the generator per port, for each interval. +J. No. of packets received by the Generator from the SUT per step per interface + in millions of packets per second. -I. No. of packets sent and received by the generator and lost by the SUT that +K. No. of packets sent and received by the generator and lost by the SUT that meet the success criteria -J. The change in the Percentage of Line Rate used over a test, The MAX and the +L. The change in the Percentage of Line Rate used over a test, The MAX and the MIN should converge to within the interval specified as the ``test-precision``. -K. Packet size supported during test. If *N/A* appears in any field the +M. Packet size supported during test. If *N/A* appears in any field the result has not been decided. -L. Calculated throughput in MPPS (Million Packets Per second) for this line - rate. +N. The Theretical Maximum no. of packets per second that can be sent for this + packet size. -M. No. of packets sent by the generator in MPPS +O. No. of packets sent by the generator in MPPS -N. No. of packets received by the generator in MPPS +P. No. of packets received by the generator in MPPS -O. No. of packets sent by SUT. +Q. No. of packets sent by SUT. -P. No. of packets received by the SUT +R. No. of packets received by the SUT -Q. Total no. of dropped packets -- Packets sent but not received back by the +S. Total no. of dropped packets -- Packets sent but not received back by the generator, these may be dropped by the SUT or the generator. -R. The tolerated no. of dropped packets. +T. The tolerated no. of dropped packets. -S. Test throughput in Gbps +U. Test throughput in Gbps -T. Latencey per Port +V. Latencey per Port + * Va - Port XE0 + * Vb - Port XE1 + * Vc - Port XE0 + * Vd - Port XE0 -U. CPU Utilization +W. CPU Utilization + * Wa - CPU Utilization of the Generator + * Wb - CPU Utilization of the SUT diff --git a/docs/testing/developer/devguide/images/PROX_Grafana_1.png b/docs/testing/developer/devguide/images/PROX_Grafana_1.png Binary files differindex d272edcf3..144000bb8 100644 --- a/docs/testing/developer/devguide/images/PROX_Grafana_1.png +++ b/docs/testing/developer/devguide/images/PROX_Grafana_1.png diff --git a/docs/testing/developer/devguide/images/PROX_Grafana_2.png b/docs/testing/developer/devguide/images/PROX_Grafana_2.png Binary files differindex 4f7fd4cf5..af1ebb315 100644 --- a/docs/testing/developer/devguide/images/PROX_Grafana_2.png +++ b/docs/testing/developer/devguide/images/PROX_Grafana_2.png diff --git a/docs/testing/developer/devguide/images/PROX_Grafana_3.png b/docs/testing/developer/devguide/images/PROX_Grafana_3.png Binary files differindex 5ae967698..a287670c9 100644 --- a/docs/testing/developer/devguide/images/PROX_Grafana_3.png +++ b/docs/testing/developer/devguide/images/PROX_Grafana_3.png diff --git a/docs/testing/developer/devguide/images/PROX_Grafana_4.png b/docs/testing/developer/devguide/images/PROX_Grafana_4.png Binary files differindex 5353d1c7e..6752dc324 100644 --- a/docs/testing/developer/devguide/images/PROX_Grafana_4.png +++ b/docs/testing/developer/devguide/images/PROX_Grafana_4.png diff --git a/docs/testing/developer/devguide/images/PROX_Grafana_5.png b/docs/testing/developer/devguide/images/PROX_Grafana_5.png Binary files differnew file mode 100644 index 000000000..45746d86b --- /dev/null +++ b/docs/testing/developer/devguide/images/PROX_Grafana_5.png diff --git a/docs/testing/developer/devguide/images/PROX_Grafana_6.png b/docs/testing/developer/devguide/images/PROX_Grafana_6.png Binary files differnew file mode 100644 index 000000000..5bdbcf26a --- /dev/null +++ b/docs/testing/developer/devguide/images/PROX_Grafana_6.png diff --git a/docs/testing/developer/devguide/images/PROX_Test_BM_Script.png b/docs/testing/developer/devguide/images/PROX_Test_BM_Script.png Binary files differindex c09f7bb1b..84e640a20 100644 --- a/docs/testing/developer/devguide/images/PROX_Test_BM_Script.png +++ b/docs/testing/developer/devguide/images/PROX_Test_BM_Script.png diff --git a/docs/testing/developer/devguide/images/PROX_Test_ovs_dpdk_Script_1.png b/docs/testing/developer/devguide/images/PROX_Test_ovs_dpdk_Script_1.png Binary files differnew file mode 100644 index 000000000..73f6f2920 --- /dev/null +++ b/docs/testing/developer/devguide/images/PROX_Test_ovs_dpdk_Script_1.png diff --git a/docs/testing/developer/devguide/images/PROX_Test_ovs_dpdk_Script_2.png b/docs/testing/developer/devguide/images/PROX_Test_ovs_dpdk_Script_2.png Binary files differnew file mode 100644 index 000000000..ad7d22822 --- /dev/null +++ b/docs/testing/developer/devguide/images/PROX_Test_ovs_dpdk_Script_2.png diff --git a/docs/testing/developer/devguide/images/vPE_Diagram.png b/docs/testing/developer/devguide/images/vPE_Diagram.png Binary files differnew file mode 100644 index 000000000..c3985b7d9 --- /dev/null +++ b/docs/testing/developer/devguide/images/vPE_Diagram.png diff --git a/docs/testing/user/userguide/04-installation.rst b/docs/testing/user/userguide/04-installation.rst index 2f8175c25..3ba312ce7 100644 --- a/docs/testing/user/userguide/04-installation.rst +++ b/docs/testing/user/userguide/04-installation.rst @@ -613,15 +613,15 @@ Run influxDB:: sudo -EH docker run -d --name influxdb \ -p 8083:8083 -p 8086:8086 --expose 8090 --expose 8099 \ tutum/influxdb - docker exec -it influxdb influx Configure influxDB:: + docker exec -it influxdb influx > CREATE USER root WITH PASSWORD 'root' WITH ALL PRIVILEGES > CREATE DATABASE yardstick; > use yardstick; > show MEASUREMENTS; - > quit + > exit Run Grafana:: diff --git a/docs/testing/user/userguide/10-yardstick-user-interface.rst b/docs/testing/user/userguide/10-yardstick-user-interface.rst index 76890b29a..b3056ec99 100644 --- a/docs/testing/user/userguide/10-yardstick-user-interface.rst +++ b/docs/testing/user/userguide/10-yardstick-user-interface.rst @@ -2,16 +2,32 @@ Yardstick User Interface ======================== -This interface provides a user to view the test result -in table format and also values pinned on to a graph. +This chapter describes how to generate HTML reports, used to view, store, share +or publish test results in table and graph formats. +The following layouts are available: -Command -======= -:: +* The compact HTML report layout is suitable for testcases producing a few + metrics over a short period of time. All metrics for all timestamps are + displayed in the data table and on the graph. + +* The dynamic HTML report layout consists of a wider data table, a graph, and + a tree that allows selecting the metrics to be displayed. This layout is + suitable for testcases, such as NSB ones, producing a lot of metrics over + a longer period of time. + + +Commands +======== + +To generate the compact HTML report, run:: yardstick report generate <task-ID> <testcase-filename> +To generate the dynamic HTML report, run:: + + yardstick report generate-nsb <task-ID> <testcase-filename> + Description =========== @@ -27,8 +43,8 @@ Description The graph is framed with Timestamp on x-axis and output values (differ from testcase to testcase) on y-axis with the help of -`Highcharts`_. +`Chart.js`_. .. _InfluxDB: https://www.influxdata.com/time-series-platform/influxdb/ .. _Jinja2: http://jinja.pocoo.org/docs/2.10/ -.. _Highcharts: https://www.highcharts.com/products/highcharts/ +.. _Chart.js: https://www.chartjs.org/ diff --git a/docs/testing/user/userguide/14-nsb-operation.rst b/docs/testing/user/userguide/14-nsb-operation.rst index 7ec5b4ec0..72b1c4bbc 100644 --- a/docs/testing/user/userguide/14-nsb-operation.rst +++ b/docs/testing/user/userguide/14-nsb-operation.rst @@ -601,3 +601,33 @@ may be changed. 2. Subsection ``runner``: specifies the test duration and the interval of TG and VNF side KPIs polling. For more details, refer to :doc:`03-architecture`. + +Preparing test run of vPE test case +----------------------------------- +The vPE (Provider Edge Router) is a :term: `VNF` approximation +serving as an Edge Router. The vPE is approximated using the +``ip_pipeline`` dpdk application. + + .. image:: images/vPE_Diagram.png + :width: 800px + :alt: NSB vPE Diagram + +The ``vpe_config`` file must be passed as it is not auto generated. +The ``vpe_script`` defines the rules applied to each of the pipelines. This can be +auto generated or a file can be passed using the ``script_file`` option in +``vnf_config`` as shown below. The ``full_tm_profile_file`` option must be +used if a traffic manager is defined in ``vpe_config``. + +.. code-block:: yaml + + vnf_config: { file: './vpe_config/vpe_config_2_ports', + action_bulk_file: './vpe_config/action_bulk_512.txt', + full_tm_profile_file: './vpe_config/full_tm_profile_10G.cfg', + script_file: './vpe_config/vpe_script_sample' } + +Testcases for vPE can be found in the ``vnf_samples/nsut/vpe`` directory. +A testcase can be started with the following command as an example: + +.. code-block:: bash + + yardstick task start /yardstick/samples/vnf_samples/nsut/vpe/tc_baremetal_rfc2544_ipv4_1flow_64B_ixia.yaml diff --git a/docs/testing/user/userguide/glossary.rst b/docs/testing/user/userguide/glossary.rst index 6a153943c..cef9b69a5 100644 --- a/docs/testing/user/userguide/glossary.rst +++ b/docs/testing/user/userguide/glossary.rst @@ -13,23 +13,54 @@ Glossary API Application Programming Interface + Barometer + OPNFV NFVi Service Assurance project. Barometer upstreams changes to + collectd, OpenStack, etc to improve features related to NFVi monitoring + and service assurance. + More info on: https://opnfv-barometer.readthedocs.io/en/latest/ + + collectd + collectd is a system statistics collection daemon. + More info on: https://collectd.org/ + + context + A context describes the environment in which a yardstick testcase will + be run. It can refer to a pre-provisioned environment, or an environment + that will be set up using OpenStack or Kubernetes. + Docker Docker provisions and manages containers. Yardstick and many other OPNFV projects are deployed in containers. Docker is required to launch the containerized versions of these projects. - DPI - Deep Packet Inspection - DPDK Data Plane Development Kit + DPI + Deep Packet Inspection + DSCP Differentiated Services Code Point + flavor + A specification of virtual resources used by OpenStack in the creation + of a VM instance. + + Grafana + A visualization tool, used in Yardstick to retrieve test data from + InfluxDB and display it. Grafana works by defining dashboards, which are + combinations of visualization panes (e.g. line charts and gauges) and + forms that assist the user in formulating SQL-like queries for InfluxDB. + More info on: https://grafana.com/ + IGMP Internet Group Management Protocol + InfluxDB + One of the Dispatchers supported by Yardstick, it allows test results to + be reported to a time-series database. + More info on: https://www.influxdata.com/ + IOPS Input/Output Operations Per Second A performance measurement used to benchmark storage devices. @@ -43,6 +74,9 @@ Glossary deployment, scaling and management of containerized applications. It is one of the contexts supported in Yardstick. + MPLS + Multiprotocol Label Switching + NFV Network Function Virtualization NFV is an initiative to take network services which were traditionally run @@ -56,6 +90,10 @@ Glossary NIC Network Interface Controller + NSB + Network Services Benchmarking. A subset of Yardstick features concerned + with NFVI and VNF characterization. + OpenStack OpenStack is a cloud operating system that controls pools of compute, storage, and networking resources. OpenStack is an open source project @@ -77,6 +115,18 @@ Glossary performance in Input/Output Operations Per Second (IOPS), throttling agreements, and performance expectations at peak load + runner + The part of a Yardstick testcase that determines how the test will be run + (e.g. for x iterations, y seconds or until state z is reached). The runner + also determines when the metrics are collected/reported. + + SampleVNF + OPNFV project providing a repository of reference VNFs. + More info on: https://opnfv-samplevnf.readthedocs.io/en/latest/ + + scenario + The part of a Yardstick testcase that describes each test step. + SLA Service Level Agreement An SLA is an agreement between a service provider and a customer to @@ -92,6 +142,10 @@ Glossary SUT System Under Test + testcase + A task in Yardstick; the yaml file that is read by Yardstick to + determine how to run a test. + ToS Type of Service diff --git a/docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst b/docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst index 723cd6f99..6c18c7d89 100644 --- a/docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst +++ b/docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst @@ -33,3 +33,6 @@ NSB PROX Test Case Descriptions tc_epc_saegw_tput_relocation_landslide tc_epc_network_service_request_landslide tc_epc_ue_service_request_landslide + tc_vfw_rfc2544 + tc_vfw_rfc2544_correlated + tc_vfw_rfc3511 diff --git a/docs/testing/user/userguide/nsb/tc_vfw_rfc2544.rst b/docs/testing/user/userguide/nsb/tc_vfw_rfc2544.rst new file mode 100644 index 000000000..139990bc3 --- /dev/null +++ b/docs/testing/user/userguide/nsb/tc_vfw_rfc2544.rst @@ -0,0 +1,189 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, 2018 Intel Corporation. + +************************************************ +Yardstick Test Case Description: NSB vFW RFC2544 +************************************************ + ++------------------------------------------------------------------------------+ +| NSB vFW test for VNF characterization | +| | ++---------------+--------------------------------------------------------------+ +| test case id | tc_{context}_rfc2544_ipv4_1rule_1flow_{pkt_size}_{tg_type} | +| | | +| | * context = baremetal, heat, heat_external, ovs, sriov | +| | heat_sriov_external contexts; | +| | * tg_type = ixia (context != heat,heat_sriov_external), | +| | trex; | +| | * pkt_size = 64B - all contexts; | +| | 128B, 256B, 512B, 1024B, 1280B, 1518B - | +| | (context = heat, tg_type = ixia) | +| | | ++---------------+--------------------------------------------------------------+ +| metric | * Network Throughput; | +| | * TG Packets Out; | +| | * TG Packets In; | +| | * TG Latency; | +| | * VNF Packets Out; | +| | * VNF Packets In; | +| | * VNF Packets Fwd; | +| | * Dropped packets; | +| | | ++---------------+--------------------------------------------------------------+ +| test purpose | The VFW RFC2544 tests measure performance characteristics of | +| | the SUT (multiple ports) and sends UDP bidirectional traffic | +| | from all TG ports to SampleVNF vFW application. The | +| | application forwards received traffic based on rules | +| | provided by the user in the TC configuration and default | +| | rules created by vFW to send traffic from uplink ports to | +| | downlink and voice versa. | +| | | ++---------------+--------------------------------------------------------------+ +| configuration | The 2 ports RFC2544 test cases are listed below: | +| | | +| | * tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_ixia.yaml | +| | * tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_trex.yaml | +| | * tc_heat_external_rfc2544_ipv4_1rule_1flow_1024B_ixia.yaml | +| | * tc_heat_external_rfc2544_ipv4_1rule_1flow_1280B_ixia.yaml | +| | * tc_heat_external_rfc2544_ipv4_1rule_1flow_128B_ixia.yaml | +| | * tc_heat_external_rfc2544_ipv4_1rule_1flow_1518B_ixia.yaml | +| | * tc_heat_external_rfc2544_ipv4_1rule_1flow_256B_ixia.yaml | +| | * tc_heat_external_rfc2544_ipv4_1rule_1flow_512B_ixia.yaml | +| | * tc_heat_external_rfc2544_ipv4_1rule_1flow_64B_ixia.yaml | +| | * tc_heat_external_rfc2544_ipv4_1rule_1flow_64B_trex.yaml | +| | * tc_heat_sriov_external_rfc2544_ipv4_1rule_1flow_64B_trex. | +| | yaml | +| | * tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex.yaml | +| | * tc_ovs_rfc2544_ipv4_1rule_1flow_64B_ixia.yaml | +| | * tc_ovs_rfc2544_ipv4_1rule_1flow_64B_trex.yaml | +| | * tc_sriov_rfc2544_ipv4_1rule_1flow_64B_ixia.yaml | +| | * tc_sriov_rfc2544_ipv4_1rule_1flow_64B_trex.yaml | +| | | +| | The 4 ports RFC2544 test cases are listed below: | +| | | +| | * tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_ixia_4port.yaml | +| | * tc_tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_trex_4port. | +| | yaml | +| | * tc_tc_heat_external_rfc2544_ipv4_1rule_1flow_64B_trex_4 | +| | port.yaml | +| | * tc_tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_4port.yaml | +| | | +| | The scale-up RFC2544 test cases are listed below: | +| | | +| | * tc_tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_scale-up.yaml | +| | | +| | The scale-out RFC2544 test cases are listed below: | +| | | +| | * tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_scale_out.yaml | +| | | +| | Test duration is set as 30 sec for each test and default | +| | number of rules are applied. These can be configured | +| | | ++---------------+--------------------------------------------------------------+ +| test tool | The vFW is a DPDK application that performs basic filtering | +| | for malformed packets and dynamic packet filtering of | +| | incoming packets using the connection tracker library. | +| | | ++---------------+--------------------------------------------------------------+ +| applicability | The vFW RFC2544 test cases can be configured with different: | +| | | +| | * packet sizes; | +| | * test duration; | +| | * tolerated loss; | +| | * traffic flows; | +| | * rules; | +| | | +| | Default values exist. | +| | | ++---------------+--------------------------------------------------------------+ +| pre-test | For OpenStack test case image (yardstick-samplevnf) needs | +| conditions | to be installed into Glance with vFW and DPDK included in | +| | it (NSB install). | +| | | +| | For Baremetal tests cases vFW and DPDK must be installed on | +| | the hosts where the test is executed. The pod.yaml file must | +| | have the necessary system and NIC information. | +| | | +| | For standalone (SA) SRIOV/OvS test cases the | +| | yardstick-samplevnf image needs to be installed on hosts and | +| | pod.yaml file must be provided with necessary system, NIC | +| | information. | +| | | ++---------------+--------------------------------------------------------------+ +| test sequence | Description and expected result | +| | | ++---------------+--------------------------------------------------------------+ +| step 1 | For Baremetal test: The TG (except IXIA) and VNF are started | +| | on the hosts based on the pod file. | +| | | +| | For Heat test: Two host VMs are booted, as Traffic generator | +| | and VNF(vFW) based on the test flavor. In case of scale-out | +| | scenario the multiple VNF VMs will be started. | +| | | +| | For Heat external test: vFW VM is booted and TG (except IXIA)| +| | generator is started on the external host based on the pod | +| | file. In case of scale-out scenario the multiple VNF VMs | +| | will be deployed. | +| | | +| | For Heat SRIOV external test: vFW VM is booted with network | +| | interfaces of `direct` type which are mapped to VFs that are | +| | available to OpenStack. TG (except IXIA) is started on the | +| | external host based on the pod file. In case of scale-out | +| | scenario the multiple VNF VMs will be deployed. | +| | | +| | For SRIOV test: VF ports are created on host's PFs specified | +| | in the TC file and VM is booed using those ports and image | +| | provided in the configuration. TG (except IXIA) is started | +| | on other host connected to VNF machine based on the pod | +| | file. The vFW is started in the booted VM. In case of | +| | scale-out scenario the multiple VNF VMs will be created. | +| | | +| | For OvS-DPDK test: OvS DPDK switch is started and bridges | +| | are created with ports specified in the TC file. DPDK vHost | +| | ports are added to corresponding bridge and VM is booed | +| | using those ports and image provided in the configuration. | +| | TG (except IXIA) is started on other host connected to VNF | +| | machine based on the pod file. The vFW is started in the | +| | booted VM. In case of scale-out scenario the multiple VNF | +| | VMs will be deployed. | +| | | ++---------------+--------------------------------------------------------------+ +| step 2 | Yardstick is connected with the TG and VNF by using ssh (in | +| | case of IXIA TG is connected via TCL interface). The test | +| | will resolve the topology and instantiate all VNFs | +| | and TG and collect the KPI's/metrics. | +| | | ++---------------+--------------------------------------------------------------+ +| step 3 | The TG will send packets to the VNFs. If the number of | +| | dropped packets is more than the tolerated loss the line | +| | rate or throughput is halved. This is done until the dropped | +| | packets are within an acceptable tolerated loss. | +| | | +| | The KPI is the number of packets per second for different | +| | packet size with an accepted minimal packet loss for the | +| | default configuration. | +| | | ++---------------+--------------------------------------------------------------+ +| step 4 | In Baremetal test: The test quits the application and unbind | +| | the DPDK ports. | +| | | +| | In Heat test: All VNF VMs and TG are deleted on test | +| | completion. | +| | | +| | In SRIOV test: The deployed VM with vFW is destroyed on the | +| | host and TG (exclude IXIA) is stopped. | +| | | +| | In Heat SRIOV test: The deployed VM with vFW is destroyed, | +| | VFs are released and TG (exclude IXIA) is stopped. | +| | | +| | In OvS test: The deployed VM with vFW is destroyed on the | +| | host and OvS DPDK switch is stopped and ports are unbinded. | +| | The TG (exclude IXIA) is stopped. | +| | | ++---------------+--------------------------------------------------------------+ +| test verdict | The test case will achieve a Throughput with an accepted | +| | minimal tolerated packet loss. | ++---------------+--------------------------------------------------------------+ + diff --git a/docs/testing/user/userguide/nsb/tc_vfw_rfc2544_correlated.rst b/docs/testing/user/userguide/nsb/tc_vfw_rfc2544_correlated.rst new file mode 100644 index 000000000..de490900d --- /dev/null +++ b/docs/testing/user/userguide/nsb/tc_vfw_rfc2544_correlated.rst @@ -0,0 +1,130 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, 2018 Intel Corporation. + +************************************************************* +Yardstick Test Case Description: NSB vFW RFC2544 (correlated) +************************************************************* + ++------------------------------------------------------------------------------+ +| NSB vFW test for VNF characterization using correlated traffic | +| | ++---------------+--------------------------------------------------------------+ +| test case id | tc_{context}_rfc2544_ipv4_1rule_1flow_64B_trex_corelated | +| | | +| | * context = baremetal, heat | +| | | ++---------------+--------------------------------------------------------------+ +| metric | * Network Throughput; | +| | * TG Packets Out; | +| | * TG Packets In; | +| | * TG Latency; | +| | * VNF Packets Out; | +| | * VNF Packets In; | +| | * VNF Packets Fwd; | +| | * Dropped packets; | +| | | +| | NOTE: For correlated TCs the TG metrics are available on | +| | uplink ports. | +| | | ++---------------+--------------------------------------------------------------+ +| test purpose | The VFW RFC2544 correlated tests measure performance | +| | characteristics of the SUT (multiple ports) and sends UDP | +| | traffic from uplink TG ports to SampleVNF vFW application. | +| | The application forwards received traffic from uplink ports | +| | to downlink ports based on rules provided by the user in the | +| | TC configuration and default rules created by vFW. The VNF | +| | downlink traffic is received by another UDPReplay VNF and it | +| | is mirrored back to the VNF on the same port. Finally, the | +| | traffic is received back to the TG uplink port. | +| | | ++---------------+--------------------------------------------------------------+ +| configuration | The 2 ports RFC2544 correlated test cases are listed below: | +| | | +| | * tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_trex_corelated | +| | _traffic.yaml | +| | | +| | Multiple VNF (2, 4, 10) RFC2544 correlated test cases are | +| | listed below: | +| | | +| | * tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_correlated | +| | _scale_10.yaml | +| | * tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_correlated_scale | +| | _2.yaml | +| | * tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_correlated_scale | +| | _4.yaml | +| | | +| | The scale-out RFC2544 test cases are listed below: | +| | | +| | * tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_correlated_scale | +| | _out.yaml | +| | | +| | Test duration is set as 30 sec for each test and default | +| | number of rules are applied. These can be configured | +| | | ++---------------+--------------------------------------------------------------+ +| test tool | The vFW is a DPDK application that performs basic filtering | +| | for malformed packets and dynamic packet filtering of | +| | incoming packets using the connection tracker library. | +| | | ++---------------+--------------------------------------------------------------+ +| applicability | The vFW RFC2544 test cases can be configured with different: | +| | | +| | * packet sizes; | +| | * test duration; | +| | * tolerated loss; | +| | * traffic flows; | +| | * rules; | +| | | +| | Default values exist. | +| | | ++---------------+--------------------------------------------------------------+ +| pre-test | For OpenStack test case image (yardstick-samplevnf) needs | +| conditions | to be installed into Glance with vFW and DPDK included in | +| | it (NSB install). | +| | | +| | For Baremetal tests cases vFW and DPDK must be installed on | +| | the hosts where the test is executed. The pod.yaml file must | +| | have the necessary system and NIC information. | +| | | ++---------------+--------------------------------------------------------------+ +| test sequence | Description and expected result | +| | | ++---------------+--------------------------------------------------------------+ +| step 1 | For Baremetal test: The TG (except IXIA), vFW and UDPReplay | +| | VNFs are started on the hosts based on the pod file. | +| | | +| | For Heat test: Three host VMs are booted, as Traffic | +| | generator, vFW and UDPReplay VNF(vFW) based on the test | +| | flavor. In case of scale-out scenario the multiple vFW VNF | +| | VMs will be started. | +| | | ++---------------+--------------------------------------------------------------+ +| step 2 | Yardstick is connected with the TG, vFW and UDPReplay VNF by | +| | using ssh (in case of IXIA TG is connected via TCL | +| | interface). The test will resolve the topology and | +| | instantiate all VNFs and TG and collect the KPI's/metrics. | +| | | ++---------------+--------------------------------------------------------------+ +| step 3 | The TG will send packets to the VNFs. If the number of | +| | dropped packets is more than the tolerated loss the line | +| | rate or throughput is halved. This is done until the dropped | +| | packets are within an acceptable tolerated loss. | +| | | +| | The KPI is the number of packets per second for 64B packet | +| | size with an accepted minimal packet loss for the default | +| | configuration. | +| | | ++---------------+--------------------------------------------------------------+ +| step 4 | In Baremetal test: The test quits the application and unbind | +| | the DPDK ports. | +| | | +| | In Heat test: All VNF VMs and TG are deleted on test | +| | completion. | +| | | ++---------------+--------------------------------------------------------------+ +| test verdict | The test case will achieve a Throughput with an accepted | +| | minimal tolerated packet loss. | ++---------------+--------------------------------------------------------------+ + diff --git a/docs/testing/user/userguide/nsb/tc_vfw_rfc3511.rst b/docs/testing/user/userguide/nsb/tc_vfw_rfc3511.rst new file mode 100644 index 000000000..9051fc4df --- /dev/null +++ b/docs/testing/user/userguide/nsb/tc_vfw_rfc3511.rst @@ -0,0 +1,133 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, 2018 Intel Corporation. + +******************************************************* +Yardstick Test Case Description: NSB vFW RFC3511 (HTTP) +******************************************************* + ++------------------------------------------------------------------------------+ +| NSB vFW test for VNF characterization based on RFC3511 and IXIA | +| | ++---------------+--------------------------------------------------------------+ +| test case id | tc_{context}_http_ixload_{http_size}_Requests-65000_{type} | +| | | +| | * context = baremetal, heat_external | +| | * http_size = 1b, 4k, 64k, 256k, 512k, 1024k payload size | +| | * type = Concurrency, Connections, Throughput | +| | | ++---------------+--------------------------------------------------------------+ +| metric | * HTTP Total Throughput (Kbps); | +| | * HTTP Simulated Users; | +| | * HTTP Concurrent Connections; | +| | * HTTP Connection Rate; | +| | * HTTP Transaction Rate | +| | | ++---------------+--------------------------------------------------------------+ +| test purpose | The vFW RFC3511 tests measure performance characteristics of | +| | the SUT by sending the HTTP traffic from uplink to downlink | +| | TG ports through vFW VNF. The application forwards received | +| | traffic based on rules provided by the user in the TC | +| | configuration and default rules created by vFW to send | +| | traffic from uplink ports to downlink and voice versa. | +| | | ++---------------+--------------------------------------------------------------+ +| configuration | The 2 ports RFC3511 test cases are listed below: | +| | | +| | * tc_baremetal_http_ixload_1024k_Requests-65000 | +| | _Concurrency.yaml | +| | * tc_baremetal_http_ixload_1b_Requests-65000 | +| | _Concurrency.yaml | +| | * tc_baremetal_http_ixload_256k_Requests-65000 | +| | _Concurrency.yaml | +| | * tc_baremetal_http_ixload_4k_Requests-65000 | +| | _Concurrency.yaml | +| | * tc_baremetal_http_ixload_512k_Requests-65000 | +| | _Concurrency.yaml | +| | * tc_baremetal_http_ixload_64k_Requests-65000 | +| | _Concurrency.yaml | +| | * tc_heat_external_http_ixload_1b_Requests-10Gbps | +| | _Throughput.yaml | +| | * tc_heat_external_http_ixload_1b_Requests-65000 | +| | _Concurrency.yaml | +| | * tc_heat_external_http_ixload_1b_Requests-65000 | +| | _Connections.yaml | +| | | +| | The 4 ports RFC3511 test cases are listed below: | +| | | +| | * tc_baremetal_http_ixload_1b_Requests-65000 | +| | _Concurrency_4port.yaml | +| | | ++---------------+--------------------------------------------------------------+ +| test tool | The vFW is a DPDK application that performs basic filtering | +| | for malformed packets and dynamic packet filtering of | +| | incoming packets using the connection tracker library. | +| | | ++---------------+--------------------------------------------------------------+ +| applicability | The vFW RFC3511 test cases can be configured with different: | +| | | +| | * http payload sizes; | +| | * traffic flows; | +| | * rules; | +| | | +| | Default values exist. | +| | | ++---------------+--------------------------------------------------------------+ +| pre-test | For OpenStack test case image (yardstick-samplevnf) needs | +| conditions | to be installed into Glance with vFW and DPDK included in | +| | it (NSB install). | +| | | +| | For Baremetal tests cases vFW and DPDK must be installed on | +| | the hosts where the test is executed. The pod.yaml file must | +| | have the necessary system and NIC information. | +| | | ++---------------+--------------------------------------------------------------+ +| test sequence | Description and expected result | +| | | ++---------------+--------------------------------------------------------------+ +| step 1 | For Baremetal test: The vFW VNF is started on the hosts | +| | based on the pod file. | +| | | +| | For Heat external test: The vFW VM are deployed and booted. | +| | | ++---------------+--------------------------------------------------------------+ +| step 2 | Yardstick is connected with the TG (IxLoad) via IxLoad API | +| | and VNF by using ssh. The test will resolve the topology and | +| | instantiate all VNFs and TG and collect the KPI's/metrics. | +| | | ++---------------+--------------------------------------------------------------+ +| step 3 | The TG simulates HTTP traffic based on selected type of TC. | +| | | +| | Concurrency: | +| | The TC attempts to simulate some number of human users. | +| | The simulated users are gradually brought online until 64K | +| | users is met (the Ramp-Up phase), then taken offline (the | +| | Ramp Down phase). | +| | | +| | Connections: | +| | The TC creates some number of HTTP connections per second. | +| | It will attempt to generate the 64K of HTTP connections | +| | per second. | +| | | +| | Throughput: | +| | TC simultaneously transmits and receives TCP payload | +| | (bytes) at a certain rate measured in Megabits per second | +| | (Mbps), Kilobits per second (Kbps), or Gigabits per | +| | second. The 10 Gbits is default throughput. | +| | | +| | At the end of the TC, the KPIs are collected and stored | +| | (depends on the selected dispatcher). | +| | | ++---------------+--------------------------------------------------------------+ +| step 4 | In Baremetal test: The test quits the application and | +| | unbinds the DPDK ports. | +| | | +| | In Heat test: All VNF VMs are deleted and connections to TG | +| | are terminated. | +| | | ++---------------+--------------------------------------------------------------+ +| test verdict | The test case will try to achieve the configured HTTP | +| | Concurrency/Throughput/Connections. | ++---------------+--------------------------------------------------------------+ + |
