NFVBENCH-115 SRIOV multi-chain with non shared networks fails

Change-Id: I5a5c271aa4f6d0f97678777965cca611c9224dac
Signed-off-by: ahothan <ahothan@cisco.com>

1 files changed, 346 insertions, 5 deletions

diff --git a/docs/testing/user/userguide/sriov.rst b/docs/testing/user/userguide/sriov.rst
index 7702d3c..f5cc7d2 100644
--- a/docs/testing/user/userguide/sriov.rst
+++ b/docs/testing/user/userguide/sriov.rst
@@ -6,13 +6,19 @@
 Testing SR-IOV
 ==============
 
-NFVbench supports SR-IOV with the PVP packet flow (PVVP is not supported). SR-IOV support is not applicable for external chains since the networks have to be setup externally (and can themselves be pre-set to use SR-IOV or not).
+NFVbench supports SR-IOV with the PVP and PVVP packet flow. Most use cases for SR-IOV only require single VNF chains (NxPVP).
+Daisy chaining VNFs with SR-IOV (PVVP) requires selecting either SR-IOV for the middle network or a fast vswitch (using the
+standard OVS for that purpose works but would be a serious bottleneck)
+
+Instructions below refer to the PVP or PVVP use cases.
+For external chains using SR-IOV, select the VLAN tagging option that corresponds to the external chains SR-IOV setting.
 
 Pre-requisites
 --------------
-To test SR-IOV you need to have compute nodes configured to support one or more SR-IOV interfaces (also knows as PF or physical function) and you need OpenStack to be configured to support SR-IOV.
+To test SR-IOV you need to have compute nodes configured to support one or more SR-IOV interfaces (also knows as PF or physical function)
+and you need OpenStack to be configured to support SR-IOV.
 You will also need to know:
-- the name of the physical networks associated to your SR-IOV interfaces (this is a configuration in Nova compute)
+- the name of the physical networks associated to the SR-IOV interfaces (this is a configuration in Nova compute)
 - the VLAN range that can be used on the switch ports that are wired to the SR-IOV ports. Such switch ports are normally configured in trunk mode with a range of VLAN ids enabled on that port
 
 For example, in the case of 2 SR-IOV ports per compute node, 2 physical networks are generally configured in OpenStack with a distinct name.
@@ -39,6 +45,39 @@ The segmentation ID fields must be different.
 In the case of PVVP, the middle network also needs to be provisioned properly.
 The same physical network can also be shared by the virtual networks but with different segmentation IDs.
 
+Multi-Chaining
+--------------
+The above configuration works for multi-chaining and shared network ("service_chain_shared_net" set to true).
+In that case all VNFs will share the same left and right network/VLAN.
+
+In the case of non shared network ("service_chain_shared_net" set to false), the segmentation_id fields must
+contain a list of distinct VLANs to use for each chain. Example of configuration for 3 chains:
+
+.. code-block:: bash
+
+    sriov: true
+    internal_networks:
+       left:
+           segmentation_id: [2000, 2001, 2002]
+           physical_network: phys_sriov0
+       right:
+           segmentation_id: [2100, 2101, 2102]
+           physical_network: phys_sriov1
+
+Alternatively it is also possible to specify different physnets per chain:
+
+.. code-block:: bash
+
+    sriov: true
+    internal_networks:
+       left:
+           segmentation_id: [2000, 2001, 2002]
+           physical_network: [phys_sriov0, phys_sriov2, phys_sriov4]
+       right:
+           segmentation_id: [2100, 2101, 2102]
+           physical_network: [phys_sriov1, phys_srviov3, phys_sriov5]
+
+
 NFVbench cores with SR-IOV
 --------------------------
 The default core count for NFVbench/TRex may not be sufficient for higher throughput line cards (greater than 10Gbps).
@@ -96,8 +135,10 @@ Failure to do so might cause the VM creation to fail with the Nova error
 "Instance creation error: Insufficient compute resources:
 Requested instance NUMA topology together with requested PCI devices cannot fit the given host NUMA topology."
 
-Example of configuration file (summary)
----------------------------------------
+Example of configuration file (shared network)
+----------------------------------------------
+
+Single chain or multi-chain with shared network (only requires 2 segmentation ID for all chains):
 
 .. code-block:: bash
 
@@ -119,3 +160,303 @@ Example of configuration file (summary)
        right:
           segmentation_id: 3831
           physical_network: phys_sriov1
+
+Example of full run 2xPVP shared network SR-IOV:
+
+.. code-block:: bash
+
+    2018-12-03 18:24:07,419 INFO Loading configuration file: /tmp/nfvbench/sriov.yaml
+    2018-12-03 18:24:07,423 INFO -c /tmp/nfvbench/sriov.yaml --rate 10Mpps --duration 1 -scc 2 --no-cleanup
+    2018-12-03 18:24:07,426 INFO Connecting to TRex (127.0.0.1)...
+    2018-12-03 18:24:07,575 INFO Connected to TRex
+    2018-12-03 18:24:07,575 INFO    Port 0: Ethernet Controller XL710 for 40GbE QSFP+ speed=40Gbps mac=3c:fd:fe:b5:3d:70 pci=0000:5e:00.0 driver=net_i40e
+    2018-12-03 18:24:07,575 INFO    Port 1: Ethernet Controller XL710 for 40GbE QSFP+ speed=40Gbps mac=3c:fd:fe:b5:3d:71 pci=0000:5e:00.1 driver=net_i40e
+    2018-12-03 18:24:07,626 INFO Found built-in VM image file nfvbenchvm-0.6.qcow2
+    2018-12-03 18:24:09,072 INFO Created flavor 'nfvbench.medium'
+    2018-12-03 18:24:10,004 INFO Created network: nfvbench-lnet.
+    2018-12-03 18:24:10,837 INFO Created network: nfvbench-rnet.
+    2018-12-03 18:24:12,065 INFO Security disabled on port nfvbench-loop-vm0-0
+    2018-12-03 18:24:13,425 INFO Security disabled on port nfvbench-loop-vm0-1
+    2018-12-03 18:24:13,425 INFO Creating instance nfvbench-loop-vm0 with AZ
+    2018-12-03 18:24:16,052 INFO Created instance nfvbench-loop-vm0 - waiting for placement resolution...
+    2018-12-03 18:24:16,240 INFO Waiting for instance nfvbench-loop-vm0 to become active (retry 1/101)...
+    <snip>
+    2018-12-03 18:24:59,266 INFO Waiting for instance nfvbench-loop-vm0 to become active (retry 21/101)...
+    2018-12-03 18:25:01,427 INFO Instance nfvbench-loop-vm0 is active and has been placed on nova:charter-compute-5
+    2018-12-03 18:25:02,819 INFO Security disabled on port nfvbench-loop-vm1-0
+    2018-12-03 18:25:04,198 INFO Security disabled on port nfvbench-loop-vm1-1
+    2018-12-03 18:25:04,199 INFO Creating instance nfvbench-loop-vm1 with AZ nova:charter-compute-5
+    2018-12-03 18:25:05,032 INFO Created instance nfvbench-loop-vm1 on nova:charter-compute-5
+    2018-12-03 18:25:05,033 INFO Instance nfvbench-loop-vm0 is ACTIVE on nova:charter-compute-5
+    2018-12-03 18:25:05,212 INFO Waiting for 1/2 instance to become active (retry 1/100)...
+    <snip>
+    2018-12-03 18:25:48,531 INFO Waiting for 1/2 instance to become active (retry 21/100)...
+    2018-12-03 18:25:50,677 INFO Instance nfvbench-loop-vm1 is ACTIVE on nova:charter-compute-5
+    2018-12-03 18:25:50,677 INFO All instances are active
+    2018-12-03 18:25:50,677 INFO Port 0: VLANs [3830, 3830]
+    2018-12-03 18:25:50,677 INFO Port 1: VLANs [3831, 3831]
+    2018-12-03 18:25:50,677 INFO Port 0: dst MAC ['fa:16:3e:de:4e:54', 'fa:16:3e:7a:26:2b']
+    2018-12-03 18:25:50,677 INFO Port 1: dst MAC ['fa:16:3e:6c:bb:cd', 'fa:16:3e:e0:48:45']
+    2018-12-03 18:25:50,678 INFO ChainRunner initialized
+    2018-12-03 18:25:50,678 INFO Starting 2xPVP benchmark...
+    2018-12-03 18:25:50,683 INFO Starting traffic generator to ensure end-to-end connectivity
+    2018-12-03 18:25:50,698 INFO Created 2 traffic streams for port 0.
+    2018-12-03 18:25:50,700 INFO Created 2 traffic streams for port 1.
+    2018-12-03 18:25:50,821 INFO Captured unique src mac 0/4, capturing return packets (retry 1/100)...
+    2018-12-03 18:25:52,944 INFO Received packet from mac: fa:16:3e:de:4e:54 (chain=0, port=0)
+    2018-12-03 18:25:52,945 INFO Received packet from mac: fa:16:3e:6c:bb:cd (chain=0, port=1)
+    2018-12-03 18:25:53,077 INFO Captured unique src mac 2/4, capturing return packets (retry 2/100)...
+    <snip>
+    2018-12-03 18:26:10,798 INFO End-to-end connectivity established
+    2018-12-03 18:26:10,816 INFO Cleared all existing streams
+    2018-12-03 18:26:10,846 INFO Created 4 traffic streams for port 0.
+    2018-12-03 18:26:10,849 INFO Created 4 traffic streams for port 1.
+    2018-12-03 18:26:10,849 INFO Starting to generate traffic...
+    2018-12-03 18:26:10,850 INFO Running traffic generator
+    2018-12-03 18:26:11,877 INFO TX: 10000004; RX: 9999999; Est. Dropped: 5; Est. Drop rate: 0.0000%
+    2018-12-03 18:26:11,877 INFO ...traffic generating ended.
+    2018-12-03 18:26:11,882 INFO Service chain 'PVP' run completed.
+    2018-12-03 18:26:11,936 INFO Clean up skipped.
+    2018-12-03 18:26:11,969 INFO
+    ========== NFVBench Summary ==========
+    Date: 2018-12-03 18:25:50
+    NFVBench version 3.0.3.dev1
+    Openstack Neutron:
+      vSwitch: OPENVSWITCH
+      Encapsulation: VLAN
+    Benchmarks:
+    > Networks:
+      > Components:
+        > Traffic Generator:
+            Profile: trex-local
+            Tool: TRex
+        > Versions:
+          > Traffic_Generator:
+              build_date: Nov 13 2017
+              version: v2.32
+              built_by: hhaim
+              mode: STL
+              build_time: 10:58:17
+          > CiscoVIM: 2.9.7-17036
+      > Service chain:
+        > PVP:
+          > Traffic:
+              Profile: traffic_profile_64B
+              Bidirectional: True
+              Flow count: 10000
+              Service chains count: 2
+              Compute nodes: [u'nova:charter-compute-5']
+
+              Run Summary:
+
+                +-----------------+-------------+----------------------+----------------------+----------------------+
+                |   L2 Frame Size |  Drop Rate  |   Avg Latency (usec) |   Min Latency (usec) |   Max Latency (usec) |
+                +=================+=============+======================+======================+======================+
+                |              64 |   0.0000%   |                   13 |                   10 |                  141 |
+                +-----------------+-------------+----------------------+----------------------+----------------------+
+
+
+              L2 frame size: 64
+
+              Run Config:
+
+                +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
+                |  Direction  |  Requested TX Rate (bps)  |  Actual TX Rate (bps)  |  RX Rate (bps)  |  Requested TX Rate (pps)  |  Actual TX Rate (pps)  |  RX Rate (pps)  |
+                +=============+===========================+========================+=================+===========================+========================+=================+
+                |   Forward   |       336.0000 Mbps       |     336.0000 Mbps      |  336.0000 Mbps  |        500,000 pps        |      500,000 pps       |   500,000 pps   |
+                +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
+                |   Reverse   |       336.0000 Mbps       |     336.0000 Mbps      |  336.0000 Mbps  |        500,000 pps        |      500,000 pps       |   500,000 pps   |
+                +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
+                |    Total    |       672.0000 Mbps       |     672.0000 Mbps      |  672.0000 Mbps  |       1,000,000 pps       |     1,000,000 pps      |  1,000,000 pps  |
+                +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
+
+              Forward Chain Packet Counters and Latency:
+
+                +---------+--------------+--------------+------------+------------+------------+
+                |  Chain  |  TRex.TX.p0  |  TRex.RX.p1  |  Avg lat.  |  Min lat.  |  Max lat.  |
+                +=========+==============+==============+============+============+============+
+                |    0    |   250,000    |   250,000    |  17 usec   |  10 usec   |  138 usec  |
+                +---------+--------------+--------------+------------+------------+------------+
+                |    1    |   250,000    |   250,000    |  17 usec   |  10 usec   |  139 usec  |
+                +---------+--------------+--------------+------------+------------+------------+
+                |  total  |   500,000    |   500,000    |  17 usec   |  10 usec   |  139 usec  |
+                +---------+--------------+--------------+------------+------------+------------+
+
+              Reverse Chain Packet Counters and Latency:
+
+                +---------+--------------+--------------+------------+------------+------------+
+                |  Chain  |  TRex.TX.p1  |  TRex.RX.p0  |  Avg lat.  |  Min lat.  |  Max lat.  |
+                +=========+==============+==============+============+============+============+
+                |    0    |   250,000    |   250,000    |  12 usec   |  10 usec   |  141 usec  |
+                +---------+--------------+--------------+------------+------------+------------+
+                |    1    |   250,000    |   250,000    |  11 usec   |  10 usec   |  132 usec  |
+                +---------+--------------+--------------+------------+------------+------------+
+                |  total  |   500,000    |   500,000    |  12 usec   |  10 usec   |  141 usec  |
+                +---------+--------------+--------------+------------+------------+------------+
+
+Example of configuration file (non shared network)
+--------------------------------------------------
+
+Multi-chain with non shared network (requires 2 segmentation ID per chain), example with 2 chains
+sharing the same 2 SRIOV ports (or PF):
+
+.. code-block:: bash
+
+    flavor:
+       # Number of vCPUs for the flavor
+       vcpus: 4
+       # Memory for the flavor in MB
+       ram: 8192
+       # Size of local disk in GB
+       disk: 0
+       extra_specs:
+          "hw:cpu_policy": dedicated
+    cores: 8
+    sriov: true
+    internal_networks:
+       left:
+            segmentation_id: [3830, 3831]
+            physical_network: phys_sriov0
+       right:
+            segmentation_id: [3832, 3833]
+            physical_network: phys_sriov1
+
+Example of full run 2xPVP non-shared network SR-IOV:
+
+.. code-block:: bash
+
+    2018-12-04 17:15:25,284 INFO -c /tmp/nfvbench/sriov.yaml --rate 1Mpps --duration 1 -scc 2 --no-cleanup
+    2018-12-04 17:15:25,287 INFO Connecting to TRex (127.0.0.1)...
+    2018-12-04 17:15:25,463 INFO Connected to TRex
+    2018-12-04 17:15:25,464 INFO    Port 0: Ethernet Controller XL710 for 40GbE QSFP+ speed=40Gbps mac=3c:fd:fe:b5:3d:70 pci=0000:5e:00.0 driver=net_i40e
+    2018-12-04 17:15:25,464 INFO    Port 1: Ethernet Controller XL710 for 40GbE QSFP+ speed=40Gbps mac=3c:fd:fe:b5:3d:71 pci=0000:5e:00.1 driver=net_i40e
+    2018-12-04 17:15:25,515 INFO Found built-in VM image file nfvbenchvm-0.6.qcow2
+    2018-12-04 17:15:26,457 INFO Created flavor 'nfvbench.medium'
+    2018-12-04 17:15:27,449 INFO Created network: nfvbench-lnet0.
+    2018-12-04 17:15:28,368 INFO Created network: nfvbench-rnet0.
+    2018-12-04 17:15:29,143 INFO Created port nfvbench-loop-vm0-0
+    2018-12-04 17:15:29,626 INFO Security disabled on port nfvbench-loop-vm0-0
+    2018-12-04 17:15:30,636 INFO Created port nfvbench-loop-vm0-1
+    2018-12-04 17:15:31,139 INFO Security disabled on port nfvbench-loop-vm0-1
+    2018-12-04 17:15:31,140 INFO Creating instance nfvbench-loop-vm0 with AZ
+    2018-12-04 17:15:34,893 INFO Created instance nfvbench-loop-vm0 - waiting for placement resolution...
+    2018-12-04 17:15:35,068 INFO Waiting for instance nfvbench-loop-vm0 to become active (retry 1/101)...
+    <snip>
+    2018-12-04 17:16:22,253 INFO Instance nfvbench-loop-vm0 is active and has been placed on nova:charter-compute-4
+    2018-12-04 17:16:23,154 INFO Created network: nfvbench-lnet1.
+    2018-12-04 17:16:23,863 INFO Created network: nfvbench-rnet1.
+    2018-12-04 17:16:24,799 INFO Created port nfvbench-loop-vm1-0
+    2018-12-04 17:16:25,267 INFO Security disabled on port nfvbench-loop-vm1-0
+    2018-12-04 17:16:26,006 INFO Created port nfvbench-loop-vm1-1
+    2018-12-04 17:16:26,612 INFO Security disabled on port nfvbench-loop-vm1-1
+    2018-12-04 17:16:26,612 INFO Creating instance nfvbench-loop-vm1 with AZ nova:charter-compute-4
+    2018-12-04 17:16:27,610 INFO Created instance nfvbench-loop-vm1 on nova:charter-compute-4
+    2018-12-04 17:16:27,610 INFO Instance nfvbench-loop-vm0 is ACTIVE on nova:charter-compute-4
+    2018-12-04 17:16:27,788 INFO Waiting for 1/2 instance to become active (retry 1/100)...
+    <snip>
+
+    2018-12-04 17:17:04,258 INFO Instance nfvbench-loop-vm1 is ACTIVE on nova:charter-compute-4
+    2018-12-04 17:17:04,258 INFO All instances are active
+    2018-12-04 17:17:04,259 INFO Port 0: VLANs [3830, 3831]
+    2018-12-04 17:17:04,259 INFO Port 1: VLANs [3832, 3833]
+    2018-12-04 17:17:04,259 INFO Port 0: dst MAC ['fa:16:3e:ef:f4:b0', 'fa:16:3e:e5:74:cd']
+    2018-12-04 17:17:04,259 INFO Port 1: dst MAC ['fa:16:3e:d6:dc:84', 'fa:16:3e:8e:d9:30']
+    2018-12-04 17:17:04,259 INFO ChainRunner initialized
+    2018-12-04 17:17:04,260 INFO Starting 2xPVP benchmark...
+    2018-12-04 17:17:04,266 INFO Starting traffic generator to ensure end-to-end connectivity
+    2018-12-04 17:17:04,297 INFO Created 2 traffic streams for port 0.
+    2018-12-04 17:17:04,300 INFO Created 2 traffic streams for port 1.
+    2018-12-04 17:17:04,420 INFO Captured unique src mac 0/4, capturing return packets (retry 1/100)...
+    2018-12-04 17:17:06,532 INFO Received packet from mac: fa:16:3e:d6:dc:84 (chain=0, port=1)
+    2018-12-04 17:17:06,532 INFO Received packet from mac: fa:16:3e:ef:f4:b0 (chain=0, port=0)
+    2018-12-04 17:17:06,644 INFO Captured unique src mac 2/4, capturing return packets (retry 2/100)...
+    <snip>
+
+    2018-12-04 17:17:24,337 INFO Received packet from mac: fa:16:3e:8e:d9:30 (chain=1, port=1)
+    2018-12-04 17:17:24,338 INFO Received packet from mac: fa:16:3e:e5:74:cd (chain=1, port=0)
+    2018-12-04 17:17:24,338 INFO End-to-end connectivity established
+    2018-12-04 17:17:24,355 INFO Cleared all existing streams
+    2018-12-04 17:17:24,383 INFO Created 4 traffic streams for port 0.
+    2018-12-04 17:17:24,386 INFO Created 4 traffic streams for port 1.
+    2018-12-04 17:17:24,386 INFO Starting to generate traffic...
+    2018-12-04 17:17:24,386 INFO Running traffic generator
+    2018-12-04 17:17:25,415 INFO TX: 1000004; RX: 1000004; Est. Dropped: 0; Est. Drop rate: 0.0000%
+    2018-12-04 17:17:25,415 INFO ...traffic generating ended.
+    2018-12-04 17:17:25,420 INFO Service chain 'PVP' run completed.
+    2018-12-04 17:17:25,471 INFO Clean up skipped.
+    2018-12-04 17:17:25,508 INFO
+    ========== NFVBench Summary ==========
+    Date: 2018-12-04 17:17:04
+    NFVBench version 3.0.3.dev1
+    Openstack Neutron:
+      vSwitch: OPENVSWITCH
+      Encapsulation: VLAN
+    Benchmarks:
+    > Networks:
+      > Components:
+        > Traffic Generator:
+            Profile: trex-local
+            Tool: TRex
+        > Versions:
+          > Traffic_Generator:
+              build_date: Nov 13 2017
+              version: v2.32
+              built_by: hhaim
+              mode: STL
+              build_time: 10:58:17
+          > CiscoVIM: 2.9.7-17036
+      > Service chain:
+        > PVP:
+          > Traffic:
+              Profile: traffic_profile_64B
+              Bidirectional: True
+              Flow count: 10000
+              Service chains count: 2
+              Compute nodes: [u'nova:charter-compute-4']
+
+                Run Summary:
+
+                  +-----------------+-------------+----------------------+----------------------+----------------------+
+                  |   L2 Frame Size |  Drop Rate  |   Avg Latency (usec) |   Min Latency (usec) |   Max Latency (usec) |
+                  +=================+=============+======================+======================+======================+
+                  |              64 |   0.0000%   |                   18 |                   10 |                  120 |
+                  +-----------------+-------------+----------------------+----------------------+----------------------+
+
+
+                L2 frame size: 64
+
+                Run Config:
+
+                  +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
+                  |  Direction  |  Requested TX Rate (bps)  |  Actual TX Rate (bps)  |  RX Rate (bps)  |  Requested TX Rate (pps)  |  Actual TX Rate (pps)  |  RX Rate (pps)  |
+                  +=============+===========================+========================+=================+===========================+========================+=================+
+                  |   Forward   |       336.0000 Mbps       |     336.0013 Mbps      |  336.0013 Mbps  |        500,000 pps        |      500,002 pps       |   500,002 pps   |
+                  +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
+                  |   Reverse   |       336.0000 Mbps       |     336.0013 Mbps      |  336.0013 Mbps  |        500,000 pps        |      500,002 pps       |   500,002 pps   |
+                  +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
+                  |    Total    |       672.0000 Mbps       |     672.0027 Mbps      |  672.0027 Mbps  |       1,000,000 pps       |     1,000,004 pps      |  1,000,004 pps  |
+                  +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
+
+                Forward Chain Packet Counters and Latency:
+
+                  +---------+--------------+--------------+------------+------------+------------+
+                  |  Chain  |  TRex.TX.p0  |  TRex.RX.p1  |  Avg lat.  |  Min lat.  |  Max lat.  |
+                  +=========+==============+==============+============+============+============+
+                  |    0    |   250,001    |   250,001    |  26 usec   |  10 usec   |  70 usec   |
+                  +---------+--------------+--------------+------------+------------+------------+
+                  |    1    |   250,001    |   250,001    |  11 usec   |  10 usec   |  39 usec   |
+                  +---------+--------------+--------------+------------+------------+------------+
+                  |  total  |   500,002    |   500,002    |  19 usec   |  10 usec   |  70 usec   |
+                  +---------+--------------+--------------+------------+------------+------------+
+
+                Reverse Chain Packet Counters and Latency:
+
+                  +---------+--------------+--------------+------------+------------+------------+
+                  |  Chain  |  TRex.TX.p1  |  TRex.RX.p0  |  Avg lat.  |  Min lat.  |  Max lat.  |
+                  +=========+==============+==============+============+============+============+
+                  |    0    |   250,001    |   250,001    |  19 usec   |  10 usec   |  119 usec  |
+                  +---------+--------------+--------------+------------+------------+------------+
+                  |    1    |   250,001    |   250,001    |  19 usec   |  10 usec   |  120 usec  |
+                  +---------+--------------+--------------+------------+------------+------------+
+                  |  total  |   500,002    |   500,002    |  19 usec   |  10 usec   |  120 usec  |
+                  +---------+--------------+--------------+------------+------------+------------+