.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. SPDX-License-Identifier: CC-BY-4.0
.. (c) Cisco Systems, Inc


Testing SR-IOV
==============

NFVbench supports SR-IOV with the PVP and PVVP packet flow. Most use cases for SR-IOV only require single VNF chains (NxPVP).
Daisy chaining VNFs with SR-IOV (PVVP) requires selecting either SR-IOV for the middle network or a fast vswitch (using the
standard OVS for that purpose works but would be a serious bottleneck)

Instructions below refer to the PVP or PVVP use cases.
For external chains using SR-IOV, select the VLAN tagging option that corresponds to the external chains SR-IOV setting.

Pre-requisites
--------------
To test SR-IOV you need to have compute nodes configured to support one or more SR-IOV interfaces (also knows as PF or physical function)
and you need OpenStack to be configured to support SR-IOV.
You will also need to know:
- the name of the physical networks associated to the SR-IOV interfaces (this is a configuration in Nova compute)
- the VLAN range that can be used on the switch ports that are wired to the SR-IOV ports. Such switch ports are normally configured in trunk mode with a range of VLAN ids enabled on that port

For example, in the case of 2 SR-IOV ports per compute node, 2 physical networks are generally configured in OpenStack with a distinct name.
The VLAN range to use is is also allocated and reserved by the network administrator and in coordination with the corresponding top of rack switch port configuration.


Configuration
-------------
To enable SR-IOV test, you will need to provide the following configuration options to NFVbench (in the configuration file).
This example instructs NFVbench to create the left and right networks of a PVP packet flow to run on 2 SRIOV ports named "phys_sriov0" and "phys_sriov1" using resp. segmentation_id 2000 and 2001:

.. code-block:: bash

    sriov: true
    internal_networks:
       left:
           segmentation_id: 2000
           physical_network: phys_sriov0
       right:
           segmentation_id: 2001
           physical_network: phys_sriov1

The segmentation ID fields must be different.
In the case of PVVP, the middle network also needs to be provisioned properly.
The same physical network can also be shared by the virtual networks but with different segmentation IDs.

Multi-Chaining
--------------
The above configuration works for multi-chaining and shared network ("service_chain_shared_net" set to true).
In that case all VNFs will share the same left and right network/VLAN.

In the case of non shared network ("service_chain_shared_net" set to false), the segmentation_id fields must
contain a list of distinct VLANs to use for each chain. Example of configuration for 3 chains:

.. code-block:: bash

    sriov: true
    internal_networks:
       left:
           segmentation_id: [2000, 2001, 2002]
           physical_network: phys_sriov0
       right:
           segmentation_id: [2100, 2101, 2102]
           physical_network: phys_sriov1

Alternatively it is also possible to specify different physnets per chain:

.. code-block:: bash

    sriov: true
    internal_networks:
       left:
           segmentation_id: [2000, 2001, 2002]
           physical_network: [phys_sriov0, phys_sriov2, phys_sriov4]
       right:
           segmentation_id: [2100, 2101, 2102]
           physical_network: [phys_sriov1, phys_srviov3, phys_sriov5]


NFVbench cores with SR-IOV
--------------------------
The default core count for NFVbench/TRex may not be sufficient for higher throughput line cards (greater than 10Gbps).
This will result in warning messages such as:

.. code-block:: bash

    INFO WARNING: There is a significant difference between requested TX rate (119047618) and actual TX rate (38897379).
    The traffic generator may not have sufficient CPU to achieve the requested TX rate.

In that case it is recommended to try allocating more cores to TRex using the cores property in the configuration
file, for example to set to 8 cores:

.. code-block:: bash

    cores: 8

It is also advisable to increase the number of vcpus in the VMs:


VM Flavor for SR-IOV and NIC NUMA socket placement
--------------------------------------------------

Because SR-IOV throughput uses a lot of CPU in the VM, it is recommended to increase the
vcpu count, for example to 4 vcpus:

.. code-block:: bash

    flavor:
      # Number of vCPUs for the flavor
      vcpus: 4
      # Memory for the flavor in MB
      ram: 8192
      # Size of local disk in GB
      disk: 0
      extra_specs:
          "hw:cpu_policy": dedicated

If the 2 selected ports reside on NICs that are on different NUMA sockets, you will need to explicitly tell Nova to use 2 numa nodes in the flavor used for the VMs in order to satisfy the filters, for example:

.. code-block:: bash

    flavor:
      # Number of vCPUs for the flavor
      vcpus: 4
      # Memory for the flavor in MB
      ram: 8192
      # Size of local disk in GB
      disk: 0
      extra_specs:
          "hw:cpu_policy": dedicated
          "hw:numa_nodes": 2

Failure to do so might cause the VM creation to fail with the Nova error
"Instance creation error: Insufficient compute resources:
Requested instance NUMA topology together with requested PCI devices cannot fit the given host NUMA topology."

Example of configuration file (shared network)
----------------------------------------------

Single chain or multi-chain with shared network (only requires 2 segmentation ID for all chains):

.. code-block:: bash

    flavor:
       # Number of vCPUs for the flavor
       vcpus: 4
       # Memory for the flavor in MB
       ram: 8192
       # Size of local disk in GB
       disk: 0
       extra_specs:
          "hw:cpu_policy": dedicated
    cores: 8
    sriov: true
    internal_networks:
       left:
          segmentation_id: 3830
          physical_network: phys_sriov0
       right:
          segmentation_id: 3831
          physical_network: phys_sriov1

Example of full run 2xPVP shared network SR-IOV:

.. code-block:: bash

    2018-12-03 18:24:07,419 INFO Loading configuration file: /tmp/nfvbench/sriov.yaml
    2018-12-03 18:24:07,423 INFO -c /tmp/nfvbench/sriov.yaml --rate 10Mpps --duration 1 -scc 2 --no-cleanup
    2018-12-03 18:24:07,426 INFO Connecting to TRex (127.0.0.1)...
    2018-12-03 18:24:07,575 INFO Connected to TRex
    2018-12-03 18:24:07,575 INFO    Port 0: Ethernet Controller XL710 for 40GbE QSFP+ speed=40Gbps mac=3c:fd:fe:b5:3d:70 pci=0000:5e:00.0 driver=net_i40e
    2018-12-03 18:24:07,575 INFO    Port 1: Ethernet Controller XL710 for 40GbE QSFP+ speed=40Gbps mac=3c:fd:fe:b5:3d:71 pci=0000:5e:00.1 driver=net_i40e
    2018-12-03 18:24:07,626 INFO Found built-in VM image file nfvbenchvm-0.6.qcow2
    2018-12-03 18:24:09,072 INFO Created flavor 'nfvbench.medium'
    2018-12-03 18:24:10,004 INFO Created network: nfvbench-lnet.
    2018-12-03 18:24:10,837 INFO Created network: nfvbench-rnet.
    2018-12-03 18:24:12,065 INFO Security disabled on port nfvbench-loop-vm0-0
    2018-12-03 18:24:13,425 INFO Security disabled on port nfvbench-loop-vm0-1
    2018-12-03 18:24:13,425 INFO Creating instance nfvbench-loop-vm0 with AZ
    2018-12-03 18:24:16,052 INFO Created instance nfvbench-loop-vm0 - waiting for placement resolution...
    2018-12-03 18:24:16,240 INFO Waiting for instance nfvbench-loop-vm0 to become active (retry 1/101)...
    <snip>
    2018-12-03 18:24:59,266 INFO Waiting for instance nfvbench-loop-vm0 to become active (retry 21/101)...
    2018-12-03 18:25:01,427 INFO Instance nfvbench-loop-vm0 is active and has been placed on nova:charter-compute-5
    2018-12-03 18:25:02,819 INFO Security disabled on port nfvbench-loop-vm1-0
    2018-12-03 18:25:04,198 INFO Security disabled on port nfvbench-loop-vm1-1
    2018-12-03 18:25:04,199 INFO Creating instance nfvbench-loop-vm1 with AZ nova:charter-compute-5
    2018-12-03 18:25:05,032 INFO Created instance nfvbench-loop-vm1 on nova:charter-compute-5
    2018-12-03 18:25:05,033 INFO Instance nfvbench-loop-vm0 is ACTIVE on nova:charter-compute-5
    2018-12-03 18:25:05,212 INFO Waiting for 1/2 instance to become active (retry 1/100)...
    <snip>
    2018-12-03 18:25:48,531 INFO Waiting for 1/2 instance to become active (retry 21/100)...
    2018-12-03 18:25:50,677 INFO Instance nfvbench-loop-vm1 is ACTIVE on nova:charter-compute-5
    2018-12-03 18:25:50,677 INFO All instances are active
    2018-12-03 18:25:50,677 INFO Port 0: VLANs [3830, 3830]
    2018-12-03 18:25:50,677 INFO Port 1: VLANs [3831, 3831]
    2018-12-03 18:25:50,677 INFO Port 0: dst MAC ['fa:16:3e:de:4e:54', 'fa:16:3e:7a:26:2b']
    2018-12-03 18:25:50,677 INFO Port 1: dst MAC ['fa:16:3e:6c:bb:cd', 'fa:16:3e:e0:48:45']
    2018-12-03 18:25:50,678 INFO ChainRunner initialized
    2018-12-03 18:25:50,678 INFO Starting 2xPVP benchmark...
    2018-12-03 18:25:50,683 INFO Starting traffic generator to ensure end-to-end connectivity
    2018-12-03 18:25:50,698 INFO Created 2 traffic streams for port 0.
    2018-12-03 18:25:50,700 INFO Created 2 traffic streams for port 1.
    2018-12-03 18:25:50,821 INFO Captured unique src mac 0/4, capturing return packets (retry 1/100)...
    2018-12-03 18:25:52,944 INFO Received packet from mac: fa:16:3e:de:4e:54 (chain=0, port=0)
    2018-12-03 18:25:52,945 INFO Received packet from mac: fa:16:3e:6c:bb:cd (chain=0, port=1)
    2018-12-03 18:25:53,077 INFO Captured unique src mac 2/4, capturing return packets (retry 2/100)...
    <snip>
    2018-12-03 18:26:10,798 INFO End-to-end connectivity established
    2018-12-03 18:26:10,816 INFO Cleared all existing streams
    2018-12-03 18:26:10,846 INFO Created 4 traffic streams for port 0.
    2018-12-03 18:26:10,849 INFO Created 4 traffic streams for port 1.
    2018-12-03 18:26:10,849 INFO Starting to generate traffic...
    2018-12-03 18:26:10,850 INFO Running traffic generator
    2018-12-03 18:26:11,877 INFO TX: 10000004; RX: 9999999; Est. Dropped: 5; Est. Drop rate: 0.0000%
    2018-12-03 18:26:11,877 INFO ...traffic generating ended.
    2018-12-03 18:26:11,882 INFO Service chain 'PVP' run completed.
    2018-12-03 18:26:11,936 INFO Clean up skipped.
    2018-12-03 18:26:11,969 INFO
    ========== NFVBench Summary ==========
    Date: 2018-12-03 18:25:50
    NFVBench version 3.0.3.dev1
    Openstack Neutron:
      vSwitch: OPENVSWITCH
      Encapsulation: VLAN
    Benchmarks:
    > Networks:
      > Components:
        > Traffic Generator:
            Profile: trex-local
            Tool: TRex
        > Versions:
          > Traffic_Generator:
              build_date: Nov 13 2017
              version: v2.32
              built_by: hhaim
              mode: STL
              build_time: 10:58:17
          > CiscoVIM: 2.9.7-17036
      > Service chain:
        > PVP:
          > Traffic:
              Profile: traffic_profile_64B
              Bidirectional: True
              Flow count: 10000
              Service chains count: 2
              Compute nodes: [u'nova:charter-compute-5']

              Run Summary:

                +-----------------+-------------+----------------------+----------------------+----------------------+
                |   L2 Frame Size |  Drop Rate  |   Avg Latency (usec) |   Min Latency (usec) |   Max Latency (usec) |
                +=================+=============+======================+======================+======================+
                |              64 |   0.0000%   |                   13 |                   10 |                  141 |
                +-----------------+-------------+----------------------+----------------------+----------------------+


              L2 frame size: 64

              Run Config:

                +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
                |  Direction  |  Requested TX Rate (bps)  |  Actual TX Rate (bps)  |  RX Rate (bps)  |  Requested TX Rate (pps)  |  Actual TX Rate (pps)  |  RX Rate (pps)  |
                +=============+===========================+========================+=================+===========================+========================+=================+
                |   Forward   |       336.0000 Mbps       |     336.0000 Mbps      |  336.0000 Mbps  |        500,000 pps        |      500,000 pps       |   500,000 pps   |
                +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
                |   Reverse   |       336.0000 Mbps       |     336.0000 Mbps      |  336.0000 Mbps  |        500,000 pps        |      500,000 pps       |   500,000 pps   |
                +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
                |    Total    |       672.0000 Mbps       |     672.0000 Mbps      |  672.0000 Mbps  |       1,000,000 pps       |     1,000,000 pps      |  1,000,000 pps  |
                +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+

              Forward Chain Packet Counters and Latency:

                +---------+--------------+--------------+------------+------------+------------+
                |  Chain  |  TRex.TX.p0  |  TRex.RX.p1  |  Avg lat.  |  Min lat.  |  Max lat.  |
                +=========+==============+==============+============+============+============+
                |    0    |   250,000    |   250,000    |  17 usec   |  10 usec   |  138 usec  |
                +---------+--------------+--------------+------------+------------+------------+
                |    1    |   250,000    |   250,000    |  17 usec   |  10 usec   |  139 usec  |
                +---------+--------------+--------------+------------+------------+------------+
                |  total  |   500,000    |   500,000    |  17 usec   |  10 usec   |  139 usec  |
                +---------+--------------+--------------+------------+------------+------------+

              Reverse Chain Packet Counters and Latency:

                +---------+--------------+--------------+------------+------------+------------+
                |  Chain  |  TRex.TX.p1  |  TRex.RX.p0  |  Avg lat.  |  Min lat.  |  Max lat.  |
                +=========+==============+==============+============+============+============+
                |    0    |   250,000    |   250,000    |  12 usec   |  10 usec   |  141 usec  |
                +---------+--------------+--------------+------------+------------+------------+
                |    1    |   250,000    |   250,000    |  11 usec   |  10 usec   |  132 usec  |
                +---------+--------------+--------------+------------+------------+------------+
                |  total  |   500,000    |   500,000    |  12 usec   |  10 usec   |  141 usec  |
                +---------+--------------+--------------+------------+------------+------------+

Example of configuration file (non shared network)
--------------------------------------------------

Multi-chain with non shared network (requires 2 segmentation ID per chain), example with 2 chains
sharing the same 2 SRIOV ports (or PF):

.. code-block:: bash

    flavor:
       # Number of vCPUs for the flavor
       vcpus: 4
       # Memory for the flavor in MB
       ram: 8192
       # Size of local disk in GB
       disk: 0
       extra_specs:
          "hw:cpu_policy": dedicated
    cores: 8
    sriov: true
    internal_networks:
       left:
            segmentation_id: [3830, 3831]
            physical_network: phys_sriov0
       right:
            segmentation_id: [3832, 3833]
            physical_network: phys_sriov1

Example of full run 2xPVP non-shared network SR-IOV:

.. code-block:: bash

    2018-12-04 17:15:25,284 INFO -c /tmp/nfvbench/sriov.yaml --rate 1Mpps --duration 1 -scc 2 --no-cleanup
    2018-12-04 17:15:25,287 INFO Connecting to TRex (127.0.0.1)...
    2018-12-04 17:15:25,463 INFO Connected to TRex
    2018-12-04 17:15:25,464 INFO    Port 0: Ethernet Controller XL710 for 40GbE QSFP+ speed=40Gbps mac=3c:fd:fe:b5:3d:70 pci=0000:5e:00.0 driver=net_i40e
    2018-12-04 17:15:25,464 INFO    Port 1: Ethernet Controller XL710 for 40GbE QSFP+ speed=40Gbps mac=3c:fd:fe:b5:3d:71 pci=0000:5e:00.1 driver=net_i40e
    2018-12-04 17:15:25,515 INFO Found built-in VM image file nfvbenchvm-0.6.qcow2
    2018-12-04 17:15:26,457 INFO Created flavor 'nfvbench.medium'
    2018-12-04 17:15:27,449 INFO Created network: nfvbench-lnet0.
    2018-12-04 17:15:28,368 INFO Created network: nfvbench-rnet0.
    2018-12-04 17:15:29,143 INFO Created port nfvbench-loop-vm0-0
    2018-12-04 17:15:29,626 INFO Security disabled on port nfvbench-loop-vm0-0
    2018-12-04 17:15:30,636 INFO Created port nfvbench-loop-vm0-1
    2018-12-04 17:15:31,139 INFO Security disabled on port nfvbench-loop-vm0-1
    2018-12-04 17:15:31,140 INFO Creating instance nfvbench-loop-vm0 with AZ
    2018-12-04 17:15:34,893 INFO Created instance nfvbench-loop-vm0 - waiting for placement resolution...
    2018-12-04 17:15:35,068 INFO Waiting for instance nfvbench-loop-vm0 to become active (retry 1/101)...
    <snip>
    2018-12-04 17:16:22,253 INFO Instance nfvbench-loop-vm0 is active and has been placed on nova:charter-compute-4
    2018-12-04 17:16:23,154 INFO Created network: nfvbench-lnet1.
    2018-12-04 17:16:23,863 INFO Created network: nfvbench-rnet1.
    2018-12-04 17:16:24,799 INFO Created port nfvbench-loop-vm1-0
    2018-12-04 17:16:25,267 INFO Security disabled on port nfvbench-loop-vm1-0
    2018-12-04 17:16:26,006 INFO Created port nfvbench-loop-vm1-1
    2018-12-04 17:16:26,612 INFO Security disabled on port nfvbench-loop-vm1-1
    2018-12-04 17:16:26,612 INFO Creating instance nfvbench-loop-vm1 with AZ nova:charter-compute-4
    2018-12-04 17:16:27,610 INFO Created instance nfvbench-loop-vm1 on nova:charter-compute-4
    2018-12-04 17:16:27,610 INFO Instance nfvbench-loop-vm0 is ACTIVE on nova:charter-compute-4
    2018-12-04 17:16:27,788 INFO Waiting for 1/2 instance to become active (retry 1/100)...
    <snip>

    2018-12-04 17:17:04,258 INFO Instance nfvbench-loop-vm1 is ACTIVE on nova:charter-compute-4
    2018-12-04 17:17:04,258 INFO All instances are active
    2018-12-04 17:17:04,259 INFO Port 0: VLANs [3830, 3831]
    2018-12-04 17:17:04,259 INFO Port 1: VLANs [3832, 3833]
    2018-12-04 17:17:04,259 INFO Port 0: dst MAC ['fa:16:3e:ef:f4:b0', 'fa:16:3e:e5:74:cd']
    2018-12-04 17:17:04,259 INFO Port 1: dst MAC ['fa:16:3e:d6:dc:84', 'fa:16:3e:8e:d9:30']
    2018-12-04 17:17:04,259 INFO ChainRunner initialized
    2018-12-04 17:17:04,260 INFO Starting 2xPVP benchmark...
    2018-12-04 17:17:04,266 INFO Starting traffic generator to ensure end-to-end connectivity
    2018-12-04 17:17:04,297 INFO Created 2 traffic streams for port 0.
    2018-12-04 17:17:04,300 INFO Created 2 traffic streams for port 1.
    2018-12-04 17:17:04,420 INFO Captured unique src mac 0/4, capturing return packets (retry 1/100)...
    2018-12-04 17:17:06,532 INFO Received packet from mac: fa:16:3e:d6:dc:84 (chain=0, port=1)
    2018-12-04 17:17:06,532 INFO Received packet from mac: fa:16:3e:ef:f4:b0 (chain=0, port=0)
    2018-12-04 17:17:06,644 INFO Captured unique src mac 2/4, capturing return packets (retry 2/100)...
    <snip>

    2018-12-04 17:17:24,337 INFO Received packet from mac: fa:16:3e:8e:d9:30 (chain=1, port=1)
    2018-12-04 17:17:24,338 INFO Received packet from mac: fa:16:3e:e5:74:cd (chain=1, port=0)
    2018-12-04 17:17:24,338 INFO End-to-end connectivity established
    2018-12-04 17:17:24,355 INFO Cleared all existing streams
    2018-12-04 17:17:24,383 INFO Created 4 traffic streams for port 0.
    2018-12-04 17:17:24,386 INFO Created 4 traffic streams for port 1.
    2018-12-04 17:17:24,386 INFO Starting to generate traffic...
    2018-12-04 17:17:24,386 INFO Running traffic generator
    2018-12-04 17:17:25,415 INFO TX: 1000004; RX: 1000004; Est. Dropped: 0; Est. Drop rate: 0.0000%
    2018-12-04 17:17:25,415 INFO ...traffic generating ended.
    2018-12-04 17:17:25,420 INFO Service chain 'PVP' run completed.
    2018-12-04 17:17:25,471 INFO Clean up skipped.
    2018-12-04 17:17:25,508 INFO
    ========== NFVBench Summary ==========
    Date: 2018-12-04 17:17:04
    NFVBench version 3.0.3.dev1
    Openstack Neutron:
      vSwitch: OPENVSWITCH
      Encapsulation: VLAN
    Benchmarks:
    > Networks:
      > Components:
        > Traffic Generator:
            Profile: trex-local
            Tool: TRex
        > Versions:
          > Traffic_Generator:
              build_date: Nov 13 2017
              version: v2.32
              built_by: hhaim
              mode: STL
              build_time: 10:58:17
          > CiscoVIM: 2.9.7-17036
      > Service chain:
        > PVP:
          > Traffic:
              Profile: traffic_profile_64B
              Bidirectional: True
              Flow count: 10000
              Service chains count: 2
              Compute nodes: [u'nova:charter-compute-4']

                Run Summary:

                  +-----------------+-------------+----------------------+----------------------+----------------------+
                  |   L2 Frame Size |  Drop Rate  |   Avg Latency (usec) |   Min Latency (usec) |   Max Latency (usec) |
                  +=================+=============+======================+======================+======================+
                  |              64 |   0.0000%   |                   18 |                   10 |                  120 |
                  +-----------------+-------------+----------------------+----------------------+----------------------+


                L2 frame size: 64

                Run Config:

                  +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
                  |  Direction  |  Requested TX Rate (bps)  |  Actual TX Rate (bps)  |  RX Rate (bps)  |  Requested TX Rate (pps)  |  Actual TX Rate (pps)  |  RX Rate (pps)  |
                  +=============+===========================+========================+=================+===========================+========================+=================+
                  |   Forward   |       336.0000 Mbps       |     336.0013 Mbps      |  336.0013 Mbps  |        500,000 pps        |      500,002 pps       |   500,002 pps   |
                  +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
                  |   Reverse   |       336.0000 Mbps       |     336.0013 Mbps      |  336.0013 Mbps  |        500,000 pps        |      500,002 pps       |   500,002 pps   |
                  +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
                  |    Total    |       672.0000 Mbps       |     672.0027 Mbps      |  672.0027 Mbps  |       1,000,000 pps       |     1,000,004 pps      |  1,000,004 pps  |
                  +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+

                Forward Chain Packet Counters and Latency:

                  +---------+--------------+--------------+------------+------------+------------+
                  |  Chain  |  TRex.TX.p0  |  TRex.RX.p1  |  Avg lat.  |  Min lat.  |  Max lat.  |
                  +=========+==============+==============+============+============+============+
                  |    0    |   250,001    |   250,001    |  26 usec   |  10 usec   |  70 usec   |
                  +---------+--------------+--------------+------------+------------+------------+
                  |    1    |   250,001    |   250,001    |  11 usec   |  10 usec   |  39 usec   |
                  +---------+--------------+--------------+------------+------------+------------+
                  |  total  |   500,002    |   500,002    |  19 usec   |  10 usec   |  70 usec   |
                  +---------+--------------+--------------+------------+------------+------------+

                Reverse Chain Packet Counters and Latency:

                  +---------+--------------+--------------+------------+------------+------------+
                  |  Chain  |  TRex.TX.p1  |  TRex.RX.p0  |  Avg lat.  |  Min lat.  |  Max lat.  |
                  +=========+==============+==============+============+============+============+
                  |    0    |   250,001    |   250,001    |  19 usec   |  10 usec   |  119 usec  |
                  +---------+--------------+--------------+------------+------------+------------+
                  |    1    |   250,001    |   250,001    |  19 usec   |  10 usec   |  120 usec  |
                  +---------+--------------+--------------+------------+------------+------------+
                  |  total  |   500,002    |   500,002    |  19 usec   |  10 usec   |  120 usec  |
                  +---------+--------------+--------------+------------+------------+------------+