Qemu: Mechanism to pass extra qemu cpu features and pin vCPU threads

JIRA: VSPERF-510

This patch is used to implement the mechanism to pass extra qemu
cpu options required to achieve better results/performance. Also
used to implement a mechanism to pin the vCPU threads to another
pCPU core which helps in reducing the noise from housekeeping
thread and packet loss.

Change-Id: Ic40fd47d7c4f5556f7e240c6ca671a0535d06ece
Signed-off-by: Gundarapu Kalyan Reddy <reddyx.gundarapu@intel.com>

3 files changed, 54 insertions, 4 deletions

diff --git a/conf/04_vnf.conf b/conf/04_vnf.conf
index 75f107e8..eafec74b 100644
--- a/conf/04_vnf.conf
+++ b/conf/04_vnf.conf
@@ -142,10 +142,19 @@ GUEST_HUGEPAGES_NR = ['1024']
 # test-pmd requires 2 VM cores
 GUEST_SMP = ['2']
 
+# cpu features to the guest, default options provided to pass all available
+# host processor features to the guest. Also tsc deadline timer for guest,
+# the guest PMU, the invariant TSC options can be provided as well.
+GUEST_CPU_OPTIONS = ['host,migratable=off']
+
 # Host cores to use to affinitize the SMP cores of a QEMU instance
 # For 2 VNFs you may use [(4,5), (6, 7)]
 GUEST_CORE_BINDING = [('#EVAL(6+2*#VMINDEX)', '#EVAL(7+2*#VMINDEX)')]
 
+# pin guest vCPU threads to another pCPU core. This reduces the noise from qemu
+# housekeeping threads. By default the GUEST_THREAD_BINDING will be none.
+GUEST_THREAD_BINDING = [ None ]
+
 # Queues per NIC inside guest for multi-queue configuration, requires switch
 # multi-queue to be enabled for dpdk. Set to 0 for disabled. Can be enabled if
 # using Vanilla OVS without enabling switch multi-queue.
diff --git a/docs/testing/user/userguide/testusage.rst b/docs/testing/user/userguide/testusage.rst
index 043ef928..f8490768 100644
--- a/docs/testing/user/userguide/testusage.rst
+++ b/docs/testing/user/userguide/testusage.rst
@@ -570,6 +570,46 @@ Note: vfio_no_iommu requires kernels equal to or greater than 4.5 and dpdk
 Please refer to the dpdk documents at http://dpdk.org/doc/guides for more
 information on these drivers.
 
+Guest Core and Thread Binding
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+VSPERF provides options to achieve better performance by guest core binding and
+guest vCPU thread binding as well. Core binding is to bind all the qemu threads.
+Thread binding is to bind the house keeping threads to some CPU and vCPU thread to
+some other CPU, this helps to reduce the noise from qemu house keeping threads.
+
+
+.. code-block:: python
+
+   GUEST_CORE_BINDING = [('#EVAL(6+2*#VMINDEX)', '#EVAL(7+2*#VMINDEX)')]
+
+**NOTE** By default the GUEST_THREAD_BINDING will be none, which means same as
+the GUEST_CORE_BINDING, i.e. the vcpu threads are sharing the physical CPUs with
+the house keeping threads. Better performance using vCPU thread binding can be
+achieved by enabling affinity in the custom configuration file.
+
+For example, if an environment requires 32,33 to be core binded and 29,30&31 for
+guest thread binding to achieve better performance.
+
+.. code-block:: python
+
+   VNF_AFFINITIZATION_ON = True
+   GUEST_CORE_BINDING = [('32','33')]
+   GUEST_THREAD_BINDING = [('29', '30', '31')]
+
+Qemu CPU features
+^^^^^^^^^^^^^^^^^
+
+QEMU default to a compatible subset of performance enhancing cpu features.
+To pass all available host processor features to the guest.
+
+.. code-block:: python
+
+   GUEST_CPU_OPTIONS = ['host,migratable=off']
+
+**NOTE** To enhance the performance, cpu features tsc deadline timer for guest,
+the guest PMU, the invariant TSC can be provided in the custom configuration file.
+
 Multi-Queue Configuration
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/vnfs/qemu/qemu.py b/vnfs/qemu/qemu.py
index a9ce176a..b48f7630 100644
--- a/vnfs/qemu/qemu.py
+++ b/vnfs/qemu/qemu.py
@@ -88,7 +88,7 @@ class IVnfQemu(IVnf):
                      S.getValue('TOOLS')['qemu-system'],
                      '-m', S.getValue('GUEST_MEMORY')[self._number],
                      '-smp', str(S.getValue('GUEST_SMP')[self._number]),
-                     '-cpu', 'host,migratable=off',
+                     '-cpu', str(S.getValue('GUEST_CPU_OPTIONS')[self._number]),
                      '-drive', 'if={},file='.format(S.getValue(
                          'GUEST_BOOT_DRIVE_TYPE')[self._number]) +
                      S.getValue('GUEST_IMAGE')[self._number],
@@ -233,12 +233,13 @@ class IVnfQemu(IVnf):
 
         for cpu in range(0, int(S.getValue('GUEST_SMP')[self._number])):
             match = None
+            guest_thread_binding = S.getValue('GUEST_THREAD_BINDING')[self._number]
+            if guest_thread_binding is None:
+                guest_thread_binding = S.getValue('GUEST_CORE_BINDING')[self._number]
             for line in output.decode(cur_locale).split('\n'):
                 match = re.search(thread_id % cpu, line)
                 if match:
-                    self._affinitize_pid(
-                        S.getValue('GUEST_CORE_BINDING')[self._number][cpu],
-                        match.group(1))
+                    self._affinitize_pid(guest_thread_binding[cpu], match.group(1))
                     break
 
             if not match: