From 6207f53e22b1c10f3ef65841f97fe52db75a8bbb Mon Sep 17 00:00:00 2001 From: raghu Date: Wed, 10 Feb 2016 13:55:50 -0800 Subject: Added license information Upstream Status : Not Applicable For detalied license information please refer to https://wiki.opnfv.org/documentation#licencing_your_documentation Change-Id: Iacba57a9162b25abc474dbac236b4dee5f27544c --- docs/all/environment-setup.rst | 4 ++ docs/all/index.rst | 6 ++- docs/all/live_migration.rst | 4 ++ docs/all/tuning.rst | 101 +++++++++++++++++++++++++++++++++++++++++ docs/all/tunning.rst | 97 --------------------------------------- 5 files changed, 114 insertions(+), 98 deletions(-) create mode 100644 docs/all/tuning.rst delete mode 100644 docs/all/tunning.rst diff --git a/docs/all/environment-setup.rst b/docs/all/environment-setup.rst index b60deb7e0..e3814310a 100644 --- a/docs/all/environment-setup.rst +++ b/docs/all/environment-setup.rst @@ -1,3 +1,7 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) + Low Latency Environment ======================= diff --git a/docs/all/index.rst b/docs/all/index.rst index fb1b137d7..7f5f7a694 100644 --- a/docs/all/index.rst +++ b/docs/all/index.rst @@ -1,3 +1,7 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) + =============== KVM4NFV project =============== @@ -40,5 +44,5 @@ Setup Guides :maxdepth: 2 environment-setup - tunning + tuning live_migration diff --git a/docs/all/live_migration.rst b/docs/all/live_migration.rst index 51265cc8d..4af19b6f4 100644 --- a/docs/all/live_migration.rst +++ b/docs/all/live_migration.rst @@ -1,3 +1,7 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) + Fast Live Migration =================== diff --git a/docs/all/tuning.rst b/docs/all/tuning.rst new file mode 100644 index 000000000..760861b8b --- /dev/null +++ b/docs/all/tuning.rst @@ -0,0 +1,101 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) + +Low Latency Tunning Suggestion +============================== + +The correct configuration is critical for improving the NFV performance/latency. +Even working on the same codebase, configurations can cause wildly different +performance/latency results. + +There are many combinations of configurations, from hardware configuration to +Operating System configuration and application level configuration. And there +is no one simple configuration that works for every case. To tune a specific +scenario, it's important to know the behaviors of different configurations and +their impact. + +Platform Configuration +---------------------- + +Some hardware features can be configured through firmware interface(like BIOS) +but others may not be configurable (e.g. SMI on most platforms). + +* **Power management:** + Most power management related features save power at the + expensive of latency. These features include: Intel®Turbo Boost Technology, + Enhanced Intel®SpeedStep, Processor C state and P state. Normally they should + be disabled but, depending on the real-time application design and latency + requirements, there might be some features that can be enabled if the impact on + deterministic execution of the workload is small. + +* **Hyper-Threading:** + The logic cores that share resource with other logic cores can introduce + latency so the recommendation is to disable this feature for realtime use + cases. + +* **Legacy USB Support/Port 60/64 Emulation:** + These features involve some emulation in firmware and can introduce random + latency. It is recommended that they are disabled. + +* **SMI (System Management Interrupt):** + SMI runs outside of the kernel code and can potentially cause + latency. It is a pity there is no simple way to disable it. Some vendors may + provide related switches in BIOS but most machines do not have this capability. + +Operating System Configuration +------------------------------ + +* **CPU isolation:** + To achieve deterministic latency, dedicated CPUs should be allocated for + realtime application. This can be achieved by isolating cpus from kernel + scheduler. Please refer to + http://lxr.free-electrons.com/source/Documentation/kernel-parameters.txt#L1608 + for more information. + +* **Memory allocation:** + Memory shoud be reserved for realtime applications and usually hugepage should + be used to reduce page fauts/TLB misses. + +* **IRQ affinity:** + All the non-realtime IRQs should be affinitized to non realtime CPUs to + reduce the impact on realtime CPUs. Some OS distributions contain an irqbalance + daemon which balances the IRQs among all the cores dynamically. It should be + disabled as well. + +* **Device assignment for VM:** + If a device is used in a VM, then device passthrough is desirable. In this case, + the IOMMU should be enabled. + +* **Tickless:** + Frequent clock ticks cause latency. CONFIG_NOHZ_FULL should be enabled in the + linux kernel. With CONFIG_NOHZ_FULL, the physical CPU will trigger many fewer + clock tick interrupts(currently, 1 tick per second). This can reduce latency + because each host timer interrupt triggers a VM exit from guest to host which + causes performance/latency impacts. + +* **TSC:** + Mark TSC clock source as reliable. A TSC clock source that seems to be + unreliable causes the kernel to continuously enable the clock source watchdog + to check if TSC frequency is still correct. On recent Intel platforms with + Constant TSC/Invariant TSC/Synchronized TSC, the TSC is reliable so the + watchdog is useless but cause latency. + +* **Idle:** + The poll option forces a polling idle loop that can slightly improve the + performance of waking up an idle CPU. + +* **RCU_NOCB:** + RCU is a kernel synchronization mechanism. Refer to + http://lxr.free-electrons.com/source/Documentation/RCU/whatisRCU.txt for more + information. With RCU_NOCB, the impact from RCU to the VNF will be reduced. + +* **Disable the RT throttling:** + RT Throttling is a Linux kernel mechanism that + occurs when a process or thread uses 100% of the core, leaving no resources for + the Linux scheduler to execute the kernel/housekeeping tasks. RT Throttling + increases the latency so should be disabled. + +* **NUMA configuration:** + To achieve the best latency. CPU/Memory and device allocated for realtime + application/VM should be in the same NUMA node. diff --git a/docs/all/tunning.rst b/docs/all/tunning.rst deleted file mode 100644 index 337255328..000000000 --- a/docs/all/tunning.rst +++ /dev/null @@ -1,97 +0,0 @@ -Low Latency Tunning Suggestion -============================== - -The correct configuration is critical for improving the NFV performance/latency. -Even working on the same codebase, configurations can cause wildly different -performance/latency results. - -There are many combinations of configurations, from hardware configuration to -Operating System configuration and application level configuration. And there -is no one simple configuration that works for every case. To tune a specific -scenario, it's important to know the behaviors of different configurations and -their impact. - -Platform Configuration ----------------------- - -Some hardware features can be configured through firmware interface(like BIOS) -but others may not be configurable (e.g. SMI on most platforms). - -* **Power management:** - Most power management related features save power at the - expensive of latency. These features include: Intel®Turbo Boost Technology, - Enhanced Intel®SpeedStep, Processor C state and P state. Normally they should - be disabled but, depending on the real-time application design and latency - requirements, there might be some features that can be enabled if the impact on - deterministic execution of the workload is small. - -* **Hyper-Threading:** - The logic cores that share resource with other logic cores can introduce - latency so the recommendation is to disable this feature for realtime use - cases. - -* **Legacy USB Support/Port 60/64 Emulation:** - These features involve some emulation in firmware and can introduce random - latency. It is recommended that they are disabled. - -* **SMI (System Management Interrupt):** - SMI runs outside of the kernel code and can potentially cause - latency. It is a pity there is no simple way to disable it. Some vendors may - provide related switches in BIOS but most machines do not have this capability. - -Operating System Configuration ------------------------------- - -* **CPU isolation:** - To achieve deterministic latency, dedicated CPUs should be allocated for - realtime application. This can be achieved by isolating cpus from kernel - scheduler. Please refer to - http://lxr.free-electrons.com/source/Documentation/kernel-parameters.txt#L1608 - for more information. - -* **Memory allocation:** - Memory shoud be reserved for realtime applications and usually hugepage should - be used to reduce page fauts/TLB misses. - -* **IRQ affinity:** - All the non-realtime IRQs should be affinitized to non realtime CPUs to - reduce the impact on realtime CPUs. Some OS distributions contain an irqbalance - daemon which balances the IRQs among all the cores dynamically. It should be - disabled as well. - -* **Device assignment for VM:** - If a device is used in a VM, then device passthrough is desirable. In this case, - the IOMMU should be enabled. - -* **Tickless:** - Frequent clock ticks cause latency. CONFIG_NOHZ_FULL should be enabled in the - linux kernel. With CONFIG_NOHZ_FULL, the physical CPU will trigger many fewer - clock tick interrupts(currently, 1 tick per second). This can reduce latency - because each host timer interrupt triggers a VM exit from guest to host which - causes performance/latency impacts. - -* **TSC:** - Mark TSC clock source as reliable. A TSC clock source that seems to be - unreliable causes the kernel to continuously enable the clock source watchdog - to check if TSC frequency is still correct. On recent Intel platforms with - Constant TSC/Invariant TSC/Synchronized TSC, the TSC is reliable so the - watchdog is useless but cause latency. - -* **Idle:** - The poll option forces a polling idle loop that can slightly improve the - performance of waking up an idle CPU. - -* **RCU_NOCB:** - RCU is a kernel synchronization mechanism. Refer to - http://lxr.free-electrons.com/source/Documentation/RCU/whatisRCU.txt for more - information. With RCU_NOCB, the impact from RCU to the VNF will be reduced. - -* **Disable the RT throttling:** - RT Throttling is a Linux kernel mechanism that - occurs when a process or thread uses 100% of the core, leaving no resources for - the Linux scheduler to execute the kernel/housekeeping tasks. RT Throttling - increases the latency so should be disabled. - -* **NUMA configuration:** - To achieve the best latency. CPU/Memory and device allocated for realtime - application/VM should be in the same NUMA node. -- cgit 1.2.3-korg