Add the documentation for kvm4nfv project

Adding documentation to kvm4nfv project based on
https://wiki.opnfv.org/documentation/tools and the contents mostly comes
from https://wiki.opnfv.org/nfv-kvm,
https://wiki.opnfv.org/nfv-kvm-tuning and
https://wiki.opnfv.org/nfv-kvm-test.

Change-Id: If321221724ec9b76db065af7cdab97ce981be740
Signed-off-by: Yunhong Jiang <yunhong.jiang@linux.intel.com>

6 files changed, 288 insertions, 150 deletions

diff --git a/docs/all/environment-setup.rst b/docs/all/environment-setup.rst
new file mode 100644
index 000000000..8448e9650
--- /dev/null
+++ b/docs/all/environment-setup.rst
@@ -0,0 +1,147 @@
+Low Latency Environment
+=======================
+
+Achieving low latency with the KVM4NFV project requires setting up a special
+test environment. This environment includes the BIOS settings, kernel
+configuration, kernel parameters and the run-time environment.
+
+Hardware Environment Description
+--------------------------------
+
+BIOS setup plays an important role in achieving real-time latency. A collection
+of relevant settings, used on the platform where the baseline performance data
+was collected, is detailed below:
+
+CPU Features
+~~~~~~~~~~~~
+
+Some special CPU features like TSC-deadline timer, invariant TSC and Process posted
+interrupts etc are helpful for latency reduction.
+
+Below is the CPU information on the baseline test platform.
+::
+        processor       : 35
+        vendor_id       : GenuineIntel
+        cpu family      : 6
+        model           : 63
+        model name      : Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
+        stepping        : 2
+        microcode       : 0x2d
+        cpu MHz         : 2294.795
+        cache size      : 46080 KB
+        physical id     : 1
+        siblings        : 18
+        core id         : 27
+        cpu cores       : 18
+        apicid          : 118
+        initial apicid  : 118
+        fpu             : yes
+        fpu_exception   : yes
+        cpuid level     : 15
+        wp              : yes
+        flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
+                          mca cmov pat pse36 clflush dts acpi mmx fxsr sse
+                          sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm
+                          constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc
+                          aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2
+                          ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt
+                          tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm arat epb
+                          pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase
+                          tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc
+                          cqm_occup_llcbugs
+        bogomips        : 4595.54
+        clflush size    : 64
+        cache_alignment : 64
+        address sizes   : 46 bits physical, 48 bits virtual
+        power management:
+
+CPU Topology
+~~~~~~~~~~~~
+
+NUMA topology is also important for latency reducation.
+
+Below is the CPU topology on the baseline test platform.
+::
+        [nfv@otcnfv02 ~]$ lscpu
+        Architecture:          x86_64
+        CPU op-mode(s):        32-bit, 64-bit
+        Byte Order:            Little Endian
+        CPU(s):                36
+        On-line CPU(s) list:   0-35
+        Thread(s) per core:    1
+        Core(s) per socket:    18
+        Socket(s):             2
+        NUMA node(s):          2
+        Vendor ID:             GenuineIntel
+        CPU family:            6
+        Model:                 63
+        Model name:            Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
+        Stepping:              2
+        CPU MHz:               2294.795
+        BogoMIPS:              4595.54
+        Virtualization:        VT-x
+        L1d cache:             32K
+        L1i cache:             32K
+        L2 cache:              256K
+        L3 cache:              46080K
+        NUMA node0 CPU(s):     0-17
+        NUMA node1 CPU(s):     18-35
+
+BIOS Setup
+~~~~~~~~~~
+
+Careful BIOS setup is important in achieving real time latency. Different
+platforms have different BIOS setups, below are the important BIOS settings on
+the platform used to collect the baseline performance data.
+::
+        CPU Power and Performance <Performance>
+        CPU C-State <Disabled>
+        C1E Autopromote <Disabled>
+        Processor C3 <Disabled>
+        Processor C6 <Disabled>
+        Select Memory RAS <Maximum Performance>
+        NUMA Optimized <Enabled>
+        Cluster-on-Die <Disabled>
+        Patrol Scrub <Disabled>
+        Demand Scrub <Disabled>
+        Correctable Error <10>
+        Intel(R) Hyper-Threading <Disabled>
+        Active Processor Cores <All>
+        Execute Disable Bit <Enabled>
+        Intel(R) Virtualization Technology <Enabled>
+        Intel(R) TXT <Disabled>
+        Enhanced Error Containment Mode <Disabled>
+        USB Controller <Enabled>
+        USB 3.0 Controller <Auto>
+        Legacy USB Support <Disabled>
+        Port 60/64 Emulation <Disabled>
+
+Software Environment Setup
+--------------------------
+Both the host and the guest environment need to be configured properly to
+reduce latency variations.  Below are some suggested kernel configurations.
+The ci/envs/ directory gives etailed implementation on how to setup the
+environment.
+
+Kernel Parameter
+~~~~~~~~~~~~~~~~
+
+Please check the default kernel configuration in the source code at:
+kernel/arch/x86/configs/opnfv.config.
+
+Below is host kernel boot line example:
+::
+        isolcpus=11-15,31-35 nohz_full=11-15,31-35 rcu_nocbs=11-15,31-35 iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G mce=off idle=poll intel_pstate=disable processor.max_cstate=1 pcie_asmp=off tsc=reliable
+
+Below is guest kernel boot line example
+::
+ isolcpus=1 nohz_full=1 rcu_nocbs=1 mce=off idle=poll default_hugepagesz=1G hugepagesz=1G
+
+Please refer to :doc:`tunning` for more explanation.
+
+Run-time Environment Setup
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Not only are special kernel parameters needed but a special run-time
+environment is also required. Please refer to :doc:`tunning` for more
+explanation.
diff --git a/docs/all/index.rst b/docs/all/index.rst
new file mode 100644
index 000000000..fed745ee6
--- /dev/null
+++ b/docs/all/index.rst
@@ -0,0 +1,44 @@
+===============
+KVM4NFV project
+===============
+
+Welcome to KVM4NFV_ project!
+
+
+
+.. _KVM4NFV: https://wiki.opnfv.org/nfv-kvm
+
+Contents:
+
+KVM4NFV Project Description
+===========================
+
+The NFV hypervisors provide crucial functionality in the NFV Infrastructure
+(NFVI). The existing hypervisors, however, are not necessarily designed or
+targeted to meet the requirements for the NFVI, and we need to make
+collaborative efforts toward enabling the NFV features.
+
+The KVM4NFV project focuses on the KVM hypervisor to enhance it for NFV, by
+looking at the following areas
+
++ Minimal Interrupt latency variation for data plane VNFs
+    * Minimal Timing Variation for Timing correctness of real-time VNFs
+    * Minimal packet latency variation for data-plane VNFs
++ Fast live migration
+
+hile these items require software development and/or specific hardware features
+there are also some adjustments that need to be made to system configuration
+information, like hardware, BIOS, OS, etc.
+
+.. toctree::
+        :numbered:
+        :maxdepth: 1
+
+Setup Guides
+============
+.. toctree::
+        :maxdepth: 2
+
+        environment-setup
+        tunning
+
diff --git a/docs/all/tunning.rst b/docs/all/tunning.rst
new file mode 100644
index 000000000..075bf0488
--- /dev/null
+++ b/docs/all/tunning.rst
@@ -0,0 +1,97 @@
+Low Latency Tunning Suggestion
+==============================
+
+The correct configuration is critical for improving the NFV performance/latency.
+Even working on the same codebase, configurations can cause wildly different
+performance/latency results.
+
+There are many combinations of configurations, from hardware configuration to
+Operating System configuration and application level configuration. And there
+is no one simple configuration that works for every case. To tune a specific
+scenario, it's important to know the behaviors of different configurations and
+their impact.
+
+Platform Configuration
+----------------------
+
+Some hardware features can be configured through firmware interface(like BIOS)
+but others may not be configurable (e.g. SMI on most platforms).
+
+* **Power management:**
+  Most power management related features save power at the
+  expensive of latency. These features include: Intel®Turbo Boost Technology,
+  Enhanced Intel®SpeedStep, Processor C state and P state. Normally they should
+  be disabled but, depending on the real-time application design and latency
+  requirements, there might be some features can be enabled if the impact on
+  deterministic execution of the workload is small.
+
+* **Hyper-Threading:**
+  The logic cores that share resource with other logic cores can introduce
+  latency so the recommendation is to disable this feature for realtime use
+  cases.
+
+* **Legacy USB Support/Port 60/64 Emulation:**
+  These features involve some emulation in firmware and can introduce random
+  latency. It is recommended that they are disabled.
+
+* **SMI (System Management Interrupt):**
+  SMI runs outside of the kernel code and can potentially cause
+  latency. It is a pity there is no simple way to disable it. Some vendors may
+  provide related switches in BIOS but most machines do not have this capability.
+
+Operating System Configuration
+------------------------------
+
+* **CPU isolation:**
+  To achieve deterministic latency, dedicated CPUs should be allocated for
+  realtime application. This can be achieved by isolating cpus from kernel
+  scheduler. Please refer to
+  http://lxr.free-electrons.com/source/Documentation/kernel-parameters.txt#L1608
+  for more information.
+
+* **Memory allocation:**
+  Memory shoud be reserved for realtime applications and usually hugepage should
+  be used to reduce page fauts/TLB misses.
+
+* **IRQ affinity:**
+  All the non-realtime IRQs should be affinitized to non realtime CPUs to
+  reduce the impact on realtime CPUs. Some OS distributions contain an irqbalance
+  deamon which balances the IRQs among all the cores dynamically. It should be
+  disabled as well.
+
+* **Device assignment for VM:**
+  If a device is used in a VM, then device passthrough is desirable. In this case,
+  the IOMMU should be enabled.
+
+* **Tickless:**
+  Frequent clock ticks cause latency. CONFIG_NOHZ_FULL should be enabled in the
+  linux kernel. With CONFIG_NOHZ_FULL, the physical CPU will trigger many fewer
+  clock tick interrupts(currently, 1 tick per second). This can reduce latency
+  because each host timer interrupt triggers a VM exit from guest to host which
+  causes performance/latency impacts.
+
+* **TSC:**
+  Mark TSC clock source as reliable. A TSC clock source that seems to be
+  unreliable causes the kernel to continuously enable the clock source watchdog
+  to check if TSC frequency is still correct. On recent Intel platforms with
+  Constant TSC/Invariant TSC/Synchronized TSC, the TSC is reliable so the
+  watchdog is useless but cause latency.
+
+* **Idle:**
+  The poll option forces a polling idle loop that can slightly improve the
+  performance of waking up an idle CPU.
+
+* **RCU_NOCB:**
+  RCU is a kernel synchronization mechanism. Refer to
+  http://lxr.free-electrons.com/source/Documentation/RCU/whatisRCU.txt for more
+  information. With RCU_NOCB, the impact from RCU to the VNF will be reduced.
+
+* **Disable the RT throttling:**
+  RT Throttling is a Linux kernel mechanism that
+  occurs when a process or thread uses 100% of the core, leaving no resources for
+  the Linux scheduler to execute the kernel/housekeeping tasks. RT Throttling
+  increases the latency so should be disabled.
+
+* **NUMA configuration:**
+  To achieve the best latency. CPU/Memory and device allocated for realtime
+  application/VM should be in the same NUMA node.
diff --git a/docs/etc/conf.py b/docs/etc/conf.py
deleted file mode 100644
index 006603516..000000000
--- a/docs/etc/conf.py
+++ /dev/null
@@ -1,34 +0,0 @@
-import datetime
-import sys
-import os
-
-try:
-    __import__('imp').find_module('sphinx.ext.numfig')
-    extensions = ['sphinx.ext.numfig']
-except ImportError:
-    # 'pip install sphinx_numfig'
-    extensions = ['sphinx_numfig']
-
-# numfig:
-number_figures = True
-figure_caption_prefix = "Fig."
-
-source_suffix = '.rst'
-master_doc = 'index'
-pygments_style = 'sphinx'
-html_use_index = False
-
-pdf_documents = [('index', u'OPNFV', u'OPNFV Project', u'OPNFV')]
-pdf_fit_mode = "shrink"
-pdf_stylesheets = ['sphinx','kerning','a4']
-#latex_domain_indices = False
-#latex_use_modindex = False
-
-latex_elements = {
-    'printindex': '',
-}
-
-project = u'OPNFV: Template documentation config'
-copyright = u'%s, OPNFV' % datetime.date.today().year
-version = u'1.0.0'
-release = u'1.0.0'
diff --git a/docs/how-to-use-docs/documentation-example.rst b/docs/how-to-use-docs/documentation-example.rst
deleted file mode 100644
index afcf75814..000000000
--- a/docs/how-to-use-docs/documentation-example.rst
+++ /dev/null
@@ -1,86 +0,0 @@
-.. two dots create a comment. please leave this logo at the top of each of your rst files.
-.. image:: ../etc/opnfv-logo.png 
-  :height: 40
-  :width: 200
-  :alt: OPNFV
-  :align: left
-.. these two pipes are to seperate the logo from the first title
-|
-|
-How to create documentation for your OPNFV project
-==================================================
-
-this is the directory structure of the docs/ directory that can be found in the root of your project directory
-
-.. code-block:: bash
-
-    ./etc
-    ./etc/opnfv-logo.png
-    ./etc/conf.py
-    ./how-to-use-docs
-    ./how-to-use-docs/documentation-example.rst
-    ./how-to-use-docs/index.rst
-
-To create your own documentation, Create any number of directories (depending on your need) and place in each of them an index.rst.
-This index file must refence your other rst files.
-
-* Here is an example index.rst
-
-.. code-block:: bash
-
-  Example Documentation table of contents
-  =======================================
-
-  Contents:
-
-  .. toctree::
-     :numbered:
-     :maxdepth: 4
-
-     documentation-example.rst
-
-  Indices and tables
-  ==================
-
-  * :ref:`search`
-
-  Revision: _sha1_
-
-  Build date: |today|
-
-
-The Sphinx Build
-================
-
-When you push documentation changes to gerrit a jenkins job will create html documentation.
-
-* Verify Jobs
-For verify jobs a link to the documentation will show up as a comment in gerrit for you to see the result.
-
-* Merge jobs
-
-Once you are happy with the look of your documentation you can submit the patchset the merge job will 
-copy the output of each documentation directory to http://artifacts.opnfv.org/$project/docs/$name_of_your_folder/index.html
-
-Here are some quick examples of how to use rst markup
-
-This is a headline::
-
-  here is some code, note that it is indented
-
-links are easy to add: Here is a link to sphinx, the tool that we are using to generate documetation http://sphinx-doc.org/
-
-* Bulleted Items
-
-  **this will be bold**
-
-.. code-block:: bash
-
-  echo "Heres is a code block with bash syntax highlighting"
-
-
-Leave these at the bottom of each of your documents they are used internally
-
-Revision: _sha1_
-
-Build date: |today|
diff --git a/docs/how-to-use-docs/index.rst b/docs/how-to-use-docs/index.rst
deleted file mode 100644
index 36710b32d..000000000
--- a/docs/how-to-use-docs/index.rst
+++ /dev/null
@@ -1,30 +0,0 @@
-.. OPNFV Release Engineering documentation, created by
-   sphinx-quickstart on Tue Jun  9 19:12:31 2015.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
-
-.. image:: ../etc/opnfv-logo.png
-  :height: 40
-  :width: 200
-  :alt: OPNFV
-  :align: left
-
-Example Documentation table of contents
-=======================================
-
-Contents:
-
-.. toctree::
-   :numbered:
-   :maxdepth: 4
-
-   documentation-example.rst
-
-Indices and tables
-==================
-
-* :ref:`search`
-
-Revision: _sha1_
-
-Build date: |today|