Add Arm work description for Container4NFV project

As we know, Arm is an active partipants of the Container4NFV project.
Arm has done a lot of work to improve container more widely used for NFV
, which includes various projects of OPNFV focused on container based NFV.
Arm highly recognized the meaning of Container4NFV project, and all Arm's
work would align with the direction of Container4NFV project at the most
extent now and in the future.
The document gives out Arm's existed work for Container4NFV and other
containerized VNF related project. The possible roadmap for Arm's future work
on Container4NFV is also given.

The document contains the following sections:
1. Architecture
2. Container Networking
3. Related Projects
4. the Current Status and Future Plan
5. Contacts

In addition to the format revising, other modifications:

1. Add an index.rst to generate the doc tree
2. Rename the file to container4nfv_on_arm.rst
3. Remove trailing whitespace again
4. Modified the image declartion in the rst

Change-Id: I5f6486ec08c96feb504a50951adab444f035bd53
Signed-off-by: Trevor Tao <trevor.tao@arm.com>

6 files changed, 319 insertions, 0 deletions

diff --git a/docs/arm/container4nfv_on_arm.rst b/docs/arm/container4nfv_on_arm.rst
new file mode 100644
index 0000000..854f17f
--- /dev/null
+++ b/docs/arm/container4nfv_on_arm.rst
@@ -0,0 +1,295 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Arm Limited.
+
+====================
+Container4NFV on Arm
+====================
+
+Abstract
+========
+
+This document gives a brief introduction on Arm's work status, strategy and possible roadmap for Container4NFV
+on arm64 server platform.
+The target of the description is to let you get to know what are Arm's current capability and future
+direction for Container4NFV project.
+
+Introduction
+============
+
+.. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
+.. _Yardsticktst: https://wiki.opnfv.org/download/attachments/2925202/opnfv_summit_-_bridging_opnfv_and_etsi.pdf?version=1&modificationDate=1458848320000&api=v2
+
+As you know, Arm is a silver member of OPNFV. Arm actively takes part in the work
+of Container4NFV project which aims to enable building a container-based NFV
+infrastructure in edge node or core-network on Arm server platform.
+We would introduce the work of Arm containerized NFV-I from the following points:
+1. Architecture
+2. Container Networking
+3. Related Projects
+4. Current Status and Future Plan
+5. Contacts
+
+Architecture
+============
+
+.. _architecture: https://wiki.opnfv.org/pages/viewpage.action?spaceKey=OpenRetriever&title=Container%27s+Architecture+for+Cloud+Native+NFV
+
+Basically, Arm's containerized NFV infrastructure aligns with the architecture_ of Container4NFV which now usually uses
+the Kubernetes as the Container Orchestration Engine(COE), and the CNI as the network framework.
+Currently a typical architecture of containerized NFV-I on Arm is composed of installer, Kubernetes, related OPNFV projects, such as
+Functest, Yardstick, and possible Arm Node Feature Discovery(A-NFD) which would enable finding certain resources and their usage status
+of Arm servers and is still to be developed. In the future, other high-level VNF orchestration engines, such as Tacker or ONAP would also
+be brought in to facilitate the deployment of actual VNFs.
+
+.. image:: images/Arm_Container4NFV_Architecture.PNG
+   :width: 800px
+   :alt: Containerized NFV Infrastruture on Arm
+
+A typical VNF networking service deployment is given as the following graph:
+
+.. image:: images/networking_service_deployment_on_Arm_server.PNG
+   :width: 800px
+   :alt: Networking Service Deployment on Arm Server
+
+
+Container Networking
+====================
+
+Basic Networking Model
+----------------------
+
+
+Since Arm's containerized NFV infrastructure uses Kubernetes as the COE, so the CNI plug-ins are used to orchestrate networking.
+Every time a POD is initialized or removed, the default CNI plug-in is called with the default configuration. This CNI plug-in
+creates a pseudo interface, attaches it to the relevant underlay network, sets the IP and routes and maps it to the POD namespace.
+
+.. _model: https://kubernetes.io/docs/concepts/cluster-administration/networking/
+.. _Flannel: https://github.com/coreos/flannel
+.. _here: https://github.com/coreos/flannel/releases
+.. _Calico: http://docs.projectcalico.org/
+
+The Kubernetes networking model_ satisfies the following fundamental requirements:
+* *1* all containers can communicate with all other containers without NAT
+* *2* all nodes can communicate with all containers (and vice-versa) without NAT
+* *3* the IP that a container sees itself as is the same IP that others see it as
+
+On the Arm platform, the most common Kubernetes networking solution is Flannel_ which uses overlay technique to resolve the pod
+communication across hosts. The arm64 version of Flannel release can be found here_. Project Calico_ is also a high performance,
+highly scalable networking solution which provides network policy for connecting Kubernetes pods based on the same IP networking
+principles as the internet. But Calico for Arm is still under development and it's one of our task to enable it for container
+networking on Arm Container4NFV.
+
+.. _guide: https://thenewstack.io/hackers-guide-kubernetes-networking/
+
+Refer to guide_, Kubernetes uses CNI plug-ins to orchestrate networking. Every time a POD is initialized or removed, the default
+CNI plug-in is called with the default configuration. This CNI plug-in creates a pseudo interface, attaches it to the relevant
+underlay network, sets the IP and routes and maps it to the POD namespace.
+
+.. _CNI: https://github.com/containernetworking/cni
+.. _release: https://github.com/containernetworking/cni/releases
+
+Most of the ordinary Kubernetes CNI_ plugins for arm64, including bridge, flannel, loopback, host-local, portmap, macvlan, ipvlan, ptp,
+noop could be found in the release_ of containernetworking CNI. Current CNI plugins stable version for arm64 is v0.6.0.
+
+
+Multiple Interfaces Support in a Pod
+------------------------------------
+
+.. _Multus: https://github.com/Intel-Corp/multus-cni
+.. _SR-IOV: https://github.com/hustcat/sriov-cni
+.. _DPDK: https://github.com/Intel-Corp/sriov-cni
+.. _PF: https://github.com/hustcat/sriov-cni/issues/14
+
+Kubernetes initially supports only one CNI interface per POD with one cluster-wide configuration. But some VNFs with data plane
+acceleration, there would be one or two interfaces used for high performance data access besides the normal interfaces, such as
+Flannel, Calico, Weave, PTP, which are still kept for control or configuration purpose.
+
+The SR-IOV_ CNI or DPDK_ CNI could be chosen to add data plane acceleration interfaces for Kubernetes Pods. Arm is doing some
+improvement on SR-IOV_ CNI to assign PF_ directly if VF is not needed or available.
+
+With the help of Multus_ CNI plugin, multiple interfaces can be added at the same time when deploying a pod. The Multus CNI has
+the following features:
+
+
+- It is a contact between the container runtime and other plugins, and it doesn't have any of its own net configuration, it calls
+  other plugins like flannel/calico to do the real net conf job.
+
+- Multus reuses the concept of invoking the delegates in flannel, it groups the multi plugins into delegates and invoke each
+  other in sequential order, according to the JSON scheme in the cni configuration.
+
+- No. of plugins supported is dependent upon the number of delegates in the conf file.
+
+- Master plugin invokes "eth0" interface in the pod, rest of plugins(Mininon plugins eg: sriov,ipam) invoke interfaces as "net0",
+  "net1".. "netn"
+
+- The "masterplugin" is the only net conf option of Multus cni, it identifies the primary network. The default route will point
+  to the primary network.
+
+
+A typical Multus CNI configuration with DPDK passthrough(SR-IOV PF) enabled is given below:
+
+::
+
+  {
+    "name": "multus-k8s-network",
+    "type": "multus",
+    "delegates": [
+        {
+                "type": "flannel",
+                "masterplugin": true,
+                "delegate": {
+                        "isDefaultGateway": true
+                }
+        },
+        {
+                "type": "sriov",
+                "master": "eth1",
+                "dpdk": {
+                        "ethernet_driver": "ixgbe",
+                        "io_driver": "vfio-pci",
+                        "dpdk_devbind": "/root/dpdk/usertools/dpdk-devbind.py"
+                }
+        },
+        {
+                "type": "sriov",
+                "master": "eth2",
+                "dpdk": {
+                        "ethernet_driver": "ixgbe",
+                        "io_driver": "vfio-pci",
+                        "dpdk_devbind": "/root/dpdk/usertools/dpdk-devbind.py"
+                }
+        }
+    ]
+  }
+
+
+Related Projects
+================
+
+For aligning with Container4NFV 'E' release requirement, there are 3 related projects on Arm Container4NFV platform, which are
+Installer(Joid or others), Yardstick for performance evaluation and Functest for basic function verification.
+
+Installer
+---------
+
+Installer is responsible for the deployment of a typical Container4NFV environment, and for 'E' release, it is considered to have
+a Kubernetes cluster, a simple networking solution(Flannel), Yardstick, and some VNFs. The possible installer for arm64 platform
+would be based on one of Compass, Joid and Apex.
+
+Yardstick
+---------
+
+.. _Yardstick: https://wiki.opnfv.org/yardstick
+.. _contribution: https://gerrit.opnfv.org/gerrit/#/c/43721/
+.. _feature: https://gerrit.opnfv.org/gerrit/#/c/43719/
+.. _enabling: https://gerrit.opnfv.org/gerrit/#/c/43723/
+
+Yardstick_ is an OPNFV Project. The project's goal is to verify infrastructure compliance, from the perspective
+of a Virtual Network Function(VNF). The latest Yardstick implementation had been enhanced to support Kubernetes context and we have
+enabled it on arm64 platform.
+
+A typical sample arm64 config file corresponding to yardstick/tests/opnfv/test_cases/opnfv_yardstick_tc080.yaml which is for x86 is
+given below:
+
+::
+
+  schema: "yardstick:task:0.1"
+
+  scenarios:
+  -
+    type: Ping
+    options:
+      packetsize: 200
+
+    host: host-k8s
+    target: target-k8s
+
+    runner:
+      type: Duration
+      duration: 60
+      interval: 1
+
+    sla:
+      max_rtt: 10
+      action: monitor
+
+    context:
+      type: Kubernetes
+      name: k8s
+
+    servers:
+      host:
+        image: openretriever/yardstick_aarch64
+        command: /bin/bash
+        args: ['-c', 'chmod 700 ~/.ssh; chmod 600 ~/.ssh/*; service ssh restart;while true ; do sleep 10000; done']
+      target:
+        image: openretriever/yardstick_aarch64
+        command: /bin/bash
+        args: ['-c', 'chmod 700 ~/.ssh; chmod 600 ~/.ssh/*; service ssh restart;while true ; do sleep 10000; done']
+
+.. _openretriever: https://hub.docker.com/u/openretriever/
+.. _container4nfv: https://hub.docker.com/u/container4nfv/
+
+Some other useful testing images for arm64, such as images for iperf3, netperf could also be found in the docker hub under openretriever_,
+which would be changed to container4nfv_ in the future because of the project renaming.
+
+Besides enabling_ some tests on Arm platform, Arm has made contributions to add more support for Kubernetes context
+and enable a special feature_ for the netperf test scenario.
+
+Functest
+--------
+
+.. _functest: https://wiki.opnfv.org/display/functest/Opnfv+Functional+Testing
+.. _Danube: http://docs.opnfv.org/en/stable-danube/submodules/functest/docs/testing/user/userguide/index.html
+
+The  Functest project provides comprehensive testing methodology, test suites and test cases to test and verify OPNFV Platform functionality
+that covers the VIM and NFVI components.
+
+Functest for Container4NFV could used to verify the basic VIM functionality to support VNF operations(create, modify, shrink, destroy). For
+the Danube_ release,  there are 4 domains(VIM, Controllers, Features, VNF) and 5 tiers(healthcheck, smoke, features, components, vnf) and more
+than 20 test cases.
+
+But now the Functest has not been extended to support Kubernetes, which is still under developing.
+
+
+Current Status and Future Plan
+==============================
+
+Now for Arm containerized NFV-I, we have enabled Multus CNI with Flannel CNI, SR-IOV/DPDK CNI. Data plane acceleration with DPDK on SR-IOV or NIC
+passthrough in containers has also been enabled and tested.
+
+
+.. image:: images/Container_Networking_Acceleration_with_DPDK.PNG
+   :width: 800px
+   :alt: Container Networking Acceleration with DPDK
+
+
+We have also enabled Yardstick to verify the compliance of the Pod communication in the Kubernetes context.
+
+.. image:: images/yardstick_container_test_environment_on_arm_nfv-i.PNG
+   :width: 800px
+   :alt: Yardstick Container Test Environment on Arm NFV-I
+
+For the future plan, we would continue to align with the development roadmap of Container4NFV. And the following work would be also be preferred
+ for Arm Contaier4NFV of the next 'F' release:
+
+* *1* Project Calico enablement for arm64
+* *2* VPP DPDK/ODP for container networking
+* *3* OPNFV installer enablement on Arm for Container4NFV
+* *4* Possible enhancement to Yardstick, Functest
+* *5* Typical VNFs w/o data plane accelerations
+* *6* CI work with Yardstick, Functest
+
+
+Contacts
+========
+
+Trevor Tao(Zijin Tao), Bin Lu, Song Zhu, Kaly Xin and Yibo Cai from Arm have made contributions to this document.
+
+Trevor Tao:  trevor.tao@arm.com
+Bin Lu:   bin.lu@arm.com
+Song Zhu: song.zhu@arm.com
+Kaly xin:  kaly.xin@arm.com
+Yibo Cai:   yibo.cai@arm.com
diff --git a/docs/arm/images/Arm_Container4NFV_Architecture.PNG b/docs/arm/images/Arm_Container4NFV_Architecture.PNG
new file mode 100644
index 0000000..8f4b163
--- /dev/null
+++ b/docs/arm/images/Arm_Container4NFV_Architecture.PNG
diff --git a/docs/arm/images/Container_Networking_Acceleration_with_DPDK.PNG b/docs/arm/images/Container_Networking_Acceleration_with_DPDK.PNG
new file mode 100644
index 0000000..a55489b
--- /dev/null
+++ b/docs/arm/images/Container_Networking_Acceleration_with_DPDK.PNG
diff --git a/docs/arm/images/networking_service_deployment_on_Arm_server.PNG b/docs/arm/images/networking_service_deployment_on_Arm_server.PNG
new file mode 100644
index 0000000..7bdd90d
--- /dev/null
+++ b/docs/arm/images/networking_service_deployment_on_Arm_server.PNG
diff --git a/docs/arm/images/yardstick_container_test_environment_on_arm_nfv-i.PNG b/docs/arm/images/yardstick_container_test_environment_on_arm_nfv-i.PNG
new file mode 100644
index 0000000..5e77ac4
--- /dev/null
+++ b/docs/arm/images/yardstick_container_test_environment_on_arm_nfv-i.PNG
diff --git a/docs/arm/index.rst b/docs/arm/index.rst
new file mode 100644
index 0000000..beeec0f
--- /dev/null
+++ b/docs/arm/index.rst
@@ -0,0 +1,24 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Arm Ltd and others.
+
+====================
+Container4NFV on Arm
+====================
+
+:Project: Container4NFV, https://wiki.opnfv.org/display/container4nfv
+
+:Editors: Trevor Tao (Arm Ltd)
+:Authors: Trevor Tao (Arm Ltd)
+
+:Abstract: This document gives a brief introduction on Arm's work status, strategy and possible
+           roadmap for Container4NFV on arm64 server platform. The target of the description
+           is to let you get to know what are Arm's current capability and future direction
+           for Container4NFV project.
+
+
+.. toctree::
+   :maxdepth: 4
+   :numbered:
+
+   container4nfv_on_arm