6 files changed, 319 insertions, 0 deletions
diff --git a/docs/arm/container4nfv_on_arm.rst b/docs/arm/container4nfv_on_arm.rst
new file mode 100644
index 0000000..854f17f
--- /dev/null
+++ b/docs/arm/container4nfv_on_arm.rst
@@ -0,0 +1,295 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Arm Limited.
+
+====================
+Container4NFV on Arm
+====================
+
+Abstract
+========
+
+This document gives a brief introduction on Arm's work status, strategy and possible roadmap for Container4NFV
+on arm64 server platform.
+The target of the description is to let you get to know what are Arm's current capability and future
+direction for Container4NFV project.
+
+Introduction
+============
+
+.. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
+.. _Yardsticktst: https://wiki.opnfv.org/download/attachments/2925202/opnfv_summit_-_bridging_opnfv_and_etsi.pdf?version=1&modificationDate=1458848320000&api=v2
+
+As you know, Arm is a silver member of OPNFV. Arm actively takes part in the work
+of Container4NFV project which aims to enable building a container-based NFV
+infrastructure in edge node or core-network on Arm server platform.
+We would introduce the work of Arm containerized NFV-I from the following points:
+1. Architecture
+2. Container Networking
+3. Related Projects
+4. Current Status and Future Plan
+5. Contacts
+
+Architecture
+============
+
+.. _architecture: https://wiki.opnfv.org/pages/viewpage.action?spaceKey=OpenRetriever&title=Container%27s+Architecture+for+Cloud+Native+NFV
+
+Basically, Arm's containerized NFV infrastructure aligns with the architecture_ of Container4NFV which now usually uses
+the Kubernetes as the Container Orchestration Engine(COE), and the CNI as the network framework.
+Currently a typical architecture of containerized NFV-I on Arm is composed of installer, Kubernetes, related OPNFV projects, such as
+Functest, Yardstick, and possible Arm Node Feature Discovery(A-NFD) which would enable finding certain resources and their usage status
+of Arm servers and is still to be developed. In the future, other high-level VNF orchestration engines, such as Tacker or ONAP would also
+be brought in to facilitate the deployment of actual VNFs.
+
+.. image:: images/Arm_Container4NFV_Architecture.PNG
+   :width: 800px
+   :alt: Containerized NFV Infrastruture on Arm
+
+A typical VNF networking service deployment is given as the following graph:
+
+.. image:: images/networking_service_deployment_on_Arm_server.PNG
+   :width: 800px
+   :alt: Networking Service Deployment on Arm Server
+
+
+Container Networking
+====================
+
+Basic Networking Model
+----------------------
+
+
+Since Arm's containerized NFV infrastructure uses Kubernetes as the COE, so the CNI plug-ins are used to orchestrate networking.
+Every time a POD is initialized or removed, the default CNI plug-in is called with the default configuration. This CNI plug-in
+creates a pseudo interface, attaches it to the relevant underlay network, sets the IP and routes and maps it to the POD namespace.
+
+.. _model: https://kubernetes.io/docs/concepts/cluster-administration/networking/
+.. _Flannel: https://github.com/coreos/flannel
+.. _here: https://github.com/coreos/flannel/releases
+.. _Calico: http://docs.projectcalico.org/
+
+The Kubernetes networking model_ satisfies the following fundamental requirements:
+* *1* all containers can communicate with all other containers without NAT
+* *2* all nodes can communicate with all containers (and vice-versa) without NAT
+* *3* the IP that a container sees itself as is the same IP that others see it as
+
+On the Arm platform, the most common Kubernetes networking solution is Flannel_ which uses overlay technique to resolve the pod
+communication across hosts. The arm64 version of Flannel release can be found here_. Project Calico_ is also a high performance,
+highly scalable networking solution which provides network policy for connecting Kubernetes pods based on the same IP networking
+principles as the internet. But Calico for Arm is still under development and it's one of our task to enable it for container
+networking on Arm Container4NFV.
+
+.. _guide: https://thenewstack.io/hackers-guide-kubernetes-networking/
+
+Refer to guide_, Kubernetes uses CNI plug-ins to orchestrate networking. Every time a POD is initialized or removed, the default
+CNI plug-in is called with the default configuration. This CNI plug-in creates a pseudo interface, attaches it to the relevant
+underlay network, sets the IP and routes and maps it to the POD namespace.
+
+.. _CNI: https://github.com/containernetworking/cni
+.. _release: https://github.com/containernetworking/cni/releases
+
+Most of the ordinary Kubernetes CNI_ plugins for arm64, including bridge, flannel, loopback, host-local, portmap, macvlan, ipvlan, ptp,
+noop could be found in the release_ of containernetworking CNI. Current CNI plugins stable version for arm64 is v0.6.0.
+
+
+Multiple Interfaces Support in a Pod
+------------------------------------
+
+.. _Multus: https://github.com/Intel-Corp/multus-cni
+.. _SR-IOV: https://github.com/hustcat/sriov-cni
+.. _DPDK: https://github.com/Intel-Corp/sriov-cni
+.. _PF: https://github.com/hustcat/sriov-cni/issues/14
+
+Kubernetes initially supports only one CNI interface per POD with one cluster-wide configuration. But some VNFs with data plane
+acceleration, there would be one or two interfaces used for high performance data access besides the normal interfaces, such as
+Flannel, Calico, Weave, PTP, which are still kept for control or configuration purpose.
+
+The SR-IOV_ CNI or DPDK_ CNI could be chosen to add data plane acceleration interfaces for Kubernetes Pods. Arm is doing some
+improvement on SR-IOV_ CNI to assign PF_ directly if VF is not needed or available.
+
+With the help of Multus_ CNI plugin, multiple interfaces can be added at the same time when deploying a pod. The Multus CNI has
+the following features:
+
+
+- It is a contact between the container runtime and other plugins, and it doesn't have any of its own net configuration, it calls
+  other plugins like flannel/calico to do the real net conf job.
+
+- Multus reuses the concept of invoking the delegates in flannel, it groups the multi plugins into delegates and invoke each
+  other in sequential order, according to the JSON scheme in the cni configuration.
+
+- No. of plugins supported is dependent upon the number of delegates in the conf file.
+
+- Master plugin invokes "eth0" interface in the pod, rest of plugins(Mininon plugins eg: sriov,ipam) invoke interfaces as "net0",
+  "net1".. "netn"
+
+- The "masterplugin" is the only net conf option of Multus cni, it identifies the primary network. The default route will point
+  to the primary network.
+
+
+A typical Multus CNI configuration with DPDK passthrough(SR-IOV PF) enabled is given below:
+
+::
+
+  {
+    "name": "multus-k8s-network",
+    "type": "multus",
+    "delegates": [
+        {
+                "type": "flannel",
+                "masterplugin": true,
+                "delegate": {
+                        "isDefaultGateway": true
+                }
+        },
+        {
+                "type": "sriov",
+                "master": "eth1",
+                "dpdk": {
+                        "ethernet_driver": "ixgbe",
+                        "io_driver": "vfio-pci",
+                        "dpdk_devbind": "/root/dpdk/usertools/dpdk-devbind.py"
+                }
+        },
+        {
+                "type": "sriov",
+                "master": "eth2",
+                "dpdk": {
+                        "ethernet_driver": "ixgbe",
+                        "io_driver": "vfio-pci",
+                        "dpdk_devbind": "/root/dpdk/usertools/dpdk-devbind.py"
+                }
+        }
+    ]
+  }
+
+
+Related Projects
+================
+
+For aligning with Container4NFV 'E' release requirement, there are 3 related projects on Arm Container4NFV platform, which are
+Installer(Joid or others), Yardstick for performance evaluation and Functest for basic function verification.
+
+Installer
+---------
+
+Installer is responsible for the deployment of a typical Container4NFV environment, and for 'E' release, it is considered to have
+a Kubernetes cluster, a simple networking solution(Flannel), Yardstick, and some VNFs. The possible installer for arm64 platform
+would be based on one of Compass, Joid and Apex.
+
+Yardstick
+---------
+
+.. _Yardstick: https://wiki.opnfv.org/yardstick
+.. _contribution: https://gerrit.opnfv.org/gerrit/#/c/43721/
+.. _feature: https://gerrit.opnfv.org/gerrit/#/c/43719/
+.. _enabling: https://gerrit.opnfv.org/gerrit/#/c/43723/
+
+Yardstick_ is an OPNFV Project. The project's goal is to verify infrastructure compliance, from the perspective
+of a Virtual Network Function(VNF). The latest Yardstick implementation had been enhanced to support Kubernetes context and we have
+enabled it on arm64 platform.
+
+A typical sample arm64 config file corresponding to yardstick/tests/opnfv/test_cases/opnfv_yardstick_tc080.yaml which is for x86 is
+given below:
+
+::
+
+  schema: "yardstick:task:0.1"
+
+  scenarios:
+  -
+    type: Ping
+    options:
+      packetsize: 200
+
+    host: host-k8s
+    target: target-k8s
+
+    runner:
+      type: Duration
+      duration: 60
+      interval: 1
+
+    sla:
+      max_rtt: 10
+      action: monitor
+
+    context:
+      type: Kubernetes
+      name: k8s
+
+    servers:
+      host:
+        image: openretriever/yardstick_aarch64
+        command: /bin/bash
+        args: ['-c', 'chmod 700 ~/.ssh; chmod 600 ~/.ssh/*; service ssh restart;while true ; do sleep 10000; done']
+      target:
+        image: openretriever/yardstick_aarch64
+        command: /bin/bash
+        args: ['-c', 'chmod 700 ~/.ssh; chmod 600 ~/.ssh/*; service ssh restart;while true ; do sleep 10000; done']
+
+.. _openretriever: https://hub.docker.com/u/openretriever/
+.. _container4nfv: https://hub.docker.com/u/container4nfv/
+
+Some other useful testing images for arm64, such as images for iperf3, netperf could also be found in the docker hub under openretriever_,
+which would be changed to container4nfv_ in the future because of the project renaming.
+
+Besides enabling_ some tests on Arm platform, Arm has made contributions to add more support for Kubernetes context
+and enable a special feature_ for the netperf test scenario.
+
+Functest
+--------
+
+.. _functest: https://wiki.opnfv.org/display/functest/Opnfv+Functional+Testing
+.. _Danube: http://docs.opnfv.org/en/stable-danube/submodules/functest/docs/testing/user/userguide/index.html
+
+The  Functest project provides comprehensive testing methodology, test suites and test cases to test and verify OPNFV Platform functionality
+that covers the VIM and NFVI components.
+
+Functest for Container4NFV could used to verify the basic VIM functionality to support VNF operations(create, modify, shrink, destroy). For
+the Danube_ release,  there are 4 domains(VIM, Controllers, Features, VNF) and 5 tiers(healthcheck, smoke, features, components, vnf) and more
+than 20 test cases.
+
+But now the Functest has not been extended to support Kubernetes, which is still under developing.
+
+
+Current Status and Future Plan
+==============================
+
+Now for Arm containerized NFV-I, we have enabled Multus CNI with Flannel CNI, SR-IOV/DPDK CNI. Data plane acceleration with DPDK on SR-IOV or NIC
+passthrough in containers has also been enabled and tested.
+
+
+.. image:: images/Container_Networking_Acceleration_with_DPDK.PNG
+   :width: 800px
+   :alt: Container Networking Acceleration with DPDK
+
+
+We have also enabled Yardstick to verify the compliance of the Pod communication in the Kubernetes context.
+
+.. image:: images/yardstick_container_test_environment_on_arm_nfv-i.PNG
+   :width: 800px
+   :alt: Yardstick Container Test Environment on Arm NFV-I
+
+For the future plan, we would continue to align with the development roadmap of Container4NFV. And the following work would be also be preferred
+ for Arm Contaier4NFV of the next 'F' release:
+
+* *1* Project Calico enablement for arm64
+* *2* VPP DPDK/ODP for container networking
+* *3* OPNFV installer enablement on Arm for Container4NFV
+* *4* Possible enhancement to Yardstick, Functest
+* *5* Typical VNFs w/o data plane accelerations
+* *6* CI work with Yardstick, Functest
+
+
+Contacts
+========
+
+Trevor Tao(Zijin Tao), Bin Lu, Song Zhu, Kaly Xin and Yibo Cai from Arm have made contributions to this document.
+
+Trevor Tao:  trevor.tao@arm.com
+Bin Lu:   bin.lu@arm.com
+Song Zhu: song.zhu@arm.com
+Kaly xin:  kaly.xin@arm.com
+Yibo Cai:   yibo.cai@arm.com
diff --git a/docs/arm/images/Arm_Container4NFV_Architecture.PNG b/docs/arm/images/Arm_Container4NFV_Architecture.PNG
new file mode 100644
index 0000000..8f4b163
--- /dev/null
+++ b/docs/arm/images/Arm_Container4NFV_Architecture.PNG
diff --git a/docs/arm/images/Container_Networking_Acceleration_with_DPDK.PNG b/docs/arm/images/Container_Networking_Acceleration_with_DPDK.PNG
new file mode 100644
index 0000000..a55489b
--- /dev/null
+++ b/docs/arm/images/Container_Networking_Acceleration_with_DPDK.PNG
diff --git a/docs/arm/images/networking_service_deployment_on_Arm_server.PNG b/docs/arm/images/networking_service_deployment_on_Arm_server.PNG
new file mode 100644
index 0000000..7bdd90d
--- /dev/null
+++ b/docs/arm/images/networking_service_deployment_on_Arm_server.PNG
diff --git a/docs/arm/images/yardstick_container_test_environment_on_arm_nfv-i.PNG b/docs/arm/images/yardstick_container_test_environment_on_arm_nfv-i.PNG
new file mode 100644
index 0000000..5e77ac4
--- /dev/null
+++ b/docs/arm/images/yardstick_container_test_environment_on_arm_nfv-i.PNG
diff --git a/docs/arm/index.rst b/docs/arm/index.rst
new file mode 100644
index 0000000..beeec0f
--- /dev/null
+++ b/docs/arm/index.rst
@@ -0,0 +1,24 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Arm Ltd and others.
+
+====================
+Container4NFV on Arm
+====================
+
+:Project: Container4NFV, https://wiki.opnfv.org/display/container4nfv
+
+:Editors: Trevor Tao (Arm Ltd)
+:Authors: Trevor Tao (Arm Ltd)
+
+:Abstract: This document gives a brief introduction on Arm's work status, strategy and possible
+           roadmap for Container4NFV on arm64 server platform. The target of the description
+           is to let you get to know what are Arm's current capability and future direction
+           for Container4NFV project.
+
+
+.. toctree::
+   :maxdepth: 4
+   :numbered:
+
+   container4nfv_on_arm