documentation for sriov cni with pf mode

Change-Id: I0015acdbed7cb1b544c673b080cc8b78c875e355
Signed-off-by: Bin Lu <bin.lu@arm.com>

2 files changed, 330 insertions, 2 deletions

diff --git a/docs/arm/data_plane_sriov_pf_deployment.rst b/docs/arm/data_plane_sriov_pf_deployment.rst
new file mode 100644
index 0000000..7cbd4d7
--- /dev/null
+++ b/docs/arm/data_plane_sriov_pf_deployment.rst
@@ -0,0 +1,327 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Arm Limited.
+
+
+
+===============================================
+SRIOV CNI with PF Mode Deployment on Arm Server
+===============================================
+
+Abstract
+========
+
+This document gives a brief introduction on how to deploy SRIOV CNI with PF mode for data plane.
+
+Introduction
+============
+.. _sriov_cni: https://github.com/hustcat/sriov-cni
+.. _Flannel: https://github.com/coreos/flannel
+.. _Multus:  https://github.com/Intel-Corp/multus-cni
+.. _cni:     https://github.com/containernetworking/cni
+.. _kubeadm: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
+.. _k8s-crd: https://kubernetes.io/docs/concepts/api-extension/custom-resources/
+.. _arm64:   https://github.com/kubernetes/website/pull/6511
+.. _files:   https://github.com/kubernetes/website/pull/6511/files
+
+
+As we know, in some cases we need to deploy multiple network interfaces
+with data-plane acceration for single Pod.
+Typically, in production area(5G, Edge computing),
+one interface we used for control plane, it usually will be flannel.
+For data plane, sriov cni + DPDK has the best throughput and the lowest lantency.
+In this case, I will introduce sriov cni with PF mode firstly.
+SRIOV with PF mode is always used in Edge computing.
+Because sriov NIC is not so common on Edge computing.
+And also PF mode is used as vFirewall, vPorxy in data center.
+
+NIC with SR-IOV capabilities works by introducing the idea of physical functions (PFs)
+and virtual functions (VFs).
+In general, PF is used by host.
+Each VFs can be treated as a separate physical NIC and assigned to one container,
+and configured with separate MAC, VLAN and IP, etc.
+If we want the best networking performance for Pods, this should be the best solution.
+
+For demo purpose, I suggest that we use Kubeadm to deploy a Kubernetes cluster firstly.
+Then I will give out a typical deployment scenario with SRIOV data plane interface added.
+
+
+Use Case Architecture
+=====================
+
+Kubelet is responsible for establishing the network interfaces for each pod;
+it does this by invoking its configured CNI plugin.
+When Multus is invoked, it recovers pod annotations related to Multus,
+in turn, then it uses these annotations to recover a Kubernetes custom resource definition (CRD),
+which is an object that informs which plugins to invoke
+and the configuration needing to be passed to them.
+
+Basic Information about Environment
+===================================
+
+Cluster Info
+
+In this case, we deploy master and slave as one node.
+Suppose it to be: 192.168.1.2
+
+In 192.168.1.2, 2 NIC as required.
+Suppose it to be: eth0, eth1, eth0 is used to be controle plane, and eth1 is used to be data plane.
+
+Deploy Kubernetes
+-----------------
+Please see link(https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/) as reference.
+
+
+Rbac Added (optional)
+---------------------
+Please make sure that rbac was added for Kubernetes cluster.
+here we name it as rbac.yaml:
+
+::
+ apiVersion: rbac.authorization.k8s.io/v1beta1
+ kind: ClusterRoleBinding
+ metadata:
+   name: fabric8-rbac
+ subjects:
+   - kind: ServiceAccount
+     # Reference to upper's `metadata.name`
+     name: default
+     # Reference to upper's `metadata.namespace`
+     namespace: default
+ roleRef:
+   kind: ClusterRole
+   name: cluster-admin
+   apiGroup: rbac.authorization.k8s.io
+
+command:
+
+::
+ kubectl create -f rbac.yaml
+
+Creat CRD
+---------
+Please make sure that CRD was added for Kubernetes cluster.
+Here we name it as crdnetwork.yaml:
+
+::
+ apiVersion: apiextensions.k8s.io/v1beta1
+ kind: CustomResourceDefinition
+ metadata:
+   # name must match the spec fields below, and be in the form: <plural>.<group>
+   name: networks.kubernetes.com
+ spec:
+   # group name to use for REST API: /apis/<group>/<version>
+   group: kubernetes.com
+   # version name to use for REST API: /apis/<group>/<version>
+   version: v1
+   # either Namespaced or Cluster
+   scope: Namespaced
+   names:
+     # plural name to be used in the URL: /apis/<group>/<version>/<plural>
+     plural: networks
+     # singular name to be used as an alias on the CLI and for display
+     singular: network
+     # kind is normally the CamelCased singular type. Your resource manifests use this.
+     kind: Network
+     # shortNames allow shorter string to match your resource on the CLI
+     shortNames:
+     - net
+
+command:
+
+::
+ kubectl create -f crdnetwork.yaml
+
+Create Flannel-network for Control Plane
+----------------------------------------
+Create flannel network as control plane.
+Here we name it as flannel-network.yaml:
+
+::
+ apiVersion: "kubernetes.com/v1"
+ kind: Network
+ metadata:
+   name: flannel-conf
+ plugin: flannel
+ args: '[
+         {
+                 "masterplugin": true,
+                 "delegate": {
+                         "isDefaultGateway": true
+                 }
+         }
+ ]'
+
+command:
+
+::
+ kubectl create -f flannel-network.yaml
+
+Create Sriov-network for Data Plane
+-----------------------------------
+Create sriov network with PF mode as data plane.
+Here we name it as sriov-network.yaml:
+
+::
+ apiVersion: "kubernetes.com/v1"
+ kind: Network
+ metadata:
+   name: sriov-conf
+ plugin: sriov
+ args: '[
+        {
+                 "master": "eth1",
+                 "pfOnly": true,
+                 "ipam": {
+                         "type": "host-local",
+                         "subnet": "192.168.123.0/24",
+                         "rangeStart": "192.168.123.2",
+                         "rangeEnd": "192.168.123.10",
+                         "routes": [
+                                 { "dst": "0.0.0.0/0" }
+                         ],
+                         "gateway": "192.168.123.1"
+                 }
+         }
+ ]'
+
+command:
+
+::
+ kubectl create -f sriov-network.yaml
+
+CNI Installation
+================
+.. _CNI: https://github.com/containernetworking/plugins
+Firstly, we should deploy all CNI plugins. The build process is following:
+
+
+::
+   git clone https://github.com/containernetworking/plugins.git
+   cd plugins
+   ./build.sh
+   cp bin/* /opt/cni/bin
+
+.. _Multus: https://github.com/Intel-Corp/multus-cni
+
+To deploy control plane and data plane interfaces, besides the Flannel CNI and SRIOV CNI,
+we need to deploy the Multus_. The build process of it is as:
+
+::
+   git clone https://github.com/Intel-Corp/multus-cni.git
+   cd multus-cni
+   ./build
+   cp bin/multus /opt/cni/bin
+
+To use the Multus_ CNI,
+we should put the Multus CNI binary to /opt/cni/bin/ where the Flannel CNI and SRIOV
+CNIs are put.
+
+.. _SRIOV: https://github.com/hustcat/sriov-cni
+The build process of it is as:
+
+::
+  git clone https://github.com/hustcat/sriov-cni.git
+  cd sriov-cni
+  ./build
+  cp bin/* /opt/cni/bin
+
+CNI Configuration
+=================
+The following multus CNI configuration is located in /etc/cni/net.d/, here we name it
+as multus-cni.conf:
+
+::
+ {
+   "name": "minion-cni-network",
+   "type": "multus",
+   "kubeconfig": "/etc/kubernetes/admin.conf",
+   "delegates": [{
+     "type": "flannel",
+     "masterplugin": true,
+     "delegate": {
+       "isDefaultGateway": true
+     }
+   }]
+ }
+
+command:
+
+::
+  step1, remove all files in /etc/cni/net.d/
+    rm /etc/cni/net.d/* -rf
+
+  step2, copy /etc/kubernetes/admin.conf into each nodes.
+
+  step3, copy multus-cni.conf into /etc/cni/net.d/
+
+  step4, restart kubelet
+    systemctl restart kubelet
+
+
+Configuring Pod with Control Plane and Data Plane
+=================================================
+
+1, Save the below following YAML to pod-sriov.yaml.
+In this case flannle-conf network object act as the primary network.
+
+::
+ apiVersion: v1
+ kind: Pod
+ metadata:
+   name: pod-sriov
+   annotations:
+     networks: '[
+         { "name": "flannel-conf" },
+         { "name": "sriov-conf" }
+     ]'
+ spec:  # specification of the pod's contents
+   containers:
+   - name: pod-sriov
+     image: "busybox"
+     command: ["top"]
+     stdin: true
+     tty: true
+
+2, Create Pod
+
+::
+ command:
+  kubectl create -f pod-sriov.yaml
+
+3, Get the details of the running pod from the master
+
+::
+ # kubectl get pods
+ NAME                   READY     STATUS    RESTARTS   AGE
+ pod-sriov              1/1       Running   0          30s
+
+Verifying Pod Network
+=====================
+
+::
+ # kubectl exec pod-sriov -- ip a
+ 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
+    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
+    inet 127.0.0.1/8 scope host lo
+       valid_lft forever preferred_lft forever
+    inet6 ::1/128 scope host
+       valid_lft forever preferred_lft forever
+ 3: eth0@if124: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue
+    link/ether 0a:58:0a:e9:40:2a brd ff:ff:ff:ff:ff:ff
+    inet 10.233.64.42/24 scope global eth0
+       valid_lft forever preferred_lft forever
+    inet6 fe80::8e6:32ff:fed3:7645/64 scope link
+       valid_lft forever preferred_lft forever
+ 4: net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
+    link/ether 52:54:00:d4:d2:e5 brd ff:ff:ff:ff:ff:ff
+    inet 192.168.123.2/24 scope global net0
+       valid_lft forever preferred_lft forever
+    inet6 fe80::5054:ff:fed4:d2e5/64 scope link
+       valid_lft forever preferred_lft forever
+
+Contacts
+========
+
+Bin Lu:      bin.lu@arm.com
diff --git a/docs/arm/index.rst b/docs/arm/index.rst
index eebea47..cb19b32 100644
--- a/docs/arm/index.rst
+++ b/docs/arm/index.rst
@@ -8,8 +8,8 @@ Container4NFV on Arm
 
 :Project: Container4NFV, https://wiki.opnfv.org/display/container4nfv
 
-:Editors: Trevor Tao (Arm Ltd)
-:Authors: Trevor Tao (Arm Ltd)
+:Editors: Trevor Tao (Arm Ltd) Bin Lu (Arm Ltd)
+:Authors: Trevor Tao (Arm Ltd) Bin Lu (Arm Ltd)
 
 :Abstract: This document gives a brief introduction on Arm's work status, strategy and possible
            roadmap for Container4NFV on arm64 server platform. The target of the description
@@ -23,3 +23,4 @@ Container4NFV on Arm
 
    container4nfv_on_arm
    multi_flannel_intfs_deployment
+   data_plane_sriov_pf_deployment