diff options
Diffstat (limited to 'docs/arm/container4nfv_on_arm.rst')
| -rw-r--r-- | docs/arm/container4nfv_on_arm.rst | 295 |
1 files changed, 295 insertions, 0 deletions
diff --git a/docs/arm/container4nfv_on_arm.rst b/docs/arm/container4nfv_on_arm.rst new file mode 100644 index 0000000..854f17f --- /dev/null +++ b/docs/arm/container4nfv_on_arm.rst @@ -0,0 +1,295 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Arm Limited. + +==================== +Container4NFV on Arm +==================== + +Abstract +======== + +This document gives a brief introduction on Arm's work status, strategy and possible roadmap for Container4NFV +on arm64 server platform. +The target of the description is to let you get to know what are Arm's current capability and future +direction for Container4NFV project. + +Introduction +============ + +.. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf +.. _Yardsticktst: https://wiki.opnfv.org/download/attachments/2925202/opnfv_summit_-_bridging_opnfv_and_etsi.pdf?version=1&modificationDate=1458848320000&api=v2 + +As you know, Arm is a silver member of OPNFV. Arm actively takes part in the work +of Container4NFV project which aims to enable building a container-based NFV +infrastructure in edge node or core-network on Arm server platform. +We would introduce the work of Arm containerized NFV-I from the following points: +1. Architecture +2. Container Networking +3. Related Projects +4. Current Status and Future Plan +5. Contacts + +Architecture +============ + +.. _architecture: https://wiki.opnfv.org/pages/viewpage.action?spaceKey=OpenRetriever&title=Container%27s+Architecture+for+Cloud+Native+NFV + +Basically, Arm's containerized NFV infrastructure aligns with the architecture_ of Container4NFV which now usually uses +the Kubernetes as the Container Orchestration Engine(COE), and the CNI as the network framework. +Currently a typical architecture of containerized NFV-I on Arm is composed of installer, Kubernetes, related OPNFV projects, such as +Functest, Yardstick, and possible Arm Node Feature Discovery(A-NFD) which would enable finding certain resources and their usage status +of Arm servers and is still to be developed. In the future, other high-level VNF orchestration engines, such as Tacker or ONAP would also +be brought in to facilitate the deployment of actual VNFs. + +.. image:: images/Arm_Container4NFV_Architecture.PNG + :width: 800px + :alt: Containerized NFV Infrastruture on Arm + +A typical VNF networking service deployment is given as the following graph: + +.. image:: images/networking_service_deployment_on_Arm_server.PNG + :width: 800px + :alt: Networking Service Deployment on Arm Server + + +Container Networking +==================== + +Basic Networking Model +---------------------- + + +Since Arm's containerized NFV infrastructure uses Kubernetes as the COE, so the CNI plug-ins are used to orchestrate networking. +Every time a POD is initialized or removed, the default CNI plug-in is called with the default configuration. This CNI plug-in +creates a pseudo interface, attaches it to the relevant underlay network, sets the IP and routes and maps it to the POD namespace. + +.. _model: https://kubernetes.io/docs/concepts/cluster-administration/networking/ +.. _Flannel: https://github.com/coreos/flannel +.. _here: https://github.com/coreos/flannel/releases +.. _Calico: http://docs.projectcalico.org/ + +The Kubernetes networking model_ satisfies the following fundamental requirements: +* *1* all containers can communicate with all other containers without NAT +* *2* all nodes can communicate with all containers (and vice-versa) without NAT +* *3* the IP that a container sees itself as is the same IP that others see it as + +On the Arm platform, the most common Kubernetes networking solution is Flannel_ which uses overlay technique to resolve the pod +communication across hosts. The arm64 version of Flannel release can be found here_. Project Calico_ is also a high performance, +highly scalable networking solution which provides network policy for connecting Kubernetes pods based on the same IP networking +principles as the internet. But Calico for Arm is still under development and it's one of our task to enable it for container +networking on Arm Container4NFV. + +.. _guide: https://thenewstack.io/hackers-guide-kubernetes-networking/ + +Refer to guide_, Kubernetes uses CNI plug-ins to orchestrate networking. Every time a POD is initialized or removed, the default +CNI plug-in is called with the default configuration. This CNI plug-in creates a pseudo interface, attaches it to the relevant +underlay network, sets the IP and routes and maps it to the POD namespace. + +.. _CNI: https://github.com/containernetworking/cni +.. _release: https://github.com/containernetworking/cni/releases + +Most of the ordinary Kubernetes CNI_ plugins for arm64, including bridge, flannel, loopback, host-local, portmap, macvlan, ipvlan, ptp, +noop could be found in the release_ of containernetworking CNI. Current CNI plugins stable version for arm64 is v0.6.0. + + +Multiple Interfaces Support in a Pod +------------------------------------ + +.. _Multus: https://github.com/Intel-Corp/multus-cni +.. _SR-IOV: https://github.com/hustcat/sriov-cni +.. _DPDK: https://github.com/Intel-Corp/sriov-cni +.. _PF: https://github.com/hustcat/sriov-cni/issues/14 + +Kubernetes initially supports only one CNI interface per POD with one cluster-wide configuration. But some VNFs with data plane +acceleration, there would be one or two interfaces used for high performance data access besides the normal interfaces, such as +Flannel, Calico, Weave, PTP, which are still kept for control or configuration purpose. + +The SR-IOV_ CNI or DPDK_ CNI could be chosen to add data plane acceleration interfaces for Kubernetes Pods. Arm is doing some +improvement on SR-IOV_ CNI to assign PF_ directly if VF is not needed or available. + +With the help of Multus_ CNI plugin, multiple interfaces can be added at the same time when deploying a pod. The Multus CNI has +the following features: + + +- It is a contact between the container runtime and other plugins, and it doesn't have any of its own net configuration, it calls + other plugins like flannel/calico to do the real net conf job. + +- Multus reuses the concept of invoking the delegates in flannel, it groups the multi plugins into delegates and invoke each + other in sequential order, according to the JSON scheme in the cni configuration. + +- No. of plugins supported is dependent upon the number of delegates in the conf file. + +- Master plugin invokes "eth0" interface in the pod, rest of plugins(Mininon plugins eg: sriov,ipam) invoke interfaces as "net0", + "net1".. "netn" + +- The "masterplugin" is the only net conf option of Multus cni, it identifies the primary network. The default route will point + to the primary network. + + +A typical Multus CNI configuration with DPDK passthrough(SR-IOV PF) enabled is given below: + +:: + + { + "name": "multus-k8s-network", + "type": "multus", + "delegates": [ + { + "type": "flannel", + "masterplugin": true, + "delegate": { + "isDefaultGateway": true + } + }, + { + "type": "sriov", + "master": "eth1", + "dpdk": { + "ethernet_driver": "ixgbe", + "io_driver": "vfio-pci", + "dpdk_devbind": "/root/dpdk/usertools/dpdk-devbind.py" + } + }, + { + "type": "sriov", + "master": "eth2", + "dpdk": { + "ethernet_driver": "ixgbe", + "io_driver": "vfio-pci", + "dpdk_devbind": "/root/dpdk/usertools/dpdk-devbind.py" + } + } + ] + } + + +Related Projects +================ + +For aligning with Container4NFV 'E' release requirement, there are 3 related projects on Arm Container4NFV platform, which are +Installer(Joid or others), Yardstick for performance evaluation and Functest for basic function verification. + +Installer +--------- + +Installer is responsible for the deployment of a typical Container4NFV environment, and for 'E' release, it is considered to have +a Kubernetes cluster, a simple networking solution(Flannel), Yardstick, and some VNFs. The possible installer for arm64 platform +would be based on one of Compass, Joid and Apex. + +Yardstick +--------- + +.. _Yardstick: https://wiki.opnfv.org/yardstick +.. _contribution: https://gerrit.opnfv.org/gerrit/#/c/43721/ +.. _feature: https://gerrit.opnfv.org/gerrit/#/c/43719/ +.. _enabling: https://gerrit.opnfv.org/gerrit/#/c/43723/ + +Yardstick_ is an OPNFV Project. The project's goal is to verify infrastructure compliance, from the perspective +of a Virtual Network Function(VNF). The latest Yardstick implementation had been enhanced to support Kubernetes context and we have +enabled it on arm64 platform. + +A typical sample arm64 config file corresponding to yardstick/tests/opnfv/test_cases/opnfv_yardstick_tc080.yaml which is for x86 is +given below: + +:: + + schema: "yardstick:task:0.1" + + scenarios: + - + type: Ping + options: + packetsize: 200 + + host: host-k8s + target: target-k8s + + runner: + type: Duration + duration: 60 + interval: 1 + + sla: + max_rtt: 10 + action: monitor + + context: + type: Kubernetes + name: k8s + + servers: + host: + image: openretriever/yardstick_aarch64 + command: /bin/bash + args: ['-c', 'chmod 700 ~/.ssh; chmod 600 ~/.ssh/*; service ssh restart;while true ; do sleep 10000; done'] + target: + image: openretriever/yardstick_aarch64 + command: /bin/bash + args: ['-c', 'chmod 700 ~/.ssh; chmod 600 ~/.ssh/*; service ssh restart;while true ; do sleep 10000; done'] + +.. _openretriever: https://hub.docker.com/u/openretriever/ +.. _container4nfv: https://hub.docker.com/u/container4nfv/ + +Some other useful testing images for arm64, such as images for iperf3, netperf could also be found in the docker hub under openretriever_, +which would be changed to container4nfv_ in the future because of the project renaming. + +Besides enabling_ some tests on Arm platform, Arm has made contributions to add more support for Kubernetes context +and enable a special feature_ for the netperf test scenario. + +Functest +-------- + +.. _functest: https://wiki.opnfv.org/display/functest/Opnfv+Functional+Testing +.. _Danube: http://docs.opnfv.org/en/stable-danube/submodules/functest/docs/testing/user/userguide/index.html + +The Functest project provides comprehensive testing methodology, test suites and test cases to test and verify OPNFV Platform functionality +that covers the VIM and NFVI components. + +Functest for Container4NFV could used to verify the basic VIM functionality to support VNF operations(create, modify, shrink, destroy). For +the Danube_ release, there are 4 domains(VIM, Controllers, Features, VNF) and 5 tiers(healthcheck, smoke, features, components, vnf) and more +than 20 test cases. + +But now the Functest has not been extended to support Kubernetes, which is still under developing. + + +Current Status and Future Plan +============================== + +Now for Arm containerized NFV-I, we have enabled Multus CNI with Flannel CNI, SR-IOV/DPDK CNI. Data plane acceleration with DPDK on SR-IOV or NIC +passthrough in containers has also been enabled and tested. + + +.. image:: images/Container_Networking_Acceleration_with_DPDK.PNG + :width: 800px + :alt: Container Networking Acceleration with DPDK + + +We have also enabled Yardstick to verify the compliance of the Pod communication in the Kubernetes context. + +.. image:: images/yardstick_container_test_environment_on_arm_nfv-i.PNG + :width: 800px + :alt: Yardstick Container Test Environment on Arm NFV-I + +For the future plan, we would continue to align with the development roadmap of Container4NFV. And the following work would be also be preferred + for Arm Contaier4NFV of the next 'F' release: + +* *1* Project Calico enablement for arm64 +* *2* VPP DPDK/ODP for container networking +* *3* OPNFV installer enablement on Arm for Container4NFV +* *4* Possible enhancement to Yardstick, Functest +* *5* Typical VNFs w/o data plane accelerations +* *6* CI work with Yardstick, Functest + + +Contacts +======== + +Trevor Tao(Zijin Tao), Bin Lu, Song Zhu, Kaly Xin and Yibo Cai from Arm have made contributions to this document. + +Trevor Tao: trevor.tao@arm.com +Bin Lu: bin.lu@arm.com +Song Zhu: song.zhu@arm.com +Kaly xin: kaly.xin@arm.com +Yibo Cai: yibo.cai@arm.com |