diff options
20 files changed, 294 insertions, 46 deletions
diff --git a/docs/release/userguide/clearwater-project.rst b/docs/release/userguide/clearwater-project.rst index 6a5ac60..38f1c7a 100644 --- a/docs/release/userguide/clearwater-project.rst +++ b/docs/release/userguide/clearwater-project.rst @@ -1,24 +1,25 @@ +*********************************** Clearwater implementation for OPNFV -=================================== +*********************************** CONTAINER4NFV setup a Kubernetes cluster on VMs running with Vagrant and kubeadm. kubeadm assumes you have a set of machines (virtual or bare metal) that are up and running. In this way we can get a cluster with one master node and 2 workers (default). If you want to increase the number of workers nodes, please check the Vagrantfile inside the project. -Is Clearwater suitable for Network Functions Virtualization? +*Is Clearwater suitable for Network Functions Virtualization?* Network Functions Virtualization or NFV is, without any doubt, the hottest topic in the telco network space right now. It’s an approach to building telco networks that moves away from proprietary boxes wherever possible to use software components running on industry-standard virtualized IT infrastructures. Over time, many telcos expect to run all their network functions operating at Layer 2 and above in an NFV environment, including IMS. Since Clearwater was designed from the ground up to run in virtualized environments and take full advantage of the flexibility of the Cloud, it is extremely well suited for NFV. Almost all of the ongoing trials of Clearwater with major network operators are closely associated with NFV-related initiatives. About Clearwater ----------------- +################ -[Clearwater](http://www.projectclearwater.org/about-clearwater/) follows [IMS](https://en.wikipedia.org/wiki/IP_Multimedia_Subsystem) architectural principles and supports all of the key standardized interfaces expected of an IMS core network. But unlike traditional implementations of IMS, Clearwater was designed from the ground up for the Cloud. By incorporating design patterns and open source software components that have been proven in many global Web applications, Clearwater achieves an unprecedented combination of massive scalability and exceptional cost-effectiveness. +`Clearwater <http://www.projectclearwater.org/about-clearwater/>`_ follows `IMS <https://en.wikipedia.org/wiki/IP_Multimedia_Subsystem>`_ architectural principles and supports all of the key standardized interfaces expected of an IMS core network. But unlike traditional implementations of IMS, Clearwater was designed from the ground up for the Cloud. By incorporating design patterns and open source software components that have been proven in many global Web applications, Clearwater achieves an unprecedented combination of massive scalability and exceptional cost-effectiveness. Clearwater provides SIP-based call control for voice and video communications and for SIP-based messaging applications. You can use Clearwater as a standalone solution for mass-market VoIP services, relying on its built-in set of basic calling features and standalone susbscriber database, or you can deploy Clearwater as an IMS core in conjunction with other elements such as Telephony Application Servers and a Home Subscriber Server. -Clearwater was designed from the ground up to be optimized for deployment in virtualized and cloud environments. It leans heavily on established design patterns for building and deploying massively scalable web applications, adapting these design patterns to fit the constraints of SIP and IMS. [The Clearwater architecture](http://www.projectclearwater.org/technical/clearwater-architecture/) therefore has some similarities to the traditional IMS architecture but is not identical. +Clearwater was designed from the ground up to be optimized for deployment in virtualized and cloud environments. It leans heavily on established design patterns for building and deploying massively scalable web applications, adapting these design patterns to fit the constraints of SIP and IMS. `The Clearwater architecture <http://www.projectclearwater.org/technical/clearwater-architecture/>`_ therefore has some similarities to the traditional IMS architecture but is not identical. - All components are horizontally scalable using simple, stateless load-balancing. - All long lived state is stored on dedicated “Vellum” nodes which make use of cloud-optimized storage technologies such as Cassandra. No long lived state is stored on other production nodes, making it quick and easy to dynamically scale the clusters and minimizing the impact if a node is lost. @@ -27,8 +28,163 @@ Clearwater was designed from the ground up to be optimized for deployment in vir Clearwater Architecture ------------------------ +####################### .. image:: img/clearwater_architecture.png :width: 800px :alt: Clearwater Architecture + + +********** +Quickstart +********** + +This repository contains instructions and resources for deploying Metaswitch's Clearwater project with Kubernetes. + + +If you need more information about Clearwater project please checkout our +[documentation](https://github.com/opnfv/container4nfv/blob/master/docs/release/userguide/clearwater-project.rst) +or the `official repository <https://github.com/Metaswitch/clearwater-docker>`_. + + +Exposed Services +################ + + +The deployment exposes: + + - the Ellis web UI on port 30080 for self-provisioning. + - STUN/TURN on port 3478 for media relay. + - SIP on port 5060 for service. + - SIP/WebSocket on port 5062 for service. + +SIP devices can register with bono.:5060 and the Ellis provisioning interface can be accessed at port 30080. + + +Prerequirement +############## + +Install Docker and Vagrant +******************************************** + +CONTAINER4NFV uses ``setup_vagrant.sh`` to install all resource used by this repository. + +:: + + container4nfv/src/vagrant# ./setup_vagrant.sh -b libvirt + +Instalation +############## + +Deploy Clearwater with kubeadm +******************************************** + +Check ``clearwater/clearwater_setup.sh`` for details about k8s deployment. + + +:: + + container4nfv/src/vagrant/kubeadm_clearwater# ./deploy.sh + + +Destroy +########## + +:: + + container4nfv/src/vagrant# ./cleanup.sh + + +Making calls through Clearwater +############################### + + +Connect to Ellis service +******************************************** +It's important to connect to Ellis to generate the SIP username, password and domain we will use with the SIP client. +Use your <master ip addres> + port 30080 (k8s default port). If you are not which Ellis's url is, please check inside your master node. + +:: + + kubeadm_clearwater# vagrant ssh master + master@vagrant# ifconfig eth0 | grep "inet addr" | cut -d ':' -f 2 | cut -d ' ' -f 1 + 192.168.121.3 + +In your browser connect to `<master_ip>:30080` (ex. 192.168.121.3:30080). + + +After that, signup and generate two users. The signup key is **secret**. Ellis will automatically allocate you a new number and display +its password to you. Remember this password as it will only be displayed once. +From now on, we will use <username> to refer to the SIP username (e.g. 6505551234) and <password> to refer to the password. + + +Config and install two SIP clients +******************************************** +We'll use both Twinkle and Blink SIP client. , since we are going to try this out inside a LAN network. +This is, of course, only a local test inside a LAN network. Configure the clients may be a little bit trickie, so we add some screenshots: + + +Blink setup +******************************************** +1. Add <username> and <password>. + +.. image:: img/blink01.png + :width: 800px + :alt: Blink SIP client + + +2. Configure a proxy to k8s. + + +.. image:: img/blink02.png + :width: 800px + :alt: Blink SIP client + + +3. Configure the network to use TCP only. + + +.. image:: img/blink03.png + :width: 800px + :alt: Blink SIP client + + +.. image:: img/blink04.png + :width: 800px + :alt: Blink SIP client + + +Twinkle setup +******************************************** + +1. Configure a proxy to k8s. + + +.. image:: img/twinkle01.png + :width: 800px + :alt: Twinkle SIP client + + +2. Add <username> and <password>. + + +.. image:: img/twinkle02.png + :width: 800px + :alt: Twinkle SIP client + + +3. Configure the network to use TCP only. + + +.. image:: img/twinkle03.png + :width: 800px + :alt: Twinkle SIP client + + +Make the call +******************************************** + + +.. image:: img/call.png + :width: 800px + :alt: Call diff --git a/docs/release/userguide/img/blink01.png b/docs/release/userguide/img/blink01.png Binary files differnew file mode 100644 index 0000000..ac74788 --- /dev/null +++ b/docs/release/userguide/img/blink01.png diff --git a/docs/release/userguide/img/blink02.png b/docs/release/userguide/img/blink02.png Binary files differnew file mode 100644 index 0000000..7eb8d46 --- /dev/null +++ b/docs/release/userguide/img/blink02.png diff --git a/docs/release/userguide/img/blink03.png b/docs/release/userguide/img/blink03.png Binary files differnew file mode 100644 index 0000000..ae6220a --- /dev/null +++ b/docs/release/userguide/img/blink03.png diff --git a/docs/release/userguide/img/blink04.png b/docs/release/userguide/img/blink04.png Binary files differnew file mode 100644 index 0000000..17511b5 --- /dev/null +++ b/docs/release/userguide/img/blink04.png diff --git a/docs/release/userguide/img/call.png b/docs/release/userguide/img/call.png Binary files differnew file mode 100644 index 0000000..ec4cdbf --- /dev/null +++ b/docs/release/userguide/img/call.png diff --git a/docs/release/userguide/img/twinkle01.png b/docs/release/userguide/img/twinkle01.png Binary files differnew file mode 100644 index 0000000..e424d51 --- /dev/null +++ b/docs/release/userguide/img/twinkle01.png diff --git a/docs/release/userguide/img/twinkle02.png b/docs/release/userguide/img/twinkle02.png Binary files differnew file mode 100644 index 0000000..8d95bae --- /dev/null +++ b/docs/release/userguide/img/twinkle02.png diff --git a/docs/release/userguide/img/twinkle03.png b/docs/release/userguide/img/twinkle03.png Binary files differnew file mode 100644 index 0000000..4b4b5c7 --- /dev/null +++ b/docs/release/userguide/img/twinkle03.png diff --git a/src/arm/kubernetes_sriov/README.rst b/src/arm/kubernetes_sriov/README.rst new file mode 100644 index 0000000..fde2f51 --- /dev/null +++ b/src/arm/kubernetes_sriov/README.rst @@ -0,0 +1,16 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, arm Limited. + +.. _Flannel: https://github.com/coreos/flannel +.. _SRIOV: https://github.com/hustcat/sriov-cni + +=============================================== +Kubernetes Deployment with SRIOV CNI +=============================================== + +The scenario would deploy pods with SRIOV/Mltus/Flannel CNI. +In this case, "eth0" would be used as the default interface, and the 2nd interface named "net0" would +used as data plane. + diff --git a/src/arm/kubernetes_sriov/k8s-build.sh b/src/arm/kubernetes_sriov/k8s-build.sh new file mode 100755 index 0000000..bc99e30 --- /dev/null +++ b/src/arm/kubernetes_sriov/k8s-build.sh @@ -0,0 +1,32 @@ +#!/bin/bash +set -e + +sudo apt-get install -y docker.io libvirt-bin virt-manager qemu qemu-efi + +WORKSPACE=`pwd` +if [ ! -d "$WORKSPACE/compass4nfv" ]; then + git clone https://gerrit.opnfv.org/gerrit/compass4nfv +fi + +cd compass4nfv + +WORKSPACE=`pwd` + +COMPASS_WORK_DIR=$WORKSPACE/../compass-work +mkdir -p $COMPASS_WORK_DIR +if [ ! -d "$WORKSPACE/work" ]; then + ln -s $COMPASS_WORK_DIR work +fi + +#TODO: remove workaround after patches merged +if [ ! -f "$WORKSPACE/patched" ]; then + + git checkout a360411cb8c775dffa24a4157cec2b566cbde6f3 + curl http://people.linaro.org/~yibo.cai/compass/0001-deploy-cobbler-drop-tcp_tw_recycle-in-sysctl.conf.patch | git apply || true + curl http://people.linaro.org/~yibo.cai/compass/0002-docker-compose-support-aarch64.patch | git apply || true + curl http://people.linaro.org/~yibo.cai/compass/0004-add-a-multus-with-sriov-interfaces-installation.patch | git apply || true + touch "$WORKSPACE/patched" +fi + +# build tarball +COMPASS_ISO_REPO='http://people.linaro.org/~yibo.cai/compass' ./build.sh diff --git a/src/arm/kubernetes_sriov/k8s-deploy.sh b/src/arm/kubernetes_sriov/k8s-deploy.sh new file mode 100755 index 0000000..f625d22 --- /dev/null +++ b/src/arm/kubernetes_sriov/k8s-deploy.sh @@ -0,0 +1,33 @@ +#!/bin/bash +set -e + +#sudo apt-get install -y docker.io libvirt-bin virt-manager qemu qemu-efi + +#!/bin/bash +cd compass4nfv + +export ADAPTER_OS_PATTERN='(?i)CentOS-7.*arm.*' +export OS_VERSION="centos7" +export KUBERNETES_VERSION="v1.7.3" +export DHA="deploy/conf/vm_environment/k8-nosdn-nofeature-noha.yml" +export NETWORK="deploy/conf/network_cfg_sriov.yaml" +export VIRT_NUMBER=2 VIRT_CPUS=2 VIRT_MEM=4096 VIRT_DISK=50G + +# enable sriov cni deployment +echo "Set sriov cni scenario" +sed -i.bak 's/^kube_network_plugin:.*$/kube_network_plugin: sriov/' \ + deploy/adapters/ansible/kubernetes/roles/kargo/files/extra-vars-aarch64.yml + +./deploy.sh + +set -ex + +# basic test: ssh to master, check k8s node status +sshpass -p root ssh root@10.1.0.50 kubectl get nodes 2>/dev/null | grep -i ready + +# scenario specific tests +# show two nics in container +sshpass -p root ssh root@10.1.0.50 \ + kubectl create -f /etc/kubernetes/sriov-test-pod.yml && \ + sleep 30 && \ + kubectl exec multus-test1 -- sh -c "ping -c 3 192.168.123.31" diff --git a/src/arm/kubernetes_sriov/setup.sh b/src/arm/kubernetes_sriov/setup.sh new file mode 100755 index 0000000..b33e990 --- /dev/null +++ b/src/arm/kubernetes_sriov/setup.sh @@ -0,0 +1,7 @@ +#!/bin/bash + +./k8s-build.sh + +sleep 2 + +./k8s-deploy.sh diff --git a/src/vagrant/kubeadm_istio/istio/deploy.sh b/src/vagrant/kubeadm_istio/istio/deploy.sh index ed873f5..051497c 100755 --- a/src/vagrant/kubeadm_istio/istio/deploy.sh +++ b/src/vagrant/kubeadm_istio/istio/deploy.sh @@ -17,12 +17,24 @@ set -ex -# Deploy istio 0.4.0 +# Get latest istio version, refer: https://git.io/getLatestIstio +if [ "x${ISTIO_VERSION}" = "x" ] ; then + ISTIO_VERSION=$(curl -L -s https://api.github.com/repos/istio/istio/releases/latest | \ + grep tag_name | sed "s/ *\"tag_name\": *\"\(.*\)\",*/\1/") +fi + +ISTIO_DIR_NAME="istio-$ISTIO_VERSION" + cd /vagrant curl -L https://git.io/getLatestIstio | sh - -mv istio-0.4.0 istio-source +mv $ISTIO_DIR_NAME istio-source cd /vagrant/istio-source/ -export PATH=$PWD/bin:$PATH + +# Persistently append istioctl bin path to PATH env +echo 'export PATH="$PATH:/vagrant/istio-source/bin"' >> ~/.bashrc +echo "source <(kubectl completion bash)" >> ~/.bashrc +source ~/.bashrc + kubectl apply -f install/kubernetes/istio.yaml # Validate the installation diff --git a/src/vagrant/kubeadm_kata/host_setup.sh b/src/vagrant/kubeadm_kata/host_setup.sh index f9e1a76..d2af951 100644 --- a/src/vagrant/kubeadm_kata/host_setup.sh +++ b/src/vagrant/kubeadm_kata/host_setup.sh @@ -25,19 +25,14 @@ cat << EOF | sudo tee /etc/hosts 192.168.1.23 worker3 EOF -sudo apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D -sudo apt-key adv -k 58118E89F3A912897C070ADBF76221572C52609D -cat << EOF | sudo tee /etc/apt/sources.list.d/docker.list -deb [arch=amd64] https://apt.dockerproject.org/repo ubuntu-xenial main -EOF - curl -s http://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list deb http://apt.kubernetes.io/ kubernetes-xenial main EOF sudo apt-get update -sudo apt-get install -y --allow-downgrades docker-engine=1.12.6-0~ubuntu-xenial kubelet=1.7.0-00 kubeadm=1.7.0-00 kubectl=1.7.0-00 kubernetes-cni=0.5.1-00 +sudo apt-get install -y kubelet kubeadm kubectl kubernetes-cni +sudo swapoff -a sudo systemctl stop kubelet sudo rm -rf /var/lib/kubelet sudo systemctl daemon-reload diff --git a/src/vagrant/kubeadm_kata/kata_setup.sh b/src/vagrant/kubeadm_kata/kata_setup.sh index 9682f3a..c14d844 100644 --- a/src/vagrant/kubeadm_kata/kata_setup.sh +++ b/src/vagrant/kubeadm_kata/kata_setup.sh @@ -17,33 +17,27 @@ set -ex -wget https://storage.googleapis.com/golang/go1.8.3.linux-amd64.tar.gz -sudo tar -xvf go1.8.3.linux-amd64.tar.gz -C /usr/local/ -mkdir -p $HOME/go/src -export GOPATH=$HOME/go -export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin - -go get github.com/clearcontainers/tests -cd $GOPATH/src/github.com/clearcontainers/tests/.ci - -echo "Install dependencies" -bash -f ./setup_env_ubuntu.sh - -echo "Install shim" -bash -f ./install_shim.sh - -echo "Install proxy" -bash -f ./install_proxy.sh - -echo "Install runtime" -bash -f ./install_runtime.sh - -echo "Install CRI-O" -bash -f ./install_crio.sh +cat << EOF | sudo tee /etc/apt/sources.list.d/cc-oci-runtime.list +deb http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04/ / +EOF +curl -fsSL http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04/Release.key | sudo apt-key add - +sudo apt-get update +sudo apt-get install -y cc-oci-runtime + +echo | sudo add-apt-repository ppa:projectatomic/ppa +sudo apt-get update +sudo apt-get install -y cri-o +sudo sed -i 's,runtime_untrusted_workload.*,runtime_untrusted_workload = "/usr/bin/cc-runtime",' /etc/crio/crio.conf +sudo sed -i 's,cgroup_manager.*,cgroup_manager = "cgroupfs",' /etc/crio/crio.conf +sudo sed -i 's,default_workload_trust.*,default_workload_trust = "untrusted",' /etc/crio/crio.conf +sudo sed -i 's,^registries.*,registries = [ "docker.io",' /etc/crio/crio.conf +sudo systemctl enable crio +sudo systemctl daemon-reload +sudo systemctl restart crio sudo systemctl stop kubelet echo "Modify kubelet systemd configuration to use CRI-O" k8s_systemd_file="/etc/systemd/system/kubelet.service.d/10-kubeadm.conf" -sudo sed -i '/KUBELET_AUTHZ_ARGS/a Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=/var/run/crio.sock --runtime-request-timeout=30m"' "$k8s_systemd_file" +sudo sed -i '/KUBELET_AUTHZ_ARGS/a Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --runtime-request-timeout=30m"' "$k8s_systemd_file" sudo systemctl daemon-reload sudo systemctl start kubelet diff --git a/src/vagrant/kubeadm_kata/master_setup.sh b/src/vagrant/kubeadm_kata/master_setup.sh index 3748f01..41dadf0 100644 --- a/src/vagrant/kubeadm_kata/master_setup.sh +++ b/src/vagrant/kubeadm_kata/master_setup.sh @@ -17,7 +17,7 @@ set -ex -sudo kubeadm init --apiserver-advertise-address=192.168.1.10 --service-cidr=10.96.0.0/16 --pod-network-cidr=10.32.0.0/12 --token 8c5adc.1cec8dbf339093f0 +sudo kubeadm init --skip-preflight-checks --apiserver-advertise-address=192.168.1.10 --service-cidr=10.96.0.0/16 --pod-network-cidr=10.32.0.0/12 --token 8c5adc.1cec8dbf339093f0 mkdir ~/.kube sudo cp /etc/kubernetes/admin.conf .kube/config sudo chown $(id -u):$(id -g) ~/.kube/config diff --git a/src/vagrant/kubeadm_kata/worker_setup.sh b/src/vagrant/kubeadm_kata/worker_setup.sh index a6e4bf4..6145793 100644 --- a/src/vagrant/kubeadm_kata/worker_setup.sh +++ b/src/vagrant/kubeadm_kata/worker_setup.sh @@ -16,7 +16,9 @@ # set -ex -sudo kubeadm join --token 8c5adc.1cec8dbf339093f0 192.168.1.10:6443 || true +sudo kubeadm join --discovery-token-unsafe-skip-ca-verification \ + --token 8c5adc.1cec8dbf339093f0 192.168.1.10:6443 \ + --ignore-preflight-errors=SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables sudo apt-get install -y putty-tools mkdir ~/.kube diff --git a/src/vagrant/kubeadm_onap/onap_setup.sh b/src/vagrant/kubeadm_onap/onap_setup.sh index b876580..4dfe1e1 100755 --- a/src/vagrant/kubeadm_onap/onap_setup.sh +++ b/src/vagrant/kubeadm_onap/onap_setup.sh @@ -39,4 +39,5 @@ echo "y\n" | plink -ssh -pw vagrant vagrant@worker1 "sudo rm -rf /dockerdata-nfs cd ~/oom/kubernetes/config && ./createConfig.sh -n onap while true; do sleep 30; kubectl get pods --all-namespaces | grep onap | wc -l | grep "^0$" && break; done source ~/oom/kubernetes/oneclick/setenv.bash +sed -i "s/aaiServiceClusterIp:.*/aaiServiceClusterIp: 10.96.0.254/" ~/oom/kubernetes/aai/values.yaml cd ~/oom/kubernetes/oneclick && ./createAll.bash -n onap diff --git a/src/vagrant/setup_vagrant.sh b/src/vagrant/setup_vagrant.sh index 2dc5ae0..fcde052 100755 --- a/src/vagrant/setup_vagrant.sh +++ b/src/vagrant/setup_vagrant.sh @@ -30,9 +30,9 @@ ${USER} ALL = (root) NOPASSWD:ALL EOF sudo apt-get update -y sudo apt-get install -y git unzip - wget https://releases.hashicorp.com/vagrant/1.8.7/vagrant_1.8.7_x86_64.deb - sudo dpkg -i vagrant_1.8.7_x86_64.deb - rm -rf vagrant_1.8.7_x86_64.deb + wget https://releases.hashicorp.com/vagrant/2.0.2/vagrant_2.0.2_x86_64.deb + sudo dpkg -i vagrant_2.0.2_x86_64.deb + rm -rf vagrant_2.0.2_x86_64.deb sudo apt-get install -y virtualbox @@ -41,7 +41,7 @@ EOF sudo apt-get update sudo apt-get build-dep vagrant ruby-libvirt -y sudo apt-get install -y bridge-utils qemu libvirt-bin ebtables dnsmasq - sudo apt-get install -y libxslt-dev libxml2-dev libvirt-dev zlib1g-dev ruby-dev + sudo apt-get install -y libffi-dev libxslt-dev libxml2-dev libvirt-dev zlib1g-dev ruby-dev vagrant plugin install vagrant-libvirt sudo adduser ${USER} libvirtd sudo service libvirtd restart |
