From 4ce4060e302c0cb98ce6658f44fede048df6aeeb Mon Sep 17 00:00:00 2001 From: Thomas F Herbert Date: Tue, 9 Feb 2016 22:17:45 -0500 Subject: Synch stable/brahmaputra with master Several commits over recent months have not been cherry-picked to stable/brahmaputra. As some of these have affected gerrit ci a set of intermediate patches will not pass gerrit ci. So this is a merge commit of the outstanding patches that need to be cherry-picked and brings stable/b to be identical to master commit 4e07dd488cb78b8333763418c4ccc25e05766956 Tue May 10 18:31:20 2016 -0400 "Fix for new location of OVS and DPDK COPR repos." The patches merged into this patch are: 0001-Build-OVS-with-dpdk.patch 0002-Fix-pathname-to-script.patch 0003-Fix-uploading-of-files-and-remove-date-from-file-nam.patch 0004-OVS-Archive-file-name-changed.patch 0005-Fixes-RPM-build-problem.-Use-correct-copr-repo.-Fix-.patch 0006-Fixed-base-name-of-intermediate-tarball-name.patch 0007-Fix-build-problem-by-freezing-copr-commits.patch 0008-fuel-plugin-Pick-updated-libpcre-packages.patch 0009-Fix-for-new-location-of-OVS-and-DPDK-COPR-repos.patch Plus final manual changes (add ci/clean.sh) to bring stable brahmaputra identical to master. Change-Id: e7ac7b4cb358b907b8756c4b95cc6b6618a76913 Signed-off-by: Billy O'Mahony Signed-off-by: Tom Herbert Reviewed-by: Jonas Bjurel Reviewed-by: Mark D. Gray Build OVS with dpdk Builds OVS with DPDK. Uses dpdk 2.2.0 and Open vSwitch 2.5. Includes Multi-Queue and recent patches. Open vSwitch RPM is now built with DPDK dataplane. Upstream copr (Fedora) spec files and patches are used for building. Also includes fixes of RPM building and command line options. Adds dpdk arg to RPM test script. Use "pure" COPR spec files without modifications. Fixed some function problems. Addressed reviewer comments. Delete local copies of spec files Build dpdk snapshot instead of using upstream snapshot. Fix clean script. "stack" VM with its default configuration doesn't support sse3 instruction set required by DPDK. Include workaround to test RPM script in host instead of VM. Install prereqs for dpdk build. Use 'set -e' in all scripts for error termination. Add PyQt4 Pre-req. Move functions and respond to reviewers comments. Change-Id: I8d5892ff61a814271972e8d2279628d645b2831b Signed-off-by: Thomas F Herbert Conflicts: build/BuildAndTestOVS.sh build/build_ovs_rpm.sh build/config build/instack_ovs.sh build/test_ovs_rpm.sh ci/buildovs.sh Fix pathname to script. This patch fixes bug that is currently blocking the daily build. Change-Id: If3624ea4e93bc42c5fe633b0c30d506f2b70efd0 Signed-off-by: Thomas F Herbert Fix uploading of files and remove date from file name Change-Id: I33c3c1bbb5c9e7cd45e693de2db3230887cde7da Signed-off-by: Thomas F Herbert OVS Archive file name changed. Change message to call archive snapshot Change-Id: Ia07f6e38e2108f061dd616aa393cd4d98d7e8f11 Signed-off-by: Thomas F Herbert Fixes RPM build problem. Use correct copr repo. Fix some version number parsing and tarball and rpm naming that was specific to old repo and old DPDK version. Now supports DPDK 16.04-rc4. Change-Id: Ia64b38d676e945df27917f4d9aa90ba4872c8cb5 Signed-off-by: Thomas F Herbert Fixed base name of intermediate tarball name. Change-Id: I23e60800e83f216474328c461a2ac7aff453412f Signed-off-by: Thomas F Herbert Fix build problem by freezing copr commits Use correct variable for copr dpdk version Change-Id: I09c5e7060924771a75376aebe5bf3bde897c38eb Signed-off-by: Thomas F Herbert fuel plugin: Pick updated libpcre packages. Account for changes in SR3 iso that caused the plugin deploy to fail when it tried to downgrade libpcre packages. Change-Id: I6a11fc2e887a76cf879b1c1a398784da691b8329 Signed-off-by: Billy O'Mahony Reviewed-by: Jonas Bjurel Reviewed-by: Mark D. Gray Reviewed-by: Tom Herbert Reviewed-by: Michal Skalski Fix for new location of OVS and DPDK COPR repos. Change-Id: Ibee81cbee778bb1b542317ed41b94d01f86426bc Signed-off-by: Thomas F Herbert Final manual changes to bring stable/brahmaputra into line with master. Signed-off-by: Billy O'Mahony --- build/BuildAndTestOVS.sh | 150 ++++++++ build/build_dpdk_rpm.sh | 142 ++++++++ build/build_ovs_rpm.sh | 214 +++++++++++ build/config | 31 +- build/instack_ovs.sh | 401 +++++++++++++++++++++ build/test_ovs_rpm.sh | 138 +++++++ ci/build.sh | 1 + ci/buildovs.sh | 143 +++----- ci/clean.sh | 43 +++ ci/upload_artifacts.sh | 20 +- .../ovs_package/ubuntu/dependencies.txt | 6 +- 11 files changed, 1173 insertions(+), 116 deletions(-) create mode 100755 build/BuildAndTestOVS.sh create mode 100755 build/build_dpdk_rpm.sh create mode 100755 build/build_ovs_rpm.sh create mode 100755 build/instack_ovs.sh create mode 100755 build/test_ovs_rpm.sh create mode 100755 ci/clean.sh diff --git a/build/BuildAndTestOVS.sh b/build/BuildAndTestOVS.sh new file mode 100755 index 0000000..10a2f05 --- /dev/null +++ b/build/BuildAndTestOVS.sh @@ -0,0 +1,150 @@ +#!/bin/bash + +# Copyright (c) 2016 Open Platform for NFV Project, Inc. and its contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -e + +echo "===============================" +echo executing $0 $@ +echo + +usage() { + echo "$0 -a -d -g -h\ + -i -p -t -u -v -- Specify major release if special kernel is required\ + The default kernel is Centos 7.2 kernel after upgrade.\ + -d -- Specify dpdk build.\ + The default is to build ovs for linux kernel data path.\ + -g -- OVS release tag or branch to build such as 2.4.\ + The default is master.\ + -h print this message\ + -i -- Specify minor release if special kernel is required.\ + The default kernel is Centos 7.2 kernel after upgrade.\ + -p -- Specify url to patches if required for ovs rpm.\ + -t -- Test rpm.\ + -u -- path to OVS repo if using fork for patch.\ + The default is https://github.com/openvswitch/ovs.git\ + -v -- Set verbose mode." +} + +while getopts "a:dg:hi:p:tu:v" opt; do + case "$opt" in + a) + kernel_major=${OPTARG} + ;; + d) + DPDK="yes" + ;; + g) + TAG=${OPTARG} + ;; + h) + usage + exit 1 + ;; + i) + kernel_minor=${OPTARG} + ;; + p) + OVS_PATCH=${OPTARG} + ;; + t) + TESTRPM="yes" + ;; + u) + OVS_REPO_URL=${OPTARG} + ;; + v) + verbose="yes" + ;; + esac +done + +if [ -z $TAG ]; then + TAG=master +fi + +if [ -z $OVS_REPO_URL ]; then + OVS_REPO_URL=https://github.com/openvswitch/ovs.git +fi + +if [ ! -z $kernel_major ] && [ ! -z $kernel_minor ]; then + kernel_version=$kernel_major.$kernel_minor + echo =================== + echo Will install kernel version: major is $kernel_major and minor is $kernel_minor + echo =================== +else + echo Will use default kernel in ovs test vm +fi + +if [ ! -z $DPDK ]; then + setbuilddpdk="-d" +fi + +if [ -z ${WORKSPACE+1} ]; then + # We are not being run by Jenkins. + export WORKSPACE=$HOME/opnfv/ovsnfv + mkdir -p opnfv + cd opnfv + git clone https://git.opnfv.org/ovsnfv +fi + +export BUILD_BASE=$WORKSPACE/build + + + +if [ ! -d $BUILD_BASE ] +then + mkdir -p $BUILD_BASE +fi + +if [ ! -f $BUILD_BASE/config ]; then + touch $BUILD_BASE/config +fi + +export PATH=$PATH:$WORKSPACE/ci:$BUILD_BASE +source $BUILD_BASE/config + +cd $BUILD_BASE +export TOPDIR=$BUILD_BASE + +# build variables + +export TMP_RELEASE_DIR=$TOPDIR/release +export CACHE_DIR=$TOPDIR/cache +export TMPDIR=$TOPDIR/scratch +export RPMDIR=$TOPDIR/rpmbuild + + +mkdir -p $RPMDIR/RPMS +mkdir -p $RPMDIR/SOURCES +mkdir -p $RPMDIR/SPECS +mkdir -p $RPMDIR/SRPMS + + +if [ ! -z $TESTRPM ]; then + # Spawn VM to do the testing. + if [ ! -z $kernel_version ]; then + instack_ovs.sh -a $kernel_major -g $TAG -i $kernel_minor -p $OVS_PATCH -t -u $OVS_REPO_URL + else + instack_ovs.sh $setbuilddpdk -g $TAG -p $OVS_PATCH -t -u $OVS_REPO_URL + fi +else + # Run build locally. + build_ovs_rpm.sh $setbuilddpdk -g $TAG -p $OVS_PATCH -u $OVS_REPO_URL + cp $HOME/rpmbuild/RPMS/* $TMP_RELEASE_DIR +fi + +exit 0 diff --git a/build/build_dpdk_rpm.sh b/build/build_dpdk_rpm.sh new file mode 100755 index 0000000..1812dc3 --- /dev/null +++ b/build/build_dpdk_rpm.sh @@ -0,0 +1,142 @@ +#!/bin/bash + +# Copyright (c) 2016 Open Platform for NFV Project, Inc. and its contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -e + +echo "===============================" +echo executing $0 $@ +echo executing on machine `uname -a` + +usage() { + echo run BuildAndTestOVS -h for complete help on options on ovsnfv scripts. +} + +while getopts "g:hp:u:v" opt; do + case "$opt" in + g) + DPDK_VERSION=${OPTARG} + ;; + h|\?) + usage + exit 1 + ;; + p) + DPDK_PATCH=${OPTARG} + ;; + u) + DPDK_REPO_URL=${OPTARG} + ;; + v) + verbose="yes" + ;; + esac +done + +if [ -z $DPDK_REPO_URL ]; then + DPDK_REPO_URL=http://dpdk.org/git/dpdk +fi +if [ -z $DPDK_VERSION ]; then + DPDK_VERSION=2.2.0 +fi + +HOME=`pwd` +TOPDIR=$HOME +TMPDIR=$TOPDIR/rpms + +if [ -d $TMPDIR ] +then + rm -rf $TMPDIR +fi + +echo "---------------------" +echo "Install dependencies for dpdk" +echo +sudo yum -y install gcc make python-devel openssl-devel autoconf automake rpm-build \ + redhat-rpm-config libtool libpcap-devel numactl-devel python-sphinx \ + libvirt-devel + + +mkdir -p $TMPDIR + +cd $TMPDIR + +mkdir -p $HOME/rpmbuild/RPMS +mkdir -p $HOME/rpmbuild/SOURCES +mkdir -p $HOME/rpmbuild/SPECS +mkdir -p $HOME/rpmbuild/SRPMS + +RPMDIR=$HOME/rpmbuild + +# +# Use Fedora copr spec file +# +echo "---------------------" +echo "Get copr distribution git" +mkdir -p copr +cd copr +git clone https://github.com/tfherbert/dpdk-snap.git +cd dpdk-snap +git checkout $COPR_DPDK_VERSION + +echo "---------------------" +echo "Build DPDK RPM version $DPDK_VERSION" +echo +cd $TMPDIR +git clone $DPDK_REPO_URL +cd dpdk +if [[ "$DPDK_VERSION" =~ "master" ]]; then + git checkout master + snapgit=`git log --pretty=oneline -n1|cut -c1-8` +else + git checkout v$DPDK_VERSION + snapgit=`grep "define snapver" $TMPDIR/copr/dpdk-snap/dpdk.spec | cut -c25-33` +fi + +cp $TMPDIR/copr/dpdk-snap/dpdk.spec $TMPDIR/dpdk +cp $TMPDIR/copr/dpdk-snap/dpdk.spec $RPMDIR/SPECS +cp $TMPDIR/copr/dpdk-snap/*.patch $TMPDIR/copr/dpdk-snap/sources $TMPDIR/copr/dpdk-snap/dpdk-snapshot.sh $RPMDIR/SOURCES +snapser=`git log --pretty=oneline | wc -l` + +makever=`make showversion` +basever=`echo ${makever} | cut -d- -f1` +prefix=dpdk-${basever:0:5} + +archive=${prefix}.tar.gz +DPDK_VERSION=$basever + +echo "-------------------------------" +echo "Creating ${archive}" +echo +git archive --prefix=${prefix}/ HEAD | gzip -9 > ${archive} +cp ${archive} $RPMDIR/SOURCES/ +echo "-------------------------------" +echo building RPM for DPDK version $DPDK_VERSION +echo +rpmbuild -bb --define "_topdir $RPMDIR" dpdk.spec + +echo "-------------------------------" +echo Delete all rpms from $HOME +echo +set +e +rm $HOME/*.rpm +set -e + +echo "-------------------------------" +echo Copy dpdk RPM +echo +cp $RPMDIR/RPMS/x86_64/*.rpm $HOME + +exit 0 diff --git a/build/build_ovs_rpm.sh b/build/build_ovs_rpm.sh new file mode 100755 index 0000000..b03f4ab --- /dev/null +++ b/build/build_ovs_rpm.sh @@ -0,0 +1,214 @@ +#!/bin/bash + +# Copyright (c) 2016 Open Platform for NFV Project, Inc. and its contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -e + +echo "===============================" +echo executing $0 $@ +echo executing on machine `uname -a` + +usage() { + echo run BuildAndTestOVS -h for help +} + +function delrpm() { + set +e + rpm -q $1 + if [ $? -eq 0 ]; then + sudo rpm -e --allmatches $1 + fi + set -e +} +function cleanrpms() { + delrpm openvswitch + delrpm dpdk-devel + delrpm dpdk-tools + delrpm dpdk-examples + delrpm dpdk +} + +while getopts "cdg:hkp:u:v" opt; do + case "$opt" in + c) + setnocheck="--without check" + ;; + d) + DPDK="yes" + ;; + g) + TAG=${OPTARG} + ;; + h|\?) + usage + exit 1 + ;; + k) + kmod="yes" + ;; + p) + OVS_PATCH=${OPTARG} + ;; + u) + OVS_REPO_URL=${OPTARG} + ;; + v) + verbose="yes" + ;; + esac +done + +HOME=`pwd` +TOPDIR=$HOME +TMPDIR=$TOPDIR/ovsrpm + +echo "---------------------------------------" +echo Clean out old working dir +echo +if [ -d $TMPDIR ] +then + rm -rf $TMPDIR +fi + +echo "----------------------------------------" +echo Install pre-reqs. +echo +sudo yum -y install gcc make python-devel openssl-devel kernel-devel graphviz \ + kernel-debug-devel autoconf automake rpm-build redhat-rpm-config \ + libtool python-twisted-core desktop-file-utils groff PyQt4 + +VERSION=2.3.90 +os_type=fedora +kernel_version=$(uname -a | awk '{print $3}') + +RPMDIR=$HOME/rpmbuild + +echo "---------------------------------------" +echo Clean out old reminents of old rpms and rpm _topdir. +echo + +rm openvswitch*.rpm || true +if [ -d $RPMDIR ]; then + rm -rf $RPMDIR +fi + +echo "---------------------------------------" +echo Create new rpm _topdir. +echo +mkdir -p $HOME/rpmbuild/RPMS +mkdir -p $HOME/rpmbuild/SOURCES +mkdir -p $HOME/rpmbuild/SPECS +mkdir -p $HOME/rpmbuild/SRPMS + + +mkdir -p $TMPDIR + +cd $TMPDIR + +if [ ! -z $DPDK ]; then + echo "----------------------------------" + echo "Build OVS for dpdk. Use Fedora copr repo" + echo + echo "----------------------------------" + echo "Clone Fedora copr repo and copy files." + echo + git clone https://github.com/tfherbert/ovs-snap.git + cd ovs-snap + git checkout $COPR_OVS_VERSION + cp $TMPDIR/ovs-snap/openvswitch.spec $RPMDIR/SPECS + cp $TMPDIR/ovs-snap/* $RPMDIR/SOURCES + snapgit=`grep "define snapver" $TMPDIR/ovs-snap/openvswitch.spec | cut -c26-33` + echo "-------------------------------------------" + echo "Remove old dpdk, ovs and dpdk development rpms" + echo + cleanrpms + + if [ -z $DPDK_VERSION ]; then + DPDK_VERSION=16.04.0 + fi + echo "-------------------------------------------" + echo "Install dpdk and dpdk development rpms for version $DPDK_VERSION" + echo + sudo rpm -ivh $HOME/dpdk-${DPDK_VERSION:0:1}*.rpm + sudo rpm -ivh $HOME/dpdk-devel*.rpm + echo "----------------------------------------" + echo "Copy DPDK RPM to SOURCES" + echo + cp $HOME/*.rpm $RPMDIR/SOURCES + echo "--------------------------------------------" + echo "Get commit from $snapgit User Space OVS version $TAG" + echo + cd $TMPDIR + git clone $OVS_REPO_URL + cd $TMPDIR/ovs + git checkout $snapgit + echo "--------------------------------------------" + echo "Creating snapshot, $archive with name same as in spec file." + echo + snapser=`git log --pretty=oneline | wc -l` + basever=`grep AC_INIT configure.ac | cut -d' ' -f2 | cut -d, -f1` + prefix=openvswitch-${basever} + archive=${prefix}-${snapser}.git${snapgit}.tar.gz + git archive --prefix=${prefix}-${snapser}.git${snapgit}/ HEAD | gzip -9 > $RPMDIR/SOURCES/${archive} + cd $TMPDIR/ovs-snap + echo "--------------------------------------------" + echo "Build openvswitch RPM" + echo +else + echo "-------------------------------------------------" + echo "Build OVS without DPDK:" + echo "Use spec files for $os_type in OVS distribution." + echo + if [[ "$TAG" =~ "master" ]]; then + git clone $OVS_REPO_URL + cd ovs + + if [[ ! "$OVS_PATCH" =~ "no" ]]; then + echo "Apply patches from $OVS_PATCH" + fi + basever=`grep AC_INIT configure.ac | cut -d' ' -f2 | cut -d, -f1` + export VERSION=$basever + + echo "--------------------------------------------" + echo making distribution tarball for Open vswitch version $VERSION + echo + ./boot.sh + ./configure + make dist + + echo cp openvswitch-*.tar.gz $HOME/rpmbuild/SOURCES + cp openvswitch-*.tar.gz $HOME/rpmbuild/SOURCES + else + export VERSION=${TAG} + echo "---------------------------------------------" + echo "Get openvswith-${VERSION}.tar.gz" + echo + curl --silent --output $HOME/rpmbuild/SOURCES/openvswitch-${VERSION}.tar.gz http://openvswitch.org/releases/openvswitch-${VERSION}.tar.gz + fi + + if [ ! -z $kmod ]; then + echo "--------------------------------------------" + echo "Building openvswitch kernel module RPM" + echo + rpmbuild -bb -D "kversion $kernel_version" -D "kflavors default" --define "_topdir `echo $RPMDIR`" $setnocheck rhel/openvswitch-kmod-${os_type}.spec + fi + echo "--------------------------------------------" + echo "Build openvswitch RPM" + echo +fi +rpmbuild -bb --define "_topdir `echo $RPMDIR`" $setnocheck openvswitch.spec + +cp $RPMDIR/RPMS/x86_64/*.rpm $HOME + +exit 0 diff --git a/build/config b/build/config index bd2bf5e..350b192 100644 --- a/build/config +++ b/build/config @@ -15,6 +15,31 @@ # config file to be populated with configuration parameters for build. -# OVS TAG -export OVSTAG=master -#export NOCHECK=true +# OVS configuration. +# +# OVS TAG or revision to build. The default is master which causes an RPM to be +# built from the top of current master. The tag uses the current ovs naming and +# release convention. This option can be overriden in the command line. +# +export OVSTAG=2.4.0 +# +# when NOCHECK is yes, the ovs rpm is build with the --without check option which +# Disables running "make check" as the RPM is built. +# +export NOCHECK=yes +# +# URL to external URL for patches to be applied for Open vSwitch. +# +export PATCH=none +# +# DPDK configuration. +# +export WITH_DPDK=yes +export DPDK_PATCH=none +export DPDK_VERSION=16.04-rc4 +export COPR_DPDK_VERSION=039002f07eb1dd96a061235f0bf0493da386bba2 +export COPR_OVS_VERSION=5e311e125035edc706efbd9cc679f67e5972dbb4 +# +# When KMOD is set, the Linux kernel module is built and tested. +# +export KMOD=no diff --git a/build/instack_ovs.sh b/build/instack_ovs.sh new file mode 100755 index 0000000..4c258a4 --- /dev/null +++ b/build/instack_ovs.sh @@ -0,0 +1,401 @@ +#!/bin/bash +############################################################################## +# Copyright (c) 2015 Red Hat Inc. and others. +# therbert@redhat.com +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Apache License, Version 2.0 +# which accompanies this distribution, and is available at +# http://www.apache.org/licenses/LICENSE-2.0 +############################################################################## + +set -e +declare -i CNT + +echo "===============================" +echo executing $0 $@ +echo path is $PATH + +usage() { + echo run BuildAndTestOVS -h for help +} + +while getopts "a:dg:hi:p:tu:v" opt; do + case "$opt" in + a) + kernel_major=${OPTARG} + ;; + d) + DPDK="yes" + setdpdk="-d" + ;; + g) + TAG=${OPTARG} + ;; + h|\?) + usage + exit 1 + ;; + i) + kernel_minor=${OPTARG} + ;; + p) + OVS_PATCH=${OPTARG} + ;; + t) + TESTRPM="yes" + ;; + u) + OVS_REPO_URL=${OPTARG} + ;; + v) + verbose="yes" + ;; + esac +done +# +# Default Config options +# +echo =============================================== +echo Default Configuration Options. +echo =============================================== +echo option NOCHECK is set to $NOCHECK +echo build DPDK option is set to $DPDK +echo DPDK Patch URL is set to $DPDK_PATCH +echo DPDK Version is set to $DPDK_VERSION +echo Option for OVS Kernel Module is set to $KMOD +echo =============================================== +if [[ $NOCHECK =~ "yes" ]]; then + setnocheck="-c" +fi +if [[ $KMOD =~ "yes" ]]; then + setkmod="-k" +fi + + + +if [ -z $OVS_REPO_URL ]; then + OVS_REPO_URL=https://github.com/openvswitch/ovs.git +fi + +if [ ! -z $kernel_major ] && [ ! -z $kernel_minor ]; then + kernel_version=$kernel_major.$kernel_minor + echo =================== + echo Will install kernel version: major is $kernel_major and minor is $kernel_minor + echo =================== +else + echo Will use default kernel in ovs test vm +fi + +if [ -z ${WORKSPACE+1} ]; then + # We are not being run by Jenkins. + export WORKSPACE=$HOME/opnfv/ovsnfv + mkdir -p opnfv + cd opnfv + git clone https://git.opnfv.org/ovsnfv +fi + +export BUILD_BASE=$WORKSPACE/build + +if [ ! -d $BUILD_BASE ] +then + mkdir -p $BUILD_BASE +fi + +if [ ! -f $BUILD_BASE/config ]; then + touch $BUILD_BASE/config +fi + +export PATH=$PATH:$WORKSPACE/ci:$BUILD_BASE +source $BUILD_BASE/config + +cd $BUILD_BASE +export TOPDIR=$BUILD_BASE + +export TMP_RELEASE_DIR=$TOPDIR/release +if [ ! -d $TMP_RELEASE_DIR ]; then + mkdir -p $TMP_RELEASE_DIR +fi + +export CACHE_DIR=$TOPDIR/cache +if [ ! -d $CACHE_DIR ]; then + mkdir -p $CACHE_DIR +fi +export TMPDIR=$TOPDIR/scratch +if [ ! -d $SCRATCH_DIR ]; then + mkdir -p $SCRATCH_DIR +fi + +rdo_images_uri=https://ci.centos.org/artifacts/rdo/images/liberty/delorean/stable + +vm_index=4 +RDO_RELEASE=liberty +SSH_OPTIONS=(-o StrictHostKeyChecking=no -o GlobalKnownHostsFile=/dev/null -o UserKnownHostsFile=/dev/null) +OPNFV_NETWORK_TYPES="admin_network private_network public_network storage_network" + +# check for dependancy packages +for i in rpm-build createrepo libguestfs-tools python-docutils bsdtar; do + if ! rpm -q $i > /dev/null; then + sudo yum install -y $i + fi +done + +# RDO Manager expects a stack user to exist, this checks for one +# and creates it if you are root +if ! id stack > /dev/null; then + sudo useradd stack; + sudo echo 'stack ALL=(root) NOPASSWD:ALL' | sudo tee -a /etc/sudoers.d/stack + sudo echo 'Defaults:stack !requiretty' | sudo tee -a /etc/sudoers.d/stack + sudo chmod 0440 /etc/sudoers.d/stack + echo 'Added user stack' +fi + +# ensure that I can ssh as the stack user +if ! sudo grep "$(cat ~/.ssh/id_rsa.pub)" /home/stack/.ssh/authorized_keys; then + if ! sudo ls -d /home/stack/.ssh/ ; then + sudo mkdir /home/stack/.ssh + sudo chown stack:stack /home/stack/.ssh + sudo chmod 700 /home/stack/.ssh + fi + USER=$(whoami) sudo sh -c "cat ~$USER/.ssh/id_rsa.pub >> /home/stack/.ssh/authorized_keys" + sudo chown stack:stack /home/stack/.ssh/authorized_keys +fi + +# clean up stack user previously build instack disk images +ssh -T ${SSH_OPTIONS[@]} stack@localhost "rm -f instack*.qcow2" + +# Yum repo setup for building the undercloud +if ! rpm -q rdo-release > /dev/null && [ "$1" != "-master" ]; then + sudo yum -y install yum-plugin-priorities + sudo yum-config-manager --disable openstack-${RDO_RELEASE} + sudo curl -o /etc/yum.repos.d/delorean.repo http://trunk.rdoproject.org/centos7-liberty/current-passed-ci/delorean.repo + sudo curl -o /etc/yum.repos.d/delorean-deps.repo http://trunk.rdoproject.org/centos7-liberty/delorean-deps.repo + sudo rm -f /etc/yum.repos.d/delorean-current.repo +elif [ "$1" == "-master" ]; then + sudo yum -y install yum-plugin-priorities + sudo yum-config-manager --disable openstack-${RDO_RELEASE} + sudo curl -o /etc/yum.repos.d/delorean.repo http://trunk.rdoproject.org/centos7/current-passed-ci/delorean.repo + sudo curl -o /etc/yum.repos.d/delorean-deps.repo http://trunk.rdoproject.org/centos7-liberty/delorean-deps.repo + sudo rm -f /etc/yum.repos.d/delorean-current.repo +fi + +# ensure the undercloud package is installed so we can build the undercloud +if ! rpm -q instack-undercloud > /dev/null; then + sudo yum install -y python-tripleoclient +fi + +# ensure openvswitch is installed +if ! rpm -q openvswitch > /dev/null; then + sudo yum install -y openvswitch +fi + +# ensure libvirt is installed +if ! rpm -q libvirt-daemon-kvm > /dev/null; then + sudo yum install -y libvirt-daemon-kvm +fi + +# clean this up incase it's there +sudo rm -f /tmp/instack.answers + +# ensure that no previous undercloud VMs are running +sudo ../ci/clean.sh +# and rebuild the bare undercloud VMs +ssh -T ${SSH_OPTIONS[@]} stack@localhost < /dev/null && [ $CNT -gt 0 ]; do + echo -n "." + sleep 3 + CNT=CNT-1 +done +# TODO fail if CNT=0 + +# yum update undercloud and reboot. +ssh -T ${SSH_OPTIONS[@]} "root@$UNDERCLOUD" < /dev/null; then + yum install http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + fi + + yum -y install yum-plugin-priorities + curl -o /etc/yum.repos.d/delorean.repo http://trunk.rdoproject.org/centos7-liberty/current-passed-ci/delorean.repo + curl -o /etc/yum.repos.d/delorean-deps.repo http://trunk.rdoproject.org/centos7-liberty/delorean-deps.repo + + cp /root/.ssh/authorized_keys /home/stack/.ssh/authorized_keys + chown stack:stack /home/stack/.ssh/authorized_keys +EOI +# +# If using special kernel version, install on undercloud vm. +# +if [ ! -z $kernel_version ]; then + echo "Install special kernel version $kernel_version on undercloud" + ssh -T ${SSH_OPTIONS[@]} "root@$UNDERCLOUD" < /dev/null || echo -n '' +virsh undefine instack --remove-all-storage 2> /dev/null || echo -n '' +virsh vol-delete instack.qcow2 --pool default 2> /dev/null +rm -f /var/lib/libvirt/images/instack.qcow2 2> /dev/null + +# Clean off baremetal VMs in case they exist +for i in $(seq 0 $vm_index); do + virsh destroy baremetalbrbm_brbm1_$i 2> /dev/null || echo -n '' + virsh undefine baremetalbrbm_brbm1_$i --remove-all-storage 2> /dev/null || echo -n '' + virsh vol-delete baremetalbrbm_brbm1_${i}.qcow2 --pool default 2> /dev/null + rm -f /var/lib/libvirt/images/baremetalbrbm_brbm1_${i}.qcow2 2> /dev/null +done + +# Clean off brbm bridges +virsh net-destroy brbm 2> /dev/null +virsh net-undefine brbm 2> /dev/null +vs-vsctl del-br brbm 2> /dev/null + +virsh net-destroy brbm1 2> /dev/null +virsh net-undefine brbm1 2> /dev/null +vs-vsctl del-br brbm1 2> /dev/null + +# clean pub keys from root's auth keys +sed -i '/stack@instack.localdomain/d' /root/.ssh/authorized_keys +sed -i '/virtual-power-key/d' /root/.ssh/authorized_keys + + +echo "Cleanup Completed" diff --git a/ci/upload_artifacts.sh b/ci/upload_artifacts.sh index 470773d..45128bd 100755 --- a/ci/upload_artifacts.sh +++ b/ci/upload_artifacts.sh @@ -19,21 +19,19 @@ set -o nounset set -o pipefail # log info to console -echo "Uploading the OVS $VERSION RPM artifacts. " +echo "Uploading the OVS and DPDK RPM artifacts. " echo "-----------------------------------------" echo -export RPMFILE_D=openvswitch-debuginfo-$VERSION-1.x86_64.rpm -export RPMFILE=openvswitch-$VERSION-1.x86_64.rpm - -# upload artifact and additional files to google storage -echo gsutil cp $TMP_RELEASE_DIR/$RPMFILE_D gs://$GS_URL/opnfv-$DATE-$RPMFILE_D -gsutil cp $TMP_RELEASE_DIR/$RPMFILE_D gs://$GS_URL/opnfv-$DATE-$RPMFILE_D - -echo gsutil cp $TMP_RELEASE_DIR/$RPMFILE gs://$GS_URL/opnfv-$DATE-$RPMFILE -gsutil cp $TMP_RELEASE_DIR/$RPMFILE gs://$GS_URL/opnfv-$DATE-$RPMFILE +cd $TMP_RELEASE_DIR +for i in `ls *.rpm` +do + echo copying $i to gs://$GS_URL/ovs4opnfv + gsutil cp $TMP_RELEASE_DIR/$i gs://$GS_URL/ovs4opnfv-$i + echo +done echo echo "------------------------------------------------------" echo "Done!" -echo "Artifacts are available as http://$GS_URL/opnfv-$DATE-$RPMFILE" +echo "Artifacts are available in http://$GS_URL/ovs4opnfv/*.rpm" diff --git a/fuel-plugin-ovsnfv/ovs_package/ubuntu/dependencies.txt b/fuel-plugin-ovsnfv/ovs_package/ubuntu/dependencies.txt index 77188cf..08c6782 100644 --- a/fuel-plugin-ovsnfv/ovs_package/ubuntu/dependencies.txt +++ b/fuel-plugin-ovsnfv/ovs_package/ubuntu/dependencies.txt @@ -2,14 +2,14 @@ http://archive.ubuntu.com/ubuntu/pool/main/a/autoconf/autoconf_2.69-6_all.deb http://archive.ubuntu.com/ubuntu/pool/main/a/automake-1.14/automake_1.14.1-2ubuntu1_all.deb http://archive.ubuntu.com/ubuntu/pool/main/a/autotools-dev/autotools-dev_20130810.1_all.deb http://archive.ubuntu.com/ubuntu/pool/main/m/m4/m4_1.4.17-4_amd64.deb -http://archive.ubuntu.com/ubuntu/pool/main/p/pcre3/libpcre3_8.31-2ubuntu2.2_amd64.deb +http://archive.ubuntu.com/ubuntu/pool/main/p/pcre3/libpcre3_8.31-2ubuntu2.3_amd64.deb http://archive.ubuntu.com/ubuntu/pool/main/libs/libselinux/libselinux1_2.2.2-1ubuntu0.1_amd64.deb http://archive.ubuntu.com/ubuntu/pool/main/libs/libsepol/libsepol1_2.2-1ubuntu0.1_amd64.deb http://archive.ubuntu.com/ubuntu/pool/main/f/fuse/fuse_2.9.2-4ubuntu4.14.04.1_amd64.deb http://archive.ubuntu.com/ubuntu/pool/main/f/fuse/libfuse2_2.9.2-4ubuntu4.14.04.1_amd64.deb -http://archive.ubuntu.com/ubuntu/pool/main/p/pcre3/libpcrecpp0_8.31-2ubuntu2.2_amd64.deb +http://archive.ubuntu.com/ubuntu/pool/main/p/pcre3/libpcrecpp0_8.31-2ubuntu2.3_amd64.deb http://archive.ubuntu.com/ubuntu/pool/main/libs/libsepol/libsepol1-dev_2.2-1ubuntu0.1_amd64.deb -http://archive.ubuntu.com/ubuntu/pool/main/p/pcre3/libpcre3-dev_8.31-2ubuntu2.2_amd64.deb +http://archive.ubuntu.com/ubuntu/pool/main/p/pcre3/libpcre3-dev_8.31-2ubuntu2.3_amd64.deb http://archive.ubuntu.com/ubuntu/pool/main/libs/libselinux/libselinux1-dev_2.2.2-1ubuntu0.1_amd64.deb http://archive.ubuntu.com/ubuntu/pool/main/f/fuse/libfuse-dev_2.9.2-4ubuntu4.14.04.1_amd64.deb http://archive.ubuntu.com/ubuntu/pool/main/libt/libtool/libltdl-dev_2.4.2-1.7ubuntu1_amd64.deb -- cgit 1.2.3-korg