diff options
Diffstat (limited to 'ci')
| -rw-r--r-- | ci/README | 101 | ||||
| -rwxr-xr-x | ci/build.sh | 269 | ||||
| -rwxr-xr-x | ci/clean_cache.sh | 38 | ||||
| -rwxr-xr-x | ci/deploy.sh | 264 |
4 files changed, 672 insertions, 0 deletions
diff --git a/ci/README b/ci/README new file mode 100644 index 000000000..46698a739 --- /dev/null +++ b/ci/README @@ -0,0 +1,101 @@ +############################################################################## +# Copyright (c) 2015 Ericsson AB and others. +# stefan.k.berg@ericsson.com +# jonas.bjurel@ericsson.com +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Apache License, Version 2.0 +# which accompanies this distribution, and is available at +# http://www.apache.org/licenses/LICENSE-2.0 +############################################################################## + +Abstract +======== +The fuel/ci directory holds all Fuel@OPNFV programatic abstractions for +the OPNFV community release and continous integration pipeline. +There are two Fuel@OPNF autonomous scripts fo this, complying to the OPNFV CI pipeline guideline: + - build.sh + - deploy.sh + +USAGE +===== +For usage information of the CI/CD scripts, please run: +./build.sh -h +./deploy.sh -h + +Details on the CI/CD deployment framework +========================================= + +Overview and purpose +-------------------- +The CI/CD deployment script relies on a configuration structure, providing base +installer configuration (part of fuel repo: deploy/config), per POD specific +configuration (part of a separate classified POD configuration repo: securedlab +, plugin configuration (part of fuel repo: deploy/config/plugins), and +deployment scenario configuration (part of fuel repo: deploy/scenario). + +- The base installer configuration resembles the least common denominator of all + HW/POD environment and deployment scenarios (These configurations are + normally carried by the the installer projects in this case (fuel@OPNFV). +- Per POD specific configuration specifies POD unique parameters, the POD + parameter possible to alter is governed by the Fuel@OPNFV project. +- Plugin configuration - providing configuration of a specific plugin. + these configurations maintain there own namespace and are normally maintained + by collaborative projects building Fuel@OPNFV plugins +- Deployment scenario configuration - provides a high level, POD/HW environment + independent scenario configuration for a specifiv deployment. It defines what + features/plugins that shall be deployed - as well needed overrides of the base + installer-, POD/HW environment- configurations. Objects allowed to override + is governed by the Fuel@OPNFV project. + +Executing a deployment +---------------------- +deploy.sh must be executed locally at the target lab/pod/jumpserver +A configuration structure must be provided - see the section below. +It is straight forward to execute a deployment task - as an example: +sudo deploy.sh -b file:///home/jenkins/config -l LF-1 -p POD-2 -s +os_odl-l2_ha -i file:///home/jenkins/MyIso.iso + +-b and -i arguments should be expressed in URI style (eg: file://... +or http://.... The resources can thus be local or remote. + +Configuration repository structure +---------------------------------- +The CI deployment engine relies on a configuration directory/file structure +pointed to by the -b option described above. +Normally this points to the secure classified OPNFV securedlab repo to which +only jenkins and andmins have access to, but you may point to any local or +remote strcture fullfilling the diectory/file structure below. +The reason that this configuration structure needs to be secure/hidden +is that there are security sensitive information in the various configuration +files. +A local stripped version of this configuration structure with virtual +deployment configurations also exist under build/config/. +Following configuration directory and file structure should be adheare to: + +TOP +! ++---- labs + ! + +---- lab-name-1 + ! ! + ! +---- pod-name-1 + ! ! ! + ! ! +---- fuel + ! ! ! + ! ! +---- config + ! ! ! + ! ! +---- dea-pod-override.yaml + ! ! ! + ! ! +---- dha.yaml + ! ! + ! +---- pod-name-2 + ! ! + ! + +---- lab-name-2 + ! ! + + +Creating a deployment scenario +------------------------------ +Please find deploy/scenario/README for instructions on how to create a new +deployment scenario.
\ No newline at end of file diff --git a/ci/build.sh b/ci/build.sh new file mode 100755 index 000000000..82049cdcd --- /dev/null +++ b/ci/build.sh @@ -0,0 +1,269 @@ +#!/bin/bash +############################################################################## +# Copyright (c) 2015 Ericsson AB and others. +# stefan.k.berg@ericsson.com +# jonas.bjurel@ericsson.com +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Apache License, Version 2.0 +# which accompanies this distribution, and is available at +# http://www.apache.org/licenses/LICENSE-2.0 +############################################################################## + +############################################################################ +# BEGIN of usage description +# +usage () +{ +cat | more << EOF +xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +`basename $0`: Builds the Fuel@OPNFV stack + +usage: `basename $0` [-s spec-file] [-c cache-URI] [-l log-file] [-f flags] + [output-directory] + +OPTIONS: + -s spec-file (NOTE! DEPRECATED!) + define the build-spec file, default ../build/config.mk. The script only + verifies that the spec-file exists. + -c cache base URI + specifies the base URI to a build cache to be used/updated, supported + methods are http://, ftp:// and file:// + -l log-file + specifies the output log-file (stdout and stderr), if not specified + logs are output to console as normal + -v + version tag to be applied to the build result + -r + alternative remote access method script/program. curl is default. + -f flag[...] + build flags: + s: Do nothing, succeed + f: Do nothing, fail + D: Debug mode + P: Clear the local cache before building. This flag is only + valid if the "-c cache-URI" options has been specified and + and the method in the cache-URI is file:// (local cache). + + -h help, prints this help text + + output-directory, specifies the directory for the output artifacts + (.iso file). If no output-directory is specified, the current path + when calling the script is used. + + +Description: + +build.sh builds the opnfv .iso artifact. +To reduce build time it uses build caches on a local or remote location. A +cache is rebuilt and uploaded if either of the below conditions are met: +1) The P(opulate) flag is set and the -c cache-base-URI is provided and set + to the method file:// , if -c is + not provided the cache will stay local. +2) If a cache is invalidated by the make system - the exact logic is encoded + in the cache.mk of the different parts of the build. +3) A valid cache does not exist on the specified -c cache-base-URI. + +A cache has a blob (binary data) and a meta file in the format of: + <SHA1>.blob + <SHA1>.meta + +Logging is by default to console, but can be directed elsewhere with the -l +option in which case both stdout and stderr is redirected to that destination. + +Built in unit testing of components is enabled by adding the t(est) flag. + +Return codes: + - 0 Success! + - 1-99 Unspecified build error + - 100-199 Build system internal error (not build it self) + - 101 Build system instance busy + - 200 Build failure + +Examples: + build -c http://opnfv.org/artifactory/fuel/cache \ + -d ~/jenkins/genesis/fuel/ci/output -f ti + +NOTE: At current the build scope is set to the git root of the repository, -d + destination locations outside that scope will not work! +EOF +} +# +# END of usage description +############################################################################ + +############################################################################ +# BEGIN of function error_exit + +error_exit() { + echo "$@" >&2 + exit 1 +} + +# +# END of function error_exit +############################################################################ + + +############################################################################ +# BEGIN of shorthand variables for internal use +# +SCRIPT_DIR=$(readlink -f $(dirname ${BASH_SOURCE[0]})) +BUILD_BASE=$(readlink -e ${SCRIPT_DIR}/../build/) +RESULT_DIR="${BUILD_BASE}/release" +BUILD_SPEC="${BUILD_BASE}/config.mk" +LOCAL_CACHE_ARCH_NAME="${LOCAL_CACHE_ARCH_NAME:-fuel-cache}" + +# +# END of variables to customize +############################################################################ + +############################################################################ +# BEGIN of script assigned default variables +# +export CACHEBASE="file://$HOME/cache" +export CACHETRANSPORT="curl --silent" +CLEAR_CACHE=0 +MAKE_ARGS="" + +# +# END of script assigned variables +############################################################################ + +build() { + echo "CI build parameters:" + echo "SCRIPT_DIR = $SCRIPT_DIR" + echo "BUILD_BASE = $BUILD_BASE" + echo "RESULT_DIR = $RESULT_DIR" + echo "BUILD_SPEC = $BUILD_SPEC" + echo "LOCAL_CACHE_ARCH_NAME = $LOCAL_CACHE_ARCH_NAME" + echo "CLEAR_CACHE = $CLEAR_CACHE" + echo "DEBUG = $DEBUG" + echo "OUTPUT_DIR = $OUTPUT_DIR" + echo "BUILD_LOG = $BUILD_LOG" + echo "MAKE_ARGS = $MAKE_ARGS" + echo "CACHEBASE = $CACHEBASE" + echo "CACHETRANSPORT = $CACHETRANSPORT" + + + if [ "$CLEAR_CACHE" -eq 1 ]; then + echo $CACHEBASE | grep -q '^file://' $CACHE_BASE + if [ $? -ne 0 ]; then + error_exit "Can't clear a non-local cache!" + else + CACHEDIR=$(echo $CACHEBASE | sed 's;file://;;') + echo "Clearing local cache at $CACHEDIR..." + rm -rvf $CACHEDIR/* + fi + fi + + echo make ${MAKE_ARGS} cache + + cd ${BUILD_BASE} + if make ${MAKE_ARGS} cache; then + echo "Copying build result into $OUTPUT_DIR" + sort ${BUILD_BASE}/gitinfo*.txt > ${OUTPUT_DIR}/gitinfo.txt + cp ${RESULT_DIR}/*.iso ${OUTPUT_DIR} + cp ${RESULT_DIR}/*.iso.txt ${OUTPUT_DIR} + else + error_exit "Build failed" + fi +} + +############################################################################ +# BEGIN of main +# +while getopts "s:c:l:v:f:r:f:h" OPTION +do + case $OPTION in + s) + BUILD_SPEC=${OPTARG} + if [ ! -f ${BUILD_SPEC} ]; then + echo "spec file does not exist: $BUILD_SPEC - exiting ...." + exit 100 + fi + ;; + c) + # This value is used by cache.sh + export CACHEBASE=${OPTARG} + ;; + l) + BUILD_LOG=$(readlink -f ${OPTARG}) + ;; + v) + MAKE_ARGS+="REVSTATE=${OPTARG}" + ;; + r) + # This value is used by cache.sh + export CACHETRANSPORT=${OPTARG} + ;; + h) + usage + rc=0 + exit $rc + ;; + f) + BUILD_FLAGS=${OPTARG} + for ((i=0; i<${#BUILD_FLAGS};i++)); do + case ${BUILD_FLAGS:$i:1} in + s) + exit 0 + ;; + + f) + exit 1 + ;; + + P) + CLEAR_CACHE=1 + ;; + + D) + DEBUG=1 + ;; + + *) + error_exit "${BUILD_FLAGS:$i:1} is not a valid build flag - exiting ...." + ;; + esac + done + ;; + + *) + echo "${OPTION} is not a valid argument" + rc=100 + exit $rc + ;; + esac +done + +# Get output directory +shift $[$OPTIND - 1] +case $# in + 0) + # No directory on command line + OUTPUT_DIR=$(pwd) + ;; + 1) + # Directory on command line + OUTPUT_DIR=$(readlink -f $1) + ;; + *) + error_exit "Too many arguments" + ;; +esac +mkdir -p $OUTPUT_DIR || error_exit "Could not access output directory $OUTPUT_DIR" + + +if [ -n "${BUILD_LOG}" ]; then + touch ${BUILD_LOG} || error_exit "Could not write to log file ${BUILD_LOG}" + build 2>&1 | tee ${BUILD_LOG} +else + build +fi + +rc=$? +exit $rc + +# +# END of main +############################################################################ diff --git a/ci/clean_cache.sh b/ci/clean_cache.sh new file mode 100755 index 000000000..177fe821b --- /dev/null +++ b/ci/clean_cache.sh @@ -0,0 +1,38 @@ +#!/bin/bash +############################################################################## +# Copyright (c) 2016 Ericsson AB and others. +# stefan.k.berg@ericsson.com +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Apache License, Version 2.0 +# which accompanies this distribution, and is available at +# http://www.apache.org/licenses/LICENSE-2.0 +############################################################################## + +# Clean the build cache according to its expiry date. Invoke with the cache +# directory as the first argument. + +if [ -z "$1" ]; then + echo "No cache directory specified, exiting..." + exit 1 +else + CACHEDIR=$1 + echo "Operating on cache $CACHEDIR" +fi + +NOW=$(date '+%s') + +cd $CACHEDIR +echo "Step 1, cleaning orphaned meta and blob files" +ls *.meta *.blob | sed 's/\..*//' | sort | uniq -u | xargs -n 1 -I {} sh -c "rm -vf {}.*" +echo "Step 2, cleaning expired files" +for cache in $(ls -1 *.meta | sed 's/\..*//') +do + blob=${cache}.blob + meta=${cache}.meta + expiry=$(grep Expires: $meta | sed 's/Expires: *//') + if [ $expiry -le $NOW ]; then + echo "$cache expired $(date -d "@$expiry"), removing..." + rm -f $blob $meta + fi +done + diff --git a/ci/deploy.sh b/ci/deploy.sh new file mode 100755 index 000000000..8411714eb --- /dev/null +++ b/ci/deploy.sh @@ -0,0 +1,264 @@ +#!/bin/bash +set -e +############################################################################## +# Copyright (c) 2015 Ericsson AB and others. +# jonas.bjurel@ericsson.com +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Apache License, Version 2.0 +# which accompanies this distribution, and is available at +# http://www.apache.org/licenses/LICENSE-2.0 +############################################################################## + +############################################################################ +# BEGIN of Exit handlers +# +do_exit () { + clean + echo "Exiting ..." +} +# +# End of Exit handlers +############################################################################ + +############################################################################ +# BEGIN of usage description +# +usage () +{ +cat << EOF +xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +`basename $0`: Deploys the Fuel@OPNFV stack + +usage: `basename $0` -b base-uri [-B PXE Bridge] [-f] [-F] [-H] -l lab-name -p pod-name -s deploy-scenario [-S image-dir] [-T timeout] -i iso + -s deployment-scenario [-S optional Deploy-scenario path URI] + [-R optional local relen repo (containing deployment Scenarios] + +OPTIONS: + -b Base-uri for the stack-configuration structure + -B PXE Bridge for booting of Fuel master + -d Dry-run + -f Deploy on existing Fuel master + -e Do not launch environment deployment + -F Do only create a Fuel master + -h Print this message and exit + -H No health check + -l Lab-name + -L Deployment log path and file name + -p Pod-name + -s Deploy-scenario short-name/base-file-name + -S Storage dir for VM images + -T Timeout, in minutes, for the deploy. + -i iso url + +Description: +Deploys the Fuel@OPNFV stack on the indicated lab resource + +This script provides the Fuel@OPNFV deployment abstraction +It depends on the OPNFV official configuration directory/file structure +and provides a fairly simple mechanism to execute a deployment. +Input parameters to the build script is: +-b Base URI to the configuration directory (needs to be provided in a URI + style, it can be a local resource: file:// or a remote resource http(s)://) +-B PXE Bridge for booting of Fuel master. It can be specified several times, + or as a comma separated list of bridges, or both: -B br1 -B br2,br3 + One NIC connected to each specified bridge will be created in the Fuel VM, + in the same order as provided in the command line. The default is pxebr. +-d Dry-run - Produces deploy config files (config/dea.yaml and + config/dha.yaml), but does not execute deploy +-f Deploy on existing Fuel master +-e Do not launch environment deployment +-F Do only create a Fuel master +-h Print this message and exit +-H Do not run fuel built in health-check after successfull deployment +-l Lab name as defined in the configuration directory, e.g. lf +-L Deployment log path and name, eg. -L /home/jenkins/logs/job888.log.tar.gz +-p POD name as defined in the configuration directory, e.g. pod-1 +-s Deployment-scenario, this points to a deployment/test scenario file as + defined in the configuration directory: + e.g fuel-ocl-heat-ceilometer_scenario_0.0.1.yaml + or a deployment short-name as defined by scenario.yaml in the deployment + scenario path. +-S Storage dir for VM images, default is fuel/deploy/images +-T Timeout, in minutes, for the deploy. It defaults to using the DEPLOY_TIMEOUT + environment variable when defined, or to the default in deploy.py otherwise +-i .iso image to be deployed (needs to be provided in a URI + style, it can be a local resource: file:// or a remote resource http(s)://) + +NOTE: Root priviledges are needed for this script to run + + +Examples: +sudo `basename $0` -b file:///home/jenkins/lab-config -l lf -p pod1 -s ha_odl-l3_heat_ceilometer -i file:///home/jenkins/myiso.iso +EOF +} + +# +# END of usage description +############################################################################ + +############################################################################ +# BEGIN of deployment clean-up +# +clean() { + echo "Cleaning up deploy tmp directories" + rm -rf ${SCRIPT_PATH}/ISO +} +# +# END of deployment clean-up +############################################################################ + +############################################################################ +# BEGIN of shorthand variables for internal use +# +SCRIPT_PATH=$(readlink -f $(dirname ${BASH_SOURCE[0]})) +DEPLOY_DIR=$(cd ${SCRIPT_PATH}/../deploy; pwd) +PXE_BRIDGE='' +NO_HEALTH_CHECK='' +USE_EXISTING_FUEL='' +FUEL_CREATION_ONLY='' +NO_DEPLOY_ENVIRONMENT='' +STORAGE_DIR='' +DRY_RUN=0 +if ! [ -z $DEPLOY_TIMEOUT ]; then + DEPLOY_TIMEOUT="-dt $DEPLOY_TIMEOUT" +else + DEPLOY_TIMEOUT="" +fi +# +# END of variables to customize +############################################################################ + +############################################################################ +# BEGIN of main +# +while getopts "b:B:dfFHl:L:p:s:S:T:i:he" OPTION +do + case $OPTION in + b) + BASE_CONFIG_URI=${OPTARG} + if [[ ! $BASE_CONFIG_URI == file://* ]] && \ + [[ ! $BASE_CONFIG_URI == http://* ]] && \ + [[ ! $BASE_CONFIG_URI == https://* ]] && \ + [[ ! $BASE_CONFIG_URI == ftp://* ]]; then + echo "-b $BASE_CONFIG_URI - Not given in URI style" + usage + exit 1 + fi + ;; + B) + for bridge in ${OPTARG//,/ }; do + PXE_BRIDGE+=" -b $bridge" + done + ;; + d) + DRY_RUN=1 + ;; + f) + USE_EXISTING_FUEL='-nf' + ;; + F) + FUEL_CREATION_ONLY='-fo' + ;; + e) + NO_DEPLOY_ENVIRONMENT='-nde' + ;; + H) + NO_HEALTH_CHECK='-nh' + ;; + l) + TARGET_LAB=${OPTARG} + ;; + L) + DEPLOY_LOG="-log ${OPTARG}" + ;; + p) + TARGET_POD=${OPTARG} + ;; + s) + DEPLOY_SCENARIO=${OPTARG} + ;; + S) + if [[ ${OPTARG} ]]; then + STORAGE_DIR="-s ${OPTARG}" + fi + ;; + T) + DEPLOY_TIMEOUT="-dt ${OPTARG}" + ;; + i) + ISO=${OPTARG} + if [[ ! $ISO == file://* ]] && \ + [[ ! $ISO == http://* ]] && \ + [[ ! $ISO == https://* ]] && \ + [[ ! $ISO == ftp://* ]]; then + echo "-i $ISO - Not given in URI style" + usage + exit 1 + fi + ;; + h) + usage + exit 0 + ;; + *) + echo "${OPTION} is not a valid argument" + echo "Arguments not according to new argument style" + echo "Trying old-style compatibility mode" + pushd ${DEPLOY_DIR} > /dev/null + python deploy.py "$@" + popd > /dev/null + exit 0 + ;; + esac +done + +if [[ $EUID -ne 0 ]]; then + echo "This script must be run as root" 1>&2 + exit 1 +fi + +if [ -z $BASE_CONFIG_URI ] || [ -z $TARGET_LAB ] || \ + [ -z $TARGET_POD ] || [ -z $DEPLOY_SCENARIO ] || \ + [ -z $ISO ]; then + echo "Arguments not according to new argument style" + echo "Trying old-style compatibility mode" + pushd ${DEPLOY_DIR} > /dev/null + python deploy.py "$@" + popd > /dev/null + exit 0 +fi + +# Enable the automatic exit trap +trap do_exit SIGINT SIGTERM EXIT + +# Set no restrictive umask so that Jenkins can removeeee any residuals +umask 0000 + +clean + +pushd ${DEPLOY_DIR} > /dev/null +# Prepare the deploy config files based on lab/pod information, deployment +# scenario, etc. + +echo "python deploy-config.py -dha ${BASE_CONFIG_URI}/labs/${TARGET_LAB}/${TARGET_POD}/fuel/config/dha.yaml -deab file://${DEPLOY_DIR}/config/dea_base.yaml -deao ${BASE_CONFIG_URI}/labs/${TARGET_LAB}/${TARGET_POD}/fuel/config/dea-pod-override.yaml -scenario-base-uri file://${DEPLOY_DIR}/scenario -scenario ${DEPLOY_SCENARIO} -plugins file://${DEPLOY_DIR}/config/plugins -output ${SCRIPT_PATH}/config" + +python deploy-config.py -dha ${BASE_CONFIG_URI}/labs/${TARGET_LAB}/${TARGET_POD}/fuel/config/dha.yaml -deab file://${DEPLOY_DIR}/config/dea_base.yaml -deao ${BASE_CONFIG_URI}/labs/${TARGET_LAB}/${TARGET_POD}/fuel/config/dea-pod-override.yaml -scenario-base-uri file://${DEPLOY_DIR}/scenario -scenario ${DEPLOY_SCENARIO} -plugins file://${DEPLOY_DIR}/config/plugins -output ${SCRIPT_PATH}/config + +if [ $DRY_RUN -eq 0 ]; then + # Download iso if it doesn't already exists locally + if [[ $ISO == file://* ]]; then + ISO=${ISO#file://} + else + mkdir -p ${SCRIPT_PATH}/ISO + curl -o ${SCRIPT_PATH}/ISO/image.iso $ISO + ISO=${SCRIPT_PATH}/ISO/image.iso + fi + # Start deployment + echo "python deploy.py $DEPLOY_LOG $STORAGE_DIR $PXE_BRIDGE $USE_EXISTING_FUEL $FUEL_CREATION_ONLY $NO_HEALTH_CHECK $NO_DEPLOY_ENVIRONMENT -dea ${SCRIPT_PATH}/config/dea.yaml -dha ${SCRIPT_PATH}/config/dha.yaml -iso $ISO $DEPLOY_TIMEOUT" + python deploy.py $DEPLOY_LOG $STORAGE_DIR $PXE_BRIDGE $USE_EXISTING_FUEL $FUEL_CREATION_ONLY $NO_HEALTH_CHECK $NO_DEPLOY_ENVIRONMENT -dea ${SCRIPT_PATH}/config/dea.yaml -dha ${SCRIPT_PATH}/config/dha.yaml -iso $ISO $DEPLOY_TIMEOUT +fi +popd > /dev/null + +# +# END of main +############################################################################ |