A simple method to separate configuration for base fuel, plugins, PODs

and deployment/test scenarios

READY TO MERGE!

Replaces: https://gerrit.opnfv.org/gerrit/#/c/3995/

Abstract
--------
This deployment framework relies on a configuration structure,
providing base installer configuration, per POD specific configuration,
plugin configuration, and deployment scenario configuration.
- The base installer configuration resembles the least common denominator
  of all HW/POD environment and deployment scenarios (These configurations
  are normally carried by the the installer projects in this case (fuel@OPNFV).
- Per POD specific configuration specifies POD unique parameters, the POD
  parameter possible to alter is governed by the Fuel@OPNFV project.
- Plugin configuration - providing configuration of a specific plugin.
  these configurations maintain there own namespace and are normally maintained
  by collaborative projects building Fuel@OPNFV plugins
- Deployment scenario configuration - provides a high level, POD/HW environment
  independent scenario configuration for a specific deployment. It defines what
  features/plugins that shall be deployed - as well needed overrides of the
  plugin config as well as the base installer-, POD/HW environment-
  configurations. Objects allowed to override
  is governed by the Fuel@OPNFV project.

Executing a deployment
----------------------
deploy.sh must be executed locally at the target lab/pod/jumpserver
A lab configuration structure must be provided - see the section below.
It is straight forward to execute a deployment task - as an example:
sudo deploy.sh -b file:///home/jenkins/config -l ericsson-1 -p pod-2
-s os_odl-l2_no-ha -i file:///home/jenkins/MyIso.iso

-b and -i arguments should be expressed in URI style. The resources can thus be
local or remote.

Feedback
--------
Please give feed-back before I'm going to far on a wrong tangent

Implemented scenarios so far:
-----------------------------
- os_ha
- os_no-ha
- os_odl-l3_ha
- os_odl-l3_no-ha
- os_odl-l2_ha
- os_odl-l2_no-ha
- os_onos_ha
- os_onos_no-ha
- os_kvm_ha
- os_kvm_no-ha
- os_ovs_ha
- os_ovs_no-ha
- os_kvm_ovs_ha
- os_kvm_ovs_no-ha

VERIFIED
READY TO MERGE

JIRA: FUEL-35

Change-Id: I94a9b477d8ed4ee8057c16d8f20fe543f7ecc20d
Signed-off-by: Jonas Bjurel <jonas.bjurel@ericsson.com>
(cherry picked from commit 7fd2619e0df370fa832eeff00790bcaa52dc4ffc)

2 files changed, 264 insertions, 97 deletions

diff --git a/ci/README b/ci/README
index aab823bc0..46698a739 100644
--- a/ci/README
+++ b/ci/README
@@ -8,99 +8,94 @@
 # http://www.apache.org/licenses/LICENSE-2.0
 ##############################################################################
 
-The genesis/fuel/ci directory holds all Fuel@OPNFV abstractions for the OPNFV
-community release and continous integration pipeline.
+Abstract
+========
+The fuel/ci directory holds all Fuel@OPNFV programatic abstractions for
+the OPNFV community release and continous integration pipeline.
 There are two Fuel@OPNF autonomous scripts fo this, complying to the OPNFV CI pipeline guideline:
  - build.sh
  - deploy.sh
 
-For more info on usage:
+USAGE
+=====
+For usage information of the CI/CD scripts, please run:
 ./build.sh -h
-sudo ./deploy.sh -h
-python deploy.py -h
-
-usage: python deploy.py [-h] [-nf] [-nh] [-fo] [-co] [-c] [-iso [ISO_FILE]]
-                        [-dea [DEA_FILE]] [-dha [DHA_FILE]] [-s STORAGE_DIR]
-                        [-b PXE_BRIDGE] [-p FUEL_PLUGINS_DIR]
-
-optional arguments:
-  -h, --help           show this help message and exit
-  -nf                  Do not install Fuel Master (and Node VMs when using
-                       libvirt)
-  -nh                  Don't run health check after deployment
-  -fo                  Install Fuel Master only (and Node VMs when using
-                       libvirt)
-  -co                  Cleanup VMs and Virtual Networks according to what is
-                       defined in DHA
-  -c                   Cleanup after deploy
-  -iso [ISO_FILE]      ISO File [default: OPNFV.iso]
-  -dea [DEA_FILE]      Deployment Environment Adapter: dea.yaml
-  -dha [DHA_FILE]      Deployment Hardware Adapter: dha.yaml
-  -s STORAGE_DIR       Storage Directory [default: images]
-  -b PXE_BRIDGE        Linux Bridge for booting up the Fuel Master VM
-                       [default: pxebr]
-  -p FUEL_PLUGINS_DIR  Fuel Plugins directory
-
-
-
-* EXAMPLES:
-
-- Install Fuel Master and deploy OPNFV Cloud from scratch on Hardware Environment:
-
-    sudo ./deploy.sh -iso ~/ISO/opnfv.iso -dea ~/CONF/hardware/dea.yaml -dha ~/CONF/hardware/dha.yaml -s /mnt/images -b pxebr
-
-
-- Install Fuel Master and deploy OPNFV Cloud from scratch on Virtual Environment:
-
-    sudo ./deploy.sh -iso ~/ISO/opnfv.iso -dea ~/CONF/virtual/dea.yaml -dha ~/CONF/virtual/dha.yaml -s /mnt/images
-
-
-- Deploy OPNFV Cloud on an already active Environment where Fuel Master VM is running so no need to install Fuel again:
-
-    sudo ./deploy.sh -nf -dea ~/CONF/virtual/dea.yaml -dha ~/CONF/virtual/dha.yaml
-
-    => with plugin installation
-    sudo ./deploy.sh -nf -dea ~/CONF/virtual/dea.yaml -dha ~/CONF/virtual/dha.yaml
-
-    => with cleanup after deployment is finished
-    sudo ./deploy.sh -nf -dea ~/CONF/virtual/dea.yaml -dha ~/CONF/virtual/dha.yaml -c
-
-    => no healthcheck after deployment is completed
-    sudo ./deploy.sh -nf -dea ~/CONF/virtual/dea.yaml -dha ~/CONF/virtual/dha.yaml -nh
-
-
-- Install Fuel Master only (and Node VMs when using virtual environment):
-
-    => for virtual environment:
-    sudo ./deploy.sh -iso ~/ISO/opnfv.iso -dea ~/CONF/virtual/dea.yaml -dha ~/CONF/virtual/dha.yaml -s /mnt/images
-
-    => for hardware environment:
-    sudo ./deploy.sh -iso ~/ISO/opnfv.iso -dea ~/CONF/hardware/dea.yaml -dha ~/CONF/hardware/dha.yaml -s /mnt/images -b pxebr
-
-
-- Cleanup a running OPNFV environment:
-
-    sudo ./deploy.sh -co -dha ~/CONF/virtual/dha.yaml
-
-
-* WARNINGS:
-
-=>  If optional argument -s <storage_dir> is not specified, Autodeployment will use
-"<current_working_dir>/images" as default, and it will create it, if it hasn't been created before
-
-=>  If optional argument -b <pxe_bridge> is not specified, Autodeployment will use "pxebr" as default,
-if the bridge does not exist, the application will terminate with an error message
-
-=>  If argument -iso [ISO_FILE] is not specified, Autodeployment will use "<current_working_dir>/OPNFV.iso"
-as default, if the iso file does not exist, the application will terminate with an error message
-
-=>  If argument -dea [DEA_FILE] is not specified, Autodeployment will use "<current_working_dir>/dea.yaml"
-as default, if DEA file does not exist, the application will terminate with an error message
-
-=>  If argument -dha [DHA_FILE] is not specified, Autodeployment will use "<current_working_dir>/dha.yaml"
-as default, if DHA file does not exist, the application will terminate with an error message
-
-=> Optional argument -b PXE_BRIDGE is not required for Autodeployment in virtual environment,
-   even if it is specified it will not be used at all because virtual environment is using a different virtual network setup
-
-=> If optional argument -p FUEL_PLUGINS_DIR is not specified, no external plugins will be installed in Fuel
\ No newline at end of file
+./deploy.sh -h
+
+Details on the CI/CD deployment framework
+=========================================
+
+Overview and purpose
+--------------------
+The CI/CD deployment script relies on a configuration structure, providing base
+installer configuration (part of fuel repo: deploy/config), per POD specific
+configuration (part of a separate classified POD configuration repo: securedlab
+, plugin configuration (part of fuel repo: deploy/config/plugins), and
+deployment scenario configuration (part of fuel repo: deploy/scenario).
+
+- The base installer configuration resembles the least common denominator of all
+  HW/POD environment and deployment scenarios (These configurations are
+  normally carried by the the installer projects in this case (fuel@OPNFV).
+- Per POD specific configuration specifies POD unique parameters, the POD
+  parameter possible to alter is governed by the Fuel@OPNFV project.
+- Plugin configuration - providing configuration of a specific plugin.
+  these configurations maintain there own namespace and are normally maintained
+  by collaborative projects building Fuel@OPNFV plugins
+- Deployment scenario configuration - provides a high level, POD/HW environment
+  independent scenario configuration for a specifiv deployment. It defines what
+  features/plugins that shall be deployed - as well needed overrides of the base
+  installer-, POD/HW environment- configurations. Objects allowed to override
+  is governed by the Fuel@OPNFV project.
+
+Executing a deployment
+----------------------
+deploy.sh must be executed locally at the target lab/pod/jumpserver
+A configuration structure must be provided - see the section below.
+It is straight forward to execute a deployment task - as an example:
+sudo deploy.sh -b file:///home/jenkins/config -l LF-1 -p POD-2 -s
+os_odl-l2_ha -i file:///home/jenkins/MyIso.iso
+
+-b and -i arguments should be expressed in URI style (eg: file://...
+or http://.... The resources can thus be local or remote.
+
+Configuration repository structure
+----------------------------------
+The CI deployment engine relies on a configuration directory/file structure
+pointed to by the -b option described above.
+Normally this points to the secure classified OPNFV securedlab repo to which
+only jenkins and andmins have access to, but you may point to any local or
+remote strcture fullfilling the diectory/file structure below.
+The reason that this configuration structure needs to be secure/hidden
+is that there are security sensitive information in the various configuration
+files.
+A local stripped version of this configuration structure with virtual
+deployment configurations also exist under build/config/.
+Following configuration directory and file structure should be adheare to:
+
+TOP
+!
++---- labs
+       !
+       +---- lab-name-1
+       !        !
+       !        +---- pod-name-1
+       !        !        !
+       !        !        +---- fuel
+       !        !               !
+       !        !               +---- config
+       !        !                       !
+       !        !                       +---- dea-pod-override.yaml
+       !        !                       !
+       !        !                       +---- dha.yaml
+       !        !
+       !        +---- pod-name-2
+       !                 !
+       !
+       +---- lab-name-2
+       !        !
+
+
+Creating a deployment scenario
+------------------------------
+Please find deploy/scenario/README for instructions on how to create a new
+deployment scenario.
\ No newline at end of file
diff --git a/ci/deploy.sh b/ci/deploy.sh
index d5b70d0d6..d1f411fa5 100755
--- a/ci/deploy.sh
+++ b/ci/deploy.sh
@@ -1,8 +1,180 @@
 #!/bin/bash
+##############################################################################
+# Copyright (c) 2015 Ericsson AB and others.
+# jonas.bjurel@ericsson.com
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Apache License, Version 2.0
+# which accompanies this distribution, and is available at
+# http://www.apache.org/licenses/LICENSE-2.0
+##############################################################################
+
+############################################################################
+# BEGIN of Exit handlers
+#
+do_exit () {
+    clean
+    echo "Exiting ..."
+}
+#
+# End of Exit handlers
+############################################################################
+
+############################################################################
+# BEGIN of usage description
+#
+usage ()
+{
+cat << EOF
+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+`basename $0`: Deploys the Fuel@OPNFV stack
+
+usage: `basename $0` -b base-uri -l lab-name -p pod-name -i iso
+       -s deployment-scenario [-S optional Deploy-scenario path URI]
+       [-R optional local relen repo (containing deployment Scenarios]
+
+OPTIONS:
+  -b  Base-uri for the stack-configuration structure
+  -l  Lab-name
+  -p  Pod-name
+  -s  Deploy-scenario short-name/base-file-name
+  -i  iso url
+
+Description:
+Deploys the Fuel@OPNFV stack on the indicated lab resource
+
+This script provides the Fuel@OPNFV deployment abstraction
+It depends on the OPNFV official configuration directory/file structure
+and provides a fairly simple mechanism to execute a deployment.
+Input parameters to the build script is:
+-b Base URI to the configuration directory (needs to be provided in a URI
+   style, it can be a local resource: file:// or a remote resource http(s)://)
+-l Lab name as defined in the configuration directory, e.g. lf
+-p POD name as defined in the configuration directory, e.g. pod-1
+-s Deployment-scenario, this points to a deployment/test scenario file as
+   defined in the configuration directory:
+   e.g fuel-ocl-heat-ceilometer_scenario_0.0.1.yaml
+   or a deployment short-name as defined by scenario.yaml in the deployment
+   scenario path.
+-i .iso image to be deployed (needs to be provided in a URI
+   style, it can be a local resource: file:// or a remote resource http(s)://)
+
+NOTE: Root priviledges are needed for this script to run
+
+Examples:
+sudo `basename $0` -b file:///home/jenkins/lab-config -l lf -p pod1 -s ha_odl-l3_heat_ceilometer -i file:///home/jenkins/myiso.iso
+EOF
+}
+
+#
+# END of usage description
+############################################################################
+
+############################################################################
+# BEGIN of deployment clean-up
+#
+clean() {
+    echo "Cleaning up deploy tmp directories"
+    rm -rf ${SCRIPT_PATH}/config
+    rm -rf ${SCRIPT_PATH}/ISO
+    rm -rf ${SCRIPT_PATH}/releng
+}
+#
+# END of deployment clean-up
+############################################################################
+
+############################################################################
+# BEGIN of shorthand variables for internal use
+#
+SCRIPT_PATH=$(readlink -f $(dirname ${BASH_SOURCE[0]}))
+DEPLOY_DIR=$(cd ${SCRIPT_PATH}/../deploy; pwd)
+RELENG_REPO=https://jonasbjurel@gerrit.opnfv.org/gerrit/releng
+#
+# END of variables to customize
+############################################################################
+
+############################################################################
+# BEGIN of main
+#
+while getopts "b:l:p:s:i:h" OPTION
+do
+    case $OPTION in
+        b)
+            BASE_CONFIG_URI=${OPTARG}
+            ;;
+        l)
+            TARGET_LAB=${OPTARG}
+            ;;
+        p)
+            TARGET_POD=${OPTARG}
+            ;;
+        s)
+            DEPLOY_SCENARIO=${OPTARG}
+            ;;
+        i)
+            ISO=${OPTARG}
+            ;;
+        h)
+            usage
+            exit 0
+            ;;
+        *)
+            echo "${OPTION} is not a valid argument"
+            echo "Arguments not according to new argument style"
+            echo "Trying old-style compatibility mode"
+            pushd ${DEPLOY_DIR} > /dev/null
+            python deploy.py "$@"
+            popd > /dev/null
+            exit 0
+            ;;
+    esac
+done
+
+if [[ $EUID -ne 0 ]]; then
+   echo "This script must be run as root" 1>&2
+   exit 1
+fi
+
+if [ -z $BASE_CONFIG_URI ] || [ -z $TARGET_LAB ] || \
+   [ -z $TARGET_POD ] || [ -z $DEPLOY_SCENARIO ] || \
+   [ -z $ISO ]; then
+    echo "Arguments not according to new argument style"
+    echo "Trying old-style compatibility mode"
+    pushd ${DEPLOY_DIR} > /dev/null
+    python deploy.py "$@"
+    popd > /dev/null
+    exit 0
+fi
+
+# Enable the automatic exit trap
 set -o errexit
-topdir=$(dirname $(readlink -f $BASH_SOURCE))
-deploydir=$(cd ${topdir}/../deploy; pwd)
-pushd ${deploydir} > /dev/null
-echo -e "python deploy.py $@\n"
-python deploy.py $@
-popd > /dev/null
\ No newline at end of file
+trap do_exit SIGINT SIGTERM EXIT
+
+clean
+
+pushd ${DEPLOY_DIR} > /dev/null
+# Prepare the deploy config files based on lab/pod information, deployment
+# scenario, etc.
+
+echo "python deploy-config.py -dha ${BASE_CONFIG_URI}/labs/${TARGET_LAB}/${TARGET_POD}/fuel/config/dha.yaml -deab ${DEPLOY_DIR}/config/dea_base.yaml -deao ${BASE_CONFIG_URI}/labs/${TARGET_LAB}/${TARGET_POD}/fuel/config/dea-pod-override.yaml -scenario-base-uri ${DEPLOY_DIR}/scenario -scenario ${DEPLOY_SCENARIO} -plugins ${DEPLOY_DIR}/config/plugins -output ${SCRIPT_PATH}/config"
+
+python deploy-config.py -dha ${BASE_CONFIG_URI}/labs/${TARGET_LAB}/${TARGET_POD}/fuel/config/dha.yaml -deab file://${DEPLOY_DIR}/config/dea_base.yaml -deao ${BASE_CONFIG_URI}/labs/${TARGET_LAB}/${TARGET_POD}/fuel/config/dea-pod-override.yaml -scenario-base-uri file://${DEPLOY_DIR}/scenario -scenario ${DEPLOY_SCENARIO} -plugins file://${DEPLOY_DIR}/config/plugins -output ${SCRIPT_PATH}/config
+
+# Download iso if it doesn't already exists locally
+if [[ $ISO == file://* ]]; then
+    ISO=${ISO#file://}
+else
+    mkdir -p ${SCRIPT_PATH}/ISO
+    curl -o ${SCRIPT_PATH}/ISO/image.iso $ISO
+    ISO=${SCRIPT_PATH}/ISO/image.iso
+fi
+# Start deployment
+echo "python deploy.py -dea ${SCRIPT_PATH}/config/dea.yaml -dha ${SCRIPT_PATH}/config/dha.yaml -iso $ISO"
+python deploy.py -dea ${SCRIPT_PATH}/config/dea.yaml -dha ${SCRIPT_PATH}/config/dha.yaml -iso $ISO
+popd > /dev/null
+
+# TBD: Upload the test-section of the scenario yaml file to the fuel master:
+# var/www/test.yaml
+
+#
+# END of main
+############################################################################