xci: Update main script and readme

This change
- updates the main script to run the playbooks
- creates the initial version of the readme file

Change-Id: I4110433bce541a5d9645f34457f9e1b1ee93df19
Signed-off-by: Fatih Degirmenci <fatih.degirmenci@ericsson.com>

2 files changed, 373 insertions, 7 deletions

diff --git a/prototypes/xci/README.rst b/prototypes/xci/README.rst
new file mode 100644
index 000000000..8318cdb52
--- /dev/null
+++ b/prototypes/xci/README.rst
@@ -0,0 +1,217 @@
+###########################
+OPNFV XCI Developer Sandbox
+###########################
+
+The XCI Developer Sandbox is created by the OPNFV community for the OPNFV
+community in order to
+
+- provide means for OPNFV developers to work with OpenStack master branch,
+  cutting the time it takes to develop new features significantly and testing
+  them on OPNFV Infrastructure
+- enable OPNFV developers to identify bugs earlier, issue fixes faster, and
+  get feedback on a daily basis
+- establish mechanisms to run additional testing on OPNFV Infrastructure to
+  provide feedback to OpenStack community
+- make the solutions we put in place available to other LF Networking Projects
+  OPNFV works with closely
+
+More information about OPNFV XCI and the sandbox can be seen on
+`OPNFV Wiki <https://wiki.opnfv.org/pages/viewpage.action?pageId=8687635>`_.
+
+===================================
+Components of XCI Developer Sandbox
+===================================
+
+The sandbox uses OpenStack projects for VM node creation, provisioning
+and OpenStack installation.
+
+- **openstack/bifrost:** Bifrost (pronounced bye-frost) is a set of Ansible
+  playbooks that automates the task of deploying a base image onto a set
+  of known hardware using ironic. It provides modular utility for one-off
+  operating system deployment with as few operational requirements as
+  reasonably possible. Bifrost supports different operating systems such as
+  Ubuntu, CentOS, and openSUSE.
+  More information about this project can be seen on
+  `Bifrost documentation <https://docs.openstack.org/developer/bifrost/>`_.
+
+- **openstack/openstack-ansible:** OpenStack-Ansible is an official OpenStack
+  project which aims to deploy production environments from source in a way
+  that makes it scalable while also being simple to operate, upgrade, and grow.
+  More information about this project can be seen on
+  `OpenStack Ansible documentation <https://docs.openstack.org/developer/openstack-ansible/>`_.
+
+- **opnfv/releng:** OPNFV Releng Project provides additional scripts, Ansible
+  playbooks and configuration options in order for developers to have easy
+  way of using openstack/bifrost and openstack/openstack-ansible by just
+  setting couple of environment variables and executing a single script.
+  More infromation about this project can be seen on
+  `OPNFV Releng documentation <https://wiki.opnfv.org/display/releng>_`.
+
+==========
+Basic Flow
+==========
+
+Here are the steps that take place upon the execution of the sandbox script
+``xci-deploy.sh``:
+
+1. Sources environment variables in order to set things up properly.
+2. Installs ansible on the host where sandbox script is executed.
+3. Creates and provisions VM nodes based on the flavor chosen by the user.
+4. Configures the host where the sandbox script is executed.
+5. Configures the deployment host which the OpenStack installation will
+   be driven from.
+6. Configures the target hosts where OpenStack will be installed.
+7. Configures the target hosts as controller(s) and compute(s) nodes.
+8. Starts the OpenStack installation.
+
+=====================
+Sandbox Prerequisites
+=====================
+
+In order to use this sandbox, the host must have certain packages installed.
+
+- libvirt
+- python
+- pip
+- git
+- <fix the list with all the dependencies>
+- passwordless sudo
+
+The host must also have enough CPU/RAM/Disk in order to host number of VM
+nodes that will be created based on the chosen flavor. See the details from
+`this link <https://wiki.opnfv.org/display/INF/XCI+Developer+Sandbox#XCIDeveloperSandbox-Prerequisites>`_.
+
+===========================
+Flavors Provided by Sandbox
+===========================
+
+OPNFV XCI Sandbox provides different flavors such as all in one (aio) which
+puts much lower requirements on the host machine and full-blown HA.
+
+* aio: Single node which acts as the deployment host, controller and compute.
+* mini: One deployment host, 1 controller node and 1 compute node.
+* noha: One deployment host, 1 controller node and 2 compute nodes.
+* ha: One deployment host, 3 controller nodes and 2 compute nodes.
+
+See the details of the flavors from
+`this link <https://wiki.opnfv.org/display/INF/XCI+Developer+Sandbox#XCIDeveloperSandbox-AvailableFlavors>`_.
+
+==========
+How to Use
+==========
+
+Basic Usage
+-----------
+
+clone OPNFV Releng repository
+
+    git clone https://gerrit.opnfv.org/gerrit/releng.git
+
+change into directory where the sandbox script is located
+
+    cd releng/prototypes/xci
+
+execute sandbox script
+
+    sudo -E ./xci-deploy.sh
+
+Issuing above command will start aio sandbox deployment and the sandbox
+should be ready between 1,5 and 2 hours depending on the host machine.
+
+Advanced Usage
+--------------
+
+The flavor to deploy, the versions of upstream components to use can
+be configured by developers by setting certain environment variables.
+Below example deploys noha flavor using the latest of openstack-ansible
+master branch and stores logs in different location than what is configured.
+
+clone OPNFV Releng repository
+
+    git clone https://gerrit.opnfv.org/gerrit/releng.git
+
+change into directory where the sandbox script is located
+
+    cd releng/prototypes/xci
+
+set the sandbox flavor
+
+    export XCI_FLAVOR=noha
+
+set the version to use for openstack-ansible
+
+    export OPENSTACK_OSA_VERSION=master
+
+set where the logs should be stored
+
+    export LOG_PATH=/home/jenkins/xcilogs
+
+execute sandbox script
+
+    sudo -E ./xci-deploy.sh
+
+Warning::
+
+    Please encure you always execute the sandbox script using **sudo -E**
+    in order to make the environment variables you set available to the
+    sandbox script or you end up with the default settings.
+
+===============
+User Variables
+===============
+
+All user variables can be set from command line by exporting them before
+executing the script. The current user variables can be seen from
+``releng/prototypes/xci/config/user-vars``.
+
+The variables can also be set directly within the file before executing
+the sandbox script.
+
+===============
+Pinned Versions
+===============
+
+As explained above, the users can pick and choose which versions to use. If
+you want to be on the safe side, you can use the pinned versions the sandbox
+provides. They can be seen from ``releng/prototypes/xci/config/pinned-versions``.
+
+How Pinned Versions are Determined
+----------------------------------
+
+OPNFV runs periodic jobs against upstream projects openstack/bifrost and
+openstack/ansible using latest on master and stable/ocata branches,
+continuously chasing the HEAD of corresponding branches.
+
+Once a working version is identified, the versions of the upstream components
+are then bumped in releng repo.
+
+===========================================
+Limitations, Known Issues, and Improvements
+===========================================
+
+The list can be seen using `this link <https://jira.opnfv.org/issues/?filter=11616>`_.
+
+=========
+Changelog
+=========
+
+Changelog can be seen using `this link <https://jira.opnfv.org/issues/?filter=11625>`_.
+
+=======
+Testing
+=======
+
+Sandbox is continuously tested by OPNFV CI to ensure the changes do not impact
+users. In fact, OPNFV CI itself uses the sandbox scripts to run daily platform
+verification jobs.
+
+=======
+Support
+=======
+
+OPNFV XCI issues are tracked on OPNFV JIRA Releng project. If you encounter
+and issue or identify a bug, please submit an issue to JIRA using
+`this link <https://jira.opnfv.org/projects/RELENG>_`.
+
+If you have questions or comments, you can ask them on ``#opnfv-pharos`` IRC
+channel on Freenode.
diff --git a/prototypes/xci/xci-deploy.sh b/prototypes/xci/xci-deploy.sh
index c5cfab36d..7eb3ecdff 100755
--- a/prototypes/xci/xci-deploy.sh
+++ b/prototypes/xci/xci-deploy.sh
@@ -2,30 +2,42 @@
 set -o errexit
 set -o nounset
 set -o pipefail
-set -o xtrace
 
+#-------------------------------------------------------------------------------
 # This script must run as root
+#-------------------------------------------------------------------------------
 if [[ $(whoami) != "root" ]]; then
     echo "Error: This script must be run as root!"
     exit 1
 fi
 
+#-------------------------------------------------------------------------------
+# Set environment variables
+#-------------------------------------------------------------------------------
+# The order of sourcing the variable files is significant so please do not
+# change it or things might stop working.
+# - user-vars: variables that can be configured or overriden by user.
+# - pinned-versions: versions to checkout. These can be overriden if you want to
+#   use different/more recent versions of the tools but you might end up using
+#   something that is not verified by OPNFV XCI.
+# - flavor-vars: settings for VM nodes for the chosen flavor.
+# - env-vars: variables for the xci itself and you should not need to change or
+#   override any of them.
+#-------------------------------------------------------------------------------
 # find where are we
 XCI_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
-
 # source user vars
 source $XCI_PATH/config/user-vars
-
 # source pinned versions
 source $XCI_PATH/config/pinned-versions
-
 # source flavor configuration
 source "$XCI_PATH/flavors/${XCI_FLAVOR}-vars"
-
 # source xci configuration
 source $XCI_PATH/config/env-vars
 
-# log info to console
+#-------------------------------------------------------------------------------
+# Log info to console
+#-------------------------------------------------------------------------------
 echo "Info: Starting XCI Deployment"
 echo "Info: Deployment parameters"
 echo "-------------------------------------------------------------------------"
@@ -36,16 +48,153 @@ echo "openstack/openstack-ansible version: $OPENSTACK_OSA_VERSION"
 echo "-------------------------------------------------------------------------"
 
 #-------------------------------------------------------------------------------
-# install ansible on localhost
+# Install ansible on localhost
 #-------------------------------------------------------------------------------
 pip install ansible==$XCI_ANSIBLE_PIP_VERSION
 
+# TODO: The xci playbooks can be put into a playbook which will be done later.
+
 #-------------------------------------------------------------------------------
 # Start provisioning VM nodes
 #-------------------------------------------------------------------------------
+# This playbook
+# - removes directories that were created by the previous xci run
+# - clones opnfv/releng and openstack/bifrost repositories
+# - combines opnfv/releng and openstack/bifrost scripts/playbooks
+# - destorys VMs, removes ironic db, leases, logs
+# - creates and provisions VMs for the chosen flavor
+#-------------------------------------------------------------------------------
 echo "Info: Starting provisining VM nodes using openstack/bifrost"
 echo "-------------------------------------------------------------------------"
 cd $XCI_PATH/playbooks
 ansible-playbook $ANSIBLE_VERBOSITY -i inventory provision-vm-nodes.yml
 echo "-----------------------------------------------------------------------"
 echo "Info: VM nodes are provisioned!"
+
+#-------------------------------------------------------------------------------
+# Configure localhost
+#-------------------------------------------------------------------------------
+# This playbook
+# - removes directories that were created by the previous xci run
+# - clones opnfv/releng repository
+# - creates log directory
+# - copies flavor files such as playbook, inventory, and var file
+#-------------------------------------------------------------------------------
+echo "Info: Configuring localhost for openstack-ansible"
+echo "-----------------------------------------------------------------------"
+cd $XCI_PATH/playbooks
+ansible-playbook $ANSIBLE_VERBOSITY -i inventory configure-localhost.yml
+echo "-----------------------------------------------------------------------"
+echo "Info: Configured localhost host for openstack-ansible"
+
+#-------------------------------------------------------------------------------
+# Configure openstack-ansible deployment host, opnfv
+#-------------------------------------------------------------------------------
+# This playbook
+# - removes directories that were created by the previous xci run
+# - clones opnfv/releng and openstack/openstack-ansible repositories
+# - configures network
+# - generates/prepares ssh keys
+# - bootstraps ansible
+# - copies flavor files to be used by openstack-ansible
+#-------------------------------------------------------------------------------
+echo "Info: Configuring opnfv deployment host for openstack-ansible"
+echo "-----------------------------------------------------------------------"
+cd $OPNFV_RELENG_PATH/prototypes/xci/playbooks
+ansible-playbook $ANSIBLE_VERBOSITY -i inventory configure-opnfvhost.yml
+echo "-----------------------------------------------------------------------"
+echo "Info: Configured opnfv deployment host for openstack-ansible"
+
+#-------------------------------------------------------------------------------
+# Skip the rest if the flavor is aio since the target host for aio is opnfv
+#-------------------------------------------------------------------------------
+if [[ $XCI_FLAVOR == "aio" ]]; then
+    echo "xci: aio has been installed"
+    exit 0
+fi
+
+#-------------------------------------------------------------------------------
+# Configure target hosts for openstack-ansible
+#-------------------------------------------------------------------------------
+# This playbook
+# - adds public keys to target hosts
+# - configures network
+# - configures nfs
+#-------------------------------------------------------------------------------
+echo "Info: Configuring target hosts for openstack-ansible"
+echo "-----------------------------------------------------------------------"
+cd $OPNFV_RELENG_PATH/prototypes/xci/playbooks
+ansible-playbook $ANSIBLE_VERBOSITY -i inventory configure-targethosts.yml
+echo "-----------------------------------------------------------------------"
+echo "Info: Configured target hosts"
+
+#-------------------------------------------------------------------------------
+# Set up target hosts for openstack-ansible
+#-------------------------------------------------------------------------------
+# This is openstack-ansible playbook. Check upstream documentation for details.
+#-------------------------------------------------------------------------------
+echo "Info: Setting up target hosts for openstack-ansible"
+echo "-----------------------------------------------------------------------"
+sudo -E /bin/sh -c "ssh root@$OPNFV_HOST_IP openstack-ansible \
+     $OPENSTACK_OSA_PATH/playbooks/setup-hosts.yml" | \
+     tee $LOG_PATH/setup-hosts.log
+echo "-----------------------------------------------------------------------"
+# check the log to see if we have any error
+if grep -q 'failed=1\|unreachable=1' $LOG_PATH/setup-hosts.log; then
+    echo "Error: OpenStack node setup failed!"
+    exit 1
+fi
+echo "Info: Set up target hosts for openstack-ansible successfuly"
+
+#-------------------------------------------------------------------------------
+# Set up infrastructure
+#-------------------------------------------------------------------------------
+# This is openstack-ansible playbook. Check upstream documentation for details.
+#-------------------------------------------------------------------------------
+echo "Info: Setting up infrastructure"
+echo "-----------------------------------------------------------------------"
+echo "xci: running ansible playbook setup-infrastructure.yml"
+sudo -E /bin/sh -c "ssh root@$OPNFV_HOST_IP openstack-ansible \
+     $OPENSTACK_OSA_PATH/playbooks//setup-infrastructure.yml" | \
+     tee $LOG_PATH/setup-infrastructure.log
+echo "-----------------------------------------------------------------------"
+# check the log to see if we have any error
+if grep -q 'failed=1\|unreachable=1' $LOG_PATH/setup-infrastructure.log; then
+    echo "Error: OpenStack node setup failed!"
+    exit 1
+fi
+
+#-------------------------------------------------------------------------------
+# Verify database cluster
+#-------------------------------------------------------------------------------
+echo "Info: Verifying database cluster"
+echo "-----------------------------------------------------------------------"
+sudo -E /bin/sh -c "ssh root@$OPNFV_HOST_IP ansible -i $OPENSTACK_OSA_PATH/playbooks/inventory/ \
+           galera_container -m shell \
+           -a "mysql -h localhost -e 'show status like \"%wsrep_cluster_%\";'"" \
+           | tee $LOG_PATH/galera.log
+echo "-----------------------------------------------------------------------"
+# check the log to see if we have any error
+if grep -q 'FAILED' $LOG_PATH/galera.log; then
+    echo "Error: Database cluster verification failed!"
+    exit 1
+fi
+echo "Info: Database cluster verification successful!"
+
+#-------------------------------------------------------------------------------
+# Install OpenStack
+#-------------------------------------------------------------------------------
+# This is openstack-ansible playbook. Check upstream documentation for details.
+#-------------------------------------------------------------------------------
+echo "Info: Installing OpenStack on target hosts"
+echo "-----------------------------------------------------------------------"
+sudo -E /bin/sh -c "ssh root@$OPNFV_HOST_IP openstack-ansible \
+     $OPENSTACK_OSA_PATH/playbooks/setup-openstack.yml" | \
+     tee $LOG_PATH/opnfv-setup-openstack.log
+echo "-----------------------------------------------------------------------"
+# check the log to see if we have any error
+if grep -q 'failed=1\|unreachable=1' $LOG_PATH/opnfv-setup-openstack.log; then
+   echo "Error: OpenStack installation failed!"
+   exit 1
+fi
+echo "Info: OpenStack installation is successfully completed!"