Merge "Adds various conveniences for hosters."stable/euphrates

4 files changed, 273 insertions, 0 deletions

diff --git a/laas-fog/LaaS_Diagram.jpg b/laas-fog/LaaS_Diagram.jpg
new file mode 100644
index 0000000..521236d
--- /dev/null
+++ b/laas-fog/LaaS_Diagram.jpg
diff --git a/laas-fog/README b/laas-fog/README
new file mode 100644
index 0000000..84317eb
--- /dev/null
+++ b/laas-fog/README
@@ -0,0 +1,167 @@
+This Lab as a Serice project aims to create on demand OPNFV resources to developers.
+This project will automate the process, to the requested extent, of running an OPNFV
+installer and creating an Openstack environment within OPNFV automatically and on demand.
+
+To run, execute (from the project root):
+    source/deploy.py
+
+To run the Pharos dahsboard listener, which will continualy poll the dashboard and run deployments in the background:
+    source/listen.py --config <conf/pharos.conf>
+
+
+For convenience, there is a bash script source/stop.sh which will stop the dashboard listener and all related scripts.
+
+BEFORE YOU CAN RUN:
+you must first:
+- Integrate FOG into your infrastructure
+- Fill out the needed configuration files
+- Populate the database with your available hosts
+
+
+FOG:
+Our OPNFV infrastructure uses a FOG server to pxe boot, read and write disk images, and otherwise control the hosts we have available for developers.
+FOG is an open source project, and you can view it here: https://fogproject.org/
+FOG provides an easy and scriptable way to completely wipe and write the disks of our hosts.
+    This makes it quick and simple for us to restore our hosts to a known, clean state after a developer has released control of it.
+
+To run the deploy script, you need to:
+    Have a FOG master running
+    Have your hosts registered to the FOG master
+    Have a 'clean' disk image of for each installer / configuration you wish to support.
+        - Fuel, Compass, and JOID all need different distros / versions to run properly
+        - There is a mapping between images and their installers in the installer's config file
+The FOG server must be reachable by whatever machine is running this LaaS software,
+and have network access to PXE boot all of your hosted dev pods.
+
+
+CONFIGURATION:
+INSTALLERS#############################################################################################
+-database               Path to the SQLite database for storing host information.
+                            Should be the same for all installers in most cases.
+-dhcp_log               Path to log file containing DHCP information for dev pods.
+-dhcp_server            IP address or hostname of the DHCP server which contains the above log file
+                            set to `null` if the same machine will be running dhcp and this project
+-fog
+--api_key               The FOG api key. You may instead give the path to a file containing the api key.
+--server                The URL of the fog server.
+                            ex: http://myServer.com/fog/
+--user_key              The FOG api key specific to your user.
+                            You may instead give the path to a secrets file containing the key.
+--image_id              The id of the image FOG will use when this installer is requested.
+-installer              The name of the installer, as seen from the dashboard.
+                            `null` will match when no installer is selected, or the `None` installer is..
+-logging_dir            The directory to create log files in.
+                            Will create the dir if it does not already exist.
+-scenario               The default scenario if one is not specified by the user.
+                            NOTE:   automation of different scenarios are not currently supported.
+                                    These values are silently ignored.
+-hypervisor_config
+--networks              Path to the config file used to define the virtual networks for this installer.
+--vms                   Path to the config file used to define the virtual machines for this installer.
+-inventory              Path to inventory file mapping dashboard host id's to FOG hostnames.
+-vpn_config             Path to the vpn config file
+
+
+#########################################################################################################
+
+DOMAINS##################################################################################################
+-jinja-template         Path to the jinja xml template used to create libvirt domain xml documents.
+-domains                A list of domains. List as many as you want, but be cognizant of hardware limitations
+--disk                  Path to the qcow2 disk image for this VM
+--interfaces            List of interfaces for the vm
+---name                 The name of the network or bridge that provides this interface
+---type                 The source of the interface. Either 'bridge' or 'network' is valid, but the bridge
+                            must already exist on the host.
+--iso
+---URL                  Where to fetch the ISO from
+---location             Where to save the ISO to
+---used                 Whether this host will use an iso as a boot drive
+                            if `false`, the ISO will not be downloaded
+--memory                Memory to allocate to the VM in KiB
+--name                  libvirt name of VM
+--vcpus                 How many vcpus to allocate to this host.
+#########################################################################################################
+
+NETWORKS#################################################################################################
+-jinja-template         Path to jinja template used to create libvirt XML network documents
+-networks               List of networks that will be created
+--brAddr                ip address of the bridge on the host
+--brName                name of the bridge on the host
+--cidr                  cidr of the virtual network
+--dhcp                  dhcp settingg
+---rangeEnd             end of DHCP address range
+---rangeStart           start of DHCP address range
+---used                 Whether to enable dhcp for this network. Should probably be false.
+--forward               Libvirt network forwarding settings
+---type                 forwarding type. See libvirt documentation for possible types.
+---used                 if `false`, the network is isolated.
+--name                  Name of this network in Libvirt
+--netmask               Netmask for this network.
+########################################################################################################
+
+PHAROS##################################################################################################
+-dashboard              url of the dashboard. https://labs.opnfv.org is the public OPNFV dashboard
+-database               path to database to store booking information.
+                            Should be the same db as the host database in most cases
+-default_configs        a mappping of installers and their configuration files.
+-inventory              path to the inventory file
+-logging_dir            Where the pharos dashboard listener should put log files.
+-poling                 How many times a second the listener will poll the dashboard
+-token                  Your paros api token. May also be a path to a file containing the token
+#######################################################################################################
+
+VPN####################################################################################################
+NOTE: this all assumes you use LDAP authentication
+-server                 Domain name of your vpn server
+-authenticaion
+--pass                  password for your 'admin' user. May also be a path to a secrets file
+--user                  full dn of your 'admin' user
+-directory
+--root                  The lowest directory that this program will need to access
+--user                  The directory where users are stored, relative to the given root dir
+-user
+--objects               A list of object classes that vpn users will belong to.
+                            Most general class should be on top, and get more specific from there.
+                            ex: -top, -inetOrgPerson because `top` is more general
+-database               The booking database
+-permanent_users        Users that you want to be persistent, even if they have no bookings active
+                            ie: your admin users
+                            All other users will be deleted when they have no mroe bookings
+#######################################################################################################
+
+INVENTORY##############################################################################################
+This file is used to map the resource id's known by pharos to the hostnames known by FOG.
+for example,
+50: fog-machine-4
+51: fog-machine-5
+52: fog-virtualPod-5.1
+#######################################################################################################
+    
+HOW IT WORKS:
+
+0) lab resources are prepared and information is stored in the database
+1) source/listen.py launches a background instance of pharos.py
+    -pharos.py continually polls the dashboard for booking info, and stores it in the database
+2) A known booking begins and pharos.py launches pod_manager.py
+    - pod_manager is launched in a new process, so that the listener continues to poll the dashboard
+      and multiple hosts can be provisioned at once
+3) pod_manager uses FOG to image the host
+4) if requested, pod_manager hands control to deployment_manager to install and deploy OPNFV
+    - deployment_manager instantiates and calls the go() function of the given source/installers/installer subclass
+5) a vpn user is created and random root password is given to the dev pod
+##########The dashboard does not yet support the following actions#############
+6) public ssh key of the user is fetched from the dashboard
+7) user is automatically notified their pod is ready, and given all needed info
+
+
+GENERAL NOTES:
+
+resetDatabase.py relies on FOG to retrieve a list of all hosts available to developers
+
+running:
+    source/resetDatabase.py --both --config <CONFIG_FILE>
+will create a database and populate it.
+WARNING: This will delete existing information if run on a previously initialized database
+
+To aid in visualization and understanding of the resulting topolgy after fully deploying OPNFV and Openstack in
+a development pod, you may review the LaaS_Diagram in this directory.
diff --git a/laas-fog/source/deploy.py b/laas-fog/source/deploy.py
new file mode 100755
index 0000000..a9c5e04
--- /dev/null
+++ b/laas-fog/source/deploy.py
@@ -0,0 +1,82 @@
+#!/usr/bin/python
+"""
+#############################################################################
+#Copyright 2017 Parker Berberian and others                                 #
+#                                                                           #
+#Licensed under the Apache License, Version 2.0 (the "License");            #
+#you may not use this file except in compliance with the License.           #
+#You may obtain a copy of the License at                                    #
+#                                                                           #
+#    http://www.apache.org/licenses/LICENSE-2.0                             #
+#                                                                           #
+#Unless required by applicable law or agreed to in writing, software        #
+#distributed under the License is distributed on an "AS IS" BASIS,          #
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.   #
+#See the License for the specific language governing permissions and        #
+#limitations under the License.                                             #
+#############################################################################
+"""
+
+import sys
+import yaml
+from pod_manager import Pod_Manager
+
+"""
+This file is the first executed when a booking begins.
+"""
+
+usage = """
+./deploy [--config CONFIG_FILE] [--host HOSTNAME] [--reset]
+"""
+
+
+def main(config_path, host):
+    """
+    starts the deployment with the given configuration.
+    """
+    config = yaml.safe_load(open(config_path))
+
+    manager = Pod_Manager(config, requested_host=host)
+    manager.start_deploy()
+
+
+def reset(config_path, host):
+    """
+    Tells the Pod Manager to clean and reset the given host.
+    """
+    config = yaml.safe_load(open(config_path))
+    Pod_Manager(config, requested_host=host, reset=True)
+
+
+if __name__ == "__main__":
+    # parse command line
+    host = None
+
+    if "--help" in sys.argv:
+        print usage
+        sys.exit(0)
+
+    if "--config" in sys.argv:
+        try:
+            conf = sys.argv[1+sys.argv.index("--config")]
+            open(conf)
+        except Exception:
+            print "bad config file"
+            sys.exit(1)
+    if "--host" in sys.argv:
+        try:
+            host = sys.argv[1+sys.argv.index("--host")]
+        except:
+            "host not provided. Exiting"
+            sys.exit(1)
+
+    try:
+        config_file = yaml.safe_load(open(conf))
+    except:
+        print "Failed to read from config file"
+        sys.exit(1)
+    # reset or deploy host
+    if "--reset" in sys.argv:
+        reset(conf, host)
+    else:
+        main(conf, host)
diff --git a/laas-fog/source/stop.sh b/laas-fog/source/stop.sh
new file mode 100755
index 0000000..e721482
--- /dev/null
+++ b/laas-fog/source/stop.sh
@@ -0,0 +1,24 @@
+#!/bin/bash
+#############################################################################
+#Copyright 2017 Parker Berberian and others                                 #
+#                                                                           #
+#Licensed under the Apache License, Version 2.0 (the "License");            #
+#you may not use this file except in compliance with the License.           #
+#You may obtain a copy of the License at                                    #
+#                                                                           #
+#    http://www.apache.org/licenses/LICENSE-2.0                             #
+#                                                                           #
+#Unless required by applicable law or agreed to in writing, software        #
+#distributed under the License is distributed on an "AS IS" BASIS,          #
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.   #
+#See the License for the specific language governing permissions and        #
+#limitations under the License.                                             #
+#############################################################################
+
+
+# This just finds all processes from this program and kills them.
+
+
+PIDS=$(ps -ef | grep laas/source/ | grep python | awk '{print $2}')
+
+kill ${PIDS[*]}