A simple method to separate configuration for base fuel, plugins, PODs

and deployment/test scenarios

READY TO MERGE!

Replaces: https://gerrit.opnfv.org/gerrit/#/c/3995/

Abstract
--------
This deployment framework relies on a configuration structure,
providing base installer configuration, per POD specific configuration,
plugin configuration, and deployment scenario configuration.
- The base installer configuration resembles the least common denominator
  of all HW/POD environment and deployment scenarios (These configurations
  are normally carried by the the installer projects in this case (fuel@OPNFV).
- Per POD specific configuration specifies POD unique parameters, the POD
  parameter possible to alter is governed by the Fuel@OPNFV project.
- Plugin configuration - providing configuration of a specific plugin.
  these configurations maintain there own namespace and are normally maintained
  by collaborative projects building Fuel@OPNFV plugins
- Deployment scenario configuration - provides a high level, POD/HW environment
  independent scenario configuration for a specific deployment. It defines what
  features/plugins that shall be deployed - as well needed overrides of the
  plugin config as well as the base installer-, POD/HW environment-
  configurations. Objects allowed to override
  is governed by the Fuel@OPNFV project.

Executing a deployment
----------------------
deploy.sh must be executed locally at the target lab/pod/jumpserver
A lab configuration structure must be provided - see the section below.
It is straight forward to execute a deployment task - as an example:
sudo deploy.sh -b file:///home/jenkins/config -l ericsson-1 -p pod-2
-s os_odl-l2_no-ha -i file:///home/jenkins/MyIso.iso

-b and -i arguments should be expressed in URI style. The resources can thus be
local or remote.

Feedback
--------
Please give feed-back before I'm going to far on a wrong tangent

Implemented scenarios so far:
-----------------------------
- os_ha
- os_no-ha
- os_odl-l3_ha
- os_odl-l3_no-ha
- os_odl-l2_ha
- os_odl-l2_no-ha
- os_onos_ha
- os_onos_no-ha
- os_kvm_ha
- os_kvm_no-ha
- os_ovs_ha
- os_ovs_no-ha
- os_kvm_ovs_ha
- os_kvm_ovs_no-ha

VERIFIED
READY TO MERGE

JIRA: FUEL-35

Change-Id: I94a9b477d8ed4ee8057c16d8f20fe543f7ecc20d
Signed-off-by: Jonas Bjurel <jonas.bjurel@ericsson.com>
(cherry picked from commit 7fd2619e0df370fa832eeff00790bcaa52dc4ffc)

1 files changed, 597 insertions, 0 deletions

diff --git a/deploy/config/dea_base.yaml b/deploy/config/dea_base.yaml
new file mode 100644
index 000000000..d3188a1af
--- /dev/null
+++ b/deploy/config/dea_base.yaml
@@ -0,0 +1,597 @@
+dea-base-config-metadata:
+  title: 'Deployment Environment Adapter Base configuration'
+  # DEA API version supported
+  version: '0.1'
+  created: 'Mon Dec 28 2015'
+  comment: 'None'
+environment:
+  mode: ha
+  net_segment_type: tun
+wanted_release: Kilo on Ubuntu 14.04
+fuel:
+  FUEL_ACCESS:
+    password: admin
+    user: admin
+  HOSTNAME: fuel
+settings:
+  editable:
+    access:
+      email:
+        description: Email address for Administrator
+        label: Email
+        regex:
+          error: Invalid email
+          source: ^\S+@\S+$
+        type: text
+        value: admin@localhost
+        weight: 40
+      metadata:
+        label: Access
+        weight: 10
+      password:
+        description: Password for Administrator
+        label: Password
+        regex:
+          error: Empty password
+          source: \S
+        type: password
+        value: admin
+        weight: 20
+      tenant:
+        description: Tenant (project) name for Administrator
+        label: Tenant
+        regex:
+          error: Invalid tenant name
+          source: ^(?!services$)(?!nova$)(?!glance$)(?!keystone$)(?!neutron$)(?!cinder$)(?!swift$)(?!ceph$)(?![Gg]uest$)(?!.*
+            +.*$).+
+        type: text
+        value: admin
+        weight: 30
+      user:
+        description: Username for Administrator
+        label: Username
+        regex:
+          error: Invalid username
+          source: ^(?!services$)(?!nova$)(?!glance$)(?!keystone$)(?!neutron$)(?!cinder$)(?!swift$)(?!ceph$)(?![Gg]uest$)(?!.*
+            +.*$).+
+        type: text
+        value: admin
+        weight: 10
+    additional_components:
+      ceilometer:
+        description: If selected, Ceilometer component will be installed
+        label: Install Ceilometer
+        type: checkbox
+        value: false
+        weight: 40
+      heat:
+        description: ''
+        label: ''
+        type: hidden
+        value: true
+        weight: 30
+      metadata:
+        label: Additional Components
+        weight: 20
+      mongo:
+        description: If selected, You can use external Mongo DB as ceilometer backend
+        label: Use external Mongo DB
+        restrictions:
+        - settings:additional_components.ceilometer.value == false
+        type: checkbox
+        value: false
+        weight: 40
+      murano:
+        description: If selected, Murano component will be installed
+        label: Install Murano
+        type: checkbox
+        value: false
+        weight: 20
+      sahara:
+        description: If selected, Sahara component will be installed
+        label: Install Sahara
+        type: checkbox
+        value: false
+        weight: 10
+    common:
+      auth_key:
+        description: Public key(s) to include in authorized_keys on deployed nodes
+        label: Public Key
+        type: textarea
+        value: ''
+        weight: 70
+      auto_assign_floating_ip:
+        description: If selected, OpenStack will automatically assign a floating IP
+          to a new instance
+        label: Auto assign floating IP
+        restrictions:
+        - action: hide
+          condition: cluster:net_provider == 'neutron'
+        type: checkbox
+        value: false
+        weight: 40
+      debug:
+        description: Debug logging mode provides more information, but requires more
+          disk space.
+        label: OpenStack debug logging
+        type: checkbox
+        value: false
+        weight: 20
+      libvirt_type:
+        label: Hypervisor type
+        type: radio
+        value: kvm
+        values:
+        - data: kvm
+          description: Choose this type of hypervisor if you run OpenStack on hardware
+          label: KVM
+        - data: qemu
+          description: Choose this type of hypervisor if you run OpenStack on virtual
+            hosts.
+          label: QEMU
+        weight: 30
+      metadata:
+        label: Common
+        weight: 30
+      nova_quota:
+        description: Quotas are used to limit CPU and memory usage for tenants. Enabling
+          quotas will increase load on the Nova database.
+        label: Nova quotas
+        type: checkbox
+        value: false
+        weight: 25
+      puppet_debug:
+        description: Debug puppet logging mode provides more information, but requires
+          more disk space.
+        label: Puppet debug logging
+        type: checkbox
+        value: true
+        weight: 20
+      resume_guests_state_on_host_boot:
+        description: Whether to resume previous guests state when the host reboots.
+          If enabled, this option causes guests assigned to the host to resume their
+          previous state. If the guest was running a restart will be attempted when
+          nova-compute starts. If the guest was not running previously, a restart
+          will not be attempted.
+        label: Resume guests state on host boot
+        type: checkbox
+        value: true
+        weight: 60
+      use_cow_images:
+        description: For most cases you will want qcow format. If it's disabled, raw
+          image format will be used to run VMs. OpenStack with raw format currently
+          does not support snapshotting.
+        label: Use qcow format for images
+        type: checkbox
+        value: true
+        weight: 50
+      use_vcenter:
+        restrictions:
+        - condition: cluster:net_provider == 'nova_network'
+        type: hidden
+        value: false
+        weight: 30
+    corosync:
+      group:
+        description: ''
+        label: Group
+        type: text
+        value: 226.94.1.1
+        weight: 10
+      metadata:
+        label: Corosync
+        restrictions:
+        - action: hide
+          condition: 'true'
+        weight: 50
+      port:
+        description: ''
+        label: Port
+        type: text
+        value: '12000'
+        weight: 20
+      verified:
+        description: Set True only if multicast is configured correctly on router.
+        label: Need to pass network verification.
+        type: checkbox
+        value: false
+        weight: 10
+    external_mongo:
+      hosts_ip:
+        description: IP Addresses of MongoDB. Use comma to split IPs
+        label: MongoDB hosts IP
+        regex:
+          error: Invalid hosts ip sequence
+          source: ^(((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?),)*((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$
+        type: text
+        value: ''
+        weight: 30
+      metadata:
+        label: External MongoDB
+        restrictions:
+        - action: hide
+          condition: settings:additional_components.mongo.value == false
+          message: Ceilometer and MongoDB are not enabled on the Additional Components
+            section
+        weight: 20
+      mongo_db_name:
+        description: Mongo database name
+        label: Database name
+        regex:
+          error: Invalid database name
+          source: ^\w+$
+        type: text
+        value: ceilometer
+        weight: 30
+      mongo_password:
+        description: Mongo database password
+        label: Password
+        regex:
+          error: Password contains spaces
+          source: ^\S*$
+        type: password
+        value: ceilometer
+        weight: 30
+      mongo_replset:
+        description: Name for Mongo replication set
+        label: Replset
+        type: text
+        value: ''
+        weight: 30
+      mongo_user:
+        description: Mongo database username
+        label: Username
+        regex:
+          error: Empty username
+          source: ^\w+$
+        type: text
+        value: ceilometer
+        weight: 30
+    kernel_params:
+      kernel:
+        description: Default kernel parameters
+        label: Initial parameters
+        type: text
+        value: console=tty0 net.ifnames=0 biosdevname=0 rootdelay=90 nomodeset
+        weight: 45
+      metadata:
+        label: Kernel parameters
+        weight: 40
+    murano_settings:
+      metadata:
+        label: Murano Settings
+        restrictions:
+        - action: hide
+          condition: settings:additional_components.murano.value == false
+          message: Murano is not enabled on the Additional Components section
+        weight: 20
+      murano_repo_url:
+        description: ''
+        label: Murano Repository URL
+        type: text
+        value: http://storage.apps.openstack.org/
+        weight: 10
+    neutron_advanced_configuration:
+      metadata:
+        label: Neutron Advanced Configuration
+        restrictions:
+        - action: hide
+          condition: cluster:net_provider != 'neutron'
+        weight: 45
+      neutron_dvr:
+        description: Enable Distributed Virtual Routers in Neutron
+        label: Neutron DVR
+        restrictions:
+        - ? networking_parameters:segmentation_type != 'vlan' and settings:neutron_advanced_configuration.neutron_l2_pop.value
+            == false
+          : DVR requires L2 population to be enabled.
+        type: checkbox
+        value: false
+        weight: 20
+      neutron_l2_pop:
+        description: Enable L2 population mechanism in Neutron
+        label: Neutron L2 population
+        restrictions:
+        - action: hide
+          condition: networking_parameters:segmentation_type == 'vlan'
+        type: checkbox
+        value: false
+        weight: 10
+    neutron_mellanox:
+      metadata:
+        enabled: true
+        label: Mellanox Neutron components
+        restrictions:
+        - action: hide
+          condition: not ('experimental' in version:feature_groups)
+        toggleable: false
+        weight: 50
+      plugin:
+        label: Mellanox drivers and SR-IOV plugin
+        type: radio
+        value: disabled
+        values:
+        - data: disabled
+          description: If selected, Mellanox drivers, Neutron and Cinder plugin will
+            not be installed.
+          label: Mellanox drivers and plugins disabled
+          restrictions:
+          - settings:storage.iser.value == true
+        - data: drivers_only
+          description: If selected, Mellanox Ethernet drivers will be installed to
+            support networking over Mellanox NIC. Mellanox Neutron plugin will not
+            be installed.
+          label: Install only Mellanox drivers
+          restrictions:
+          - settings:common.libvirt_type.value != 'kvm'
+        - data: ethernet
+          description: If selected, both Mellanox Ethernet drivers and Mellanox network
+            acceleration (Neutron) plugin will be installed.
+          label: Install Mellanox drivers and SR-IOV plugin
+          restrictions:
+          - settings:common.libvirt_type.value != 'kvm' or not (cluster:net_provider
+            == 'neutron' and networking_parameters:segmentation_type == 'vlan')
+        weight: 60
+      vf_num:
+        description: Note that one virtual function will be reserved to the storage
+          network, in case of choosing iSER.
+        label: Number of virtual NICs
+        restrictions:
+        - settings:neutron_mellanox.plugin.value != 'ethernet'
+        type: text
+        value: '16'
+        weight: 70
+    provision:
+      metadata:
+        label: Provision
+        restrictions:
+        - action: hide
+          condition: 'true'
+        weight: 80
+      method:
+        type: hidden
+        value: image
+    public_network_assignment:
+      assign_to_all_nodes:
+        description: When disabled, public network will be assigned to controllers
+          only
+        label: Assign public network to all nodes
+        type: checkbox
+        value: true
+        weight: 10
+      metadata:
+        label: Public network assignment
+        restrictions:
+        - action: hide
+          condition: cluster:net_provider != 'neutron'
+        weight: 50
+    public_ssl:
+      cert_data:
+        description: Certificate and private key data, concatenated into a single
+          file
+        label: Certificate
+        restrictions:
+        - action: hide
+          condition: (settings:public_ssl.cert_source.value != 'user_uploaded') or
+            (settings:public_ssl.horizon.value == false and settings:public_ssl.services.value
+            == false)
+        type: file
+        value: ''
+        weight: 40
+      cert_source:
+        description: From where we'll get certificate and private key
+        label: Select source for certificate
+        restrictions:
+        - action: hide
+          condition: settings:public_ssl.horizon.value == false and settings:public_ssl.services.value
+            == false
+        type: radio
+        value: self_signed
+        values:
+        - data: self_signed
+          description: Generate private key and certificate that will be signed by
+            this key
+          label: Self-signed
+        - data: user_uploaded
+          description: Use pre-generated key and certificate
+          label: I have my own keypair with certificate
+        weight: 30
+      horizon:
+        description: Secure access to Horizon enabling HTTPS instead of HTTP
+        label: HTTPS for Horizon
+        type: checkbox
+        value: false
+        weight: 10
+      hostname:
+        description: Your DNS entries should point to this name. Self-signed certificates
+          also will use this hostname
+        label: DNS hostname for public TLS endpoints
+        restrictions:
+        - action: hide
+          condition: settings:public_ssl.horizon.value == false and settings:public_ssl.services.value
+            == false
+        type: text
+        value: public.fuel.local
+        weight: 50
+      metadata:
+        label: Public TLS
+        weight: 110
+      services:
+        description: Enable TLS termination on HAProxy for OpenStack services
+        label: TLS for OpenStack public endpoints
+        type: checkbox
+        value: false
+        weight: 20
+    repo_setup:
+      metadata:
+        always_editable: true
+        label: Repositories
+        weight: 50
+      repos:
+        description: 'Please note: the first repository will be considered the operating
+          system mirror that will be used during node provisioning.
+
+          To create a local repository mirror on the Fuel master node, please follow
+          the instructions provided by running "fuel-createmirror --help" on the Fuel
+          master node.
+
+          Please make sure your Fuel master node has Internet access to the repository
+          before attempting to create a mirror.
+
+          For more details, please refer to the documentation (https://docs.mirantis.com/openstack/fuel/fuel-7.0/operations.html#external-ubuntu-ops).
+
+          '
+        extra_priority: null
+        type: custom_repo_configuration
+        value:
+        - name: ubuntu
+          priority: null
+          section: main
+          suite: trusty
+          type: deb
+          uri: http://10.20.0.2:8080/ubuntu-part
+        - name: ubuntu-updates
+          priority: null
+          section: main
+          suite: trusty
+          type: deb
+          uri: http://10.20.0.2:8080/ubuntu-part
+        - name: ubuntu-security
+          priority: null
+          section: main
+          suite: trusty
+          type: deb
+          uri: http://10.20.0.2:8080/ubuntu-part
+        - name: mos
+          priority: 1050
+          section: main restricted
+          suite: mos7.0
+          type: deb
+          uri: http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/x86_64
+        - name: mos-updates
+          priority: 1050
+          section: main restricted
+          suite: mos7.0-updates
+          type: deb
+          uri: http://10.20.0.2:8080/mos-ubuntu
+        - name: mos-security
+          priority: 1050
+          section: main restricted
+          suite: mos7.0-security
+          type: deb
+          uri: http://10.20.0.2:8080/mos-ubuntu
+        - name: mos-holdback
+          priority: 1100
+          section: main restricted
+          suite: mos7.0-holdback
+          type: deb
+          uri: http://10.20.0.2:8080/mos-ubuntu
+        - name: Auxiliary
+          priority: 1150
+          section: main restricted
+          suite: auxiliary
+          type: deb
+          uri: http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/auxiliary
+    storage:
+      ephemeral_ceph:
+        description: Configures Nova to store ephemeral volumes in RBD. This works
+          best if Ceph is enabled for volumes and images, too. Enables live migration
+          of all types of Ceph backed VMs (without this option, live migration will
+          only work with VMs launched from Cinder volumes).
+        label: Ceph RBD for ephemeral volumes (Nova)
+        type: checkbox
+        value: true
+        weight: 75
+      images_ceph:
+        description: Configures Glance to use the Ceph RBD backend to store images.
+          If enabled, this option will prevent Swift from installing.
+        label: Ceph RBD for images (Glance)
+        restrictions:
+        - settings:storage.images_vcenter.value == true: Only one Glance backend could
+            be selected.
+        type: checkbox
+        value: true
+        weight: 30
+      images_vcenter:
+        description: Configures Glance to use the vCenter/ESXi backend to store images.
+          If enabled, this option will prevent Swift from installing.
+        label: VMWare vCenter/ESXi datastore for images (Glance)
+        restrictions:
+        - action: hide
+          condition: settings:common.use_vcenter.value != true
+        - condition: settings:storage.images_ceph.value == true
+          message: Only one Glance backend could be selected.
+        type: checkbox
+        value: false
+        weight: 35
+      iser:
+        description: 'High performance block storage: Cinder volumes over iSER protocol
+          (iSCSI over RDMA). This feature requires SR-IOV capabilities in the NIC,
+          and will use a dedicated virtual function for the storage network.'
+        label: iSER protocol for volumes (Cinder)
+        restrictions:
+        - settings:storage.volumes_lvm.value != true or settings:common.libvirt_type.value
+          != 'kvm'
+        - action: hide
+          condition: not ('experimental' in version:feature_groups)
+        type: checkbox
+        value: false
+        weight: 11
+      metadata:
+        label: Storage
+        weight: 60
+      objects_ceph:
+        description: Configures RadosGW front end for Ceph RBD. This exposes S3 and
+          Swift API Interfaces. If enabled, this option will prevent Swift from installing.
+        label: Ceph RadosGW for objects (Swift API)
+        restrictions:
+        - settings:storage.images_ceph.value == false
+        type: checkbox
+        value: true
+        weight: 80
+      osd_pool_size:
+        description: Configures the default number of object replicas in Ceph. This
+          number must be equal to or lower than the number of deployed 'Storage -
+          Ceph OSD' nodes.
+        label: Ceph object replication factor
+        regex:
+          error: Invalid number
+          source: ^[1-9]\d*$
+        type: text
+        value: '2'
+        weight: 85
+      volumes_ceph:
+        description: Configures Cinder to store volumes in Ceph RBD images.
+        label: Ceph RBD for volumes (Cinder)
+        restrictions:
+        - settings:storage.volumes_lvm.value == true
+        type: checkbox
+        value: true
+        weight: 20
+      volumes_lvm:
+        description: It is recommended to have at least one Storage - Cinder LVM node.
+        label: Cinder LVM over iSCSI for volumes
+        restrictions:
+        - settings:storage.volumes_ceph.value == true
+        type: checkbox
+        value: false
+        weight: 10
+    workloads_collector:
+      enabled:
+        type: hidden
+        value: true
+      metadata:
+        label: Workloads Collector User
+        restrictions:
+        - action: hide
+          condition: 'true'
+        weight: 10
+      password:
+        type: password
+        value: YqvwqNhL
+      tenant:
+        type: text
+        value: services
+      user:
+        type: text
+        value: fuel_stats_user