Add NOVCP HA OVS scenario (baremetal, virtual)

Add a new class of scenarios, based on existing baremetal HA
scenarios, but instead of having a virtualized control plane (VCP),
all Openstack controller services will run directly on the cluster
nodes.
This change adds the common scaffolding, as well as the OVS scenario.

The new scenario(s) can be used on full-baremetal clusters, soon on
full-virtual clusters and later on hybrid (virt + bare) clusters.
This change defines old (current) style scenario definitions for
both baremetal and virtual, both named:
- os-nosdn-nofeature-novcp-ha;

Prerequisites:
1. Merge-able by name reclass.storage.node definitions
   Each cluster (e.g. database, telemetry) adds its own set of
   reclass storage node defitions, which for novcp scenarios should
   be merged into a single node (kvm) based on the 'name' property.
   This is not currently supported by upstream reclass 'node.sls'
   high state, so add support for it via an early patch (required
   before salt-master-init.sh tries to handle reclass.storage).
2. common reclass classes for novcp
   Some of the classes in `baremetal-...-common-ha` are not fit for
   novcp as they define VCP-specific config/inheritance, so add new
   versions of said classes with novcp in mind or adapt old classes:
   - parameterize ctl hostname in `openstack_compute.yml`;
   - new `openstack_control_novcp.yml`;
   - new `openstack_init_novcp.yml`;
3. Handle hard set names in state files for baremetal nodes
   Some of our state files (e.g. maas) hardcodes baremetal node names
   to 'kvm', 'cmp', so we need to align the names in novcp scenario
   with these values to re-use the maas state. As a future improvement
   we should parameterize these names in all state files.
   As a consequence, our baremetal controller nodes will also use
   'kvm*' hostnames (instead of 'ctl*').
4. Add 'noifupdown' to all interfaces on kvm nodes to prevent duplicate
   IPs/routes created at *any* ifup due to /etc/network/route-br-ex.
   Patch salt-formula-linux to skip network restart on 'noifupdown',
   also when routes are present on that interface.

JIRA: FUEL-310

Change-Id: Ic67778f63e5ee0334dbfe9547c7109ec1a938d61
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>

6 files changed, 121 insertions, 5 deletions

diff --git a/mcp/patches/0011-routes-Skip-network-restart-on-noifupdown.patch b/mcp/patches/0011-routes-Skip-network-restart-on-noifupdown.patch
new file mode 100644
index 000000000..5ccd04aef
--- /dev/null
+++ b/mcp/patches/0011-routes-Skip-network-restart-on-noifupdown.patch
@@ -0,0 +1,29 @@
+From: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+Date: Mon, 8 Jan 2018 05:09:11 +0100
+Subject: [PATCH] routes: Skip network restart on 'noifupdown'
+
+Previously, setting up routes did not allow passing 'require_reboot',
+so each route change would lead to a networking service restart,
+rendering interface configuration options like 'noifupdown' useless.
+Allow disabling network restart per-interface using the existing
+'noifupdown' option.
+
+Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+---
+ linux/network/interface.sls | 3 +++
+ 1 file changed, 3 insertions(+)
+
+diff --git a/linux/network/interface.sls b/linux/network/interface.sls
+index 921ceac..6ebc670 100644
+--- a/linux/network/interface.sls
++++ b/linux/network/interface.sls
+@@ -338,6 +338,9 @@ linux_network_{{ interface_name }}_routes:
+       gateway: {{ route.gateway }}
+       {%- endif %}
+     {%- endfor %}
++  {%- if interface.noifupdown is defined %}
++  - require_reboot: {{ interface.noifupdown }}
++  {%- endif %}
+
+ {%- endif %}
+
diff --git a/mcp/patches/0014-reclass.storage.node-Merge-duplicate-nodes.patch b/mcp/patches/0014-reclass.storage.node-Merge-duplicate-nodes.patch
new file mode 100644
index 000000000..25159bc4a
--- /dev/null
+++ b/mcp/patches/0014-reclass.storage.node-Merge-duplicate-nodes.patch
@@ -0,0 +1,44 @@
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+: Copyright (c) 2017 Mirantis Inc., Enea AB and others.
+:
+: All rights reserved. This program and the accompanying materials
+: are made available under the terms of the Apache License, Version 2.0
+: which accompanies this distribution, and is available at
+: http://www.apache.org/licenses/LICENSE-2.0
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+From: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+Date: Sun, 03 Dec 2017 22:03:01 +0200
+Subject: [PATCH] reclass.storage.node: Merge duplicate nodes
+
+Reclass does not support duplicate nodes in top pillar, so merge all
+nodes with the same name into a single node, inheriting classes from
+all instances.
+
+This allows using multiple "system.reclass.storage.system.*_cluster"
+classes for the same node, based on re-using the name (hostname).
+
+NOTE: defaults.merge module does not merge lists (e.g. for classes),
+so handle that case separately.
+
+Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+---
+
+diff --git a/reclass/storage/node.sls b/reclass/storage/node.sls
+--- a/reclass/storage/node.sls
++++ b/reclass/storage/node.sls
+@@ -1,7 +1,15 @@
+ {%- from "reclass/map.jinja" import storage with context %}
+ {%- if storage.enabled %}
+
++{%- set storage_by_name = {} %}
+ {%- for node_name, node in storage.get('node', {}).iteritems() %}
++{%- set new_node_name = node.get('name', node_name) %}
++{%- set new_node = storage_by_name.get(new_node_name, {'classes': []}) %}
++{%- do salt['defaults.merge'](node, {'classes': new_node.classes + node.classes}) %}
++{%- do salt['defaults.merge'](storage_by_name, {new_node_name: node}) %}
++{%- endfor %}
++
++{%- for node_name, node in storage_by_name.iteritems() %}
+
+ {%- if node.repeat is defined %}
+
diff --git a/mcp/patches/patch.sh b/mcp/patches/patch.sh
index 1da3bc597..bb48dcd07 100755
--- a/mcp/patches/patch.sh
+++ b/mcp/patches/patch.sh
@@ -11,10 +11,9 @@ CI_DEBUG=${CI_DEBUG:-0}; [[ "${CI_DEBUG}" =~ (false|0) ]] || set -x
 
 if [ -r "$1" ]; then
   while IFS=': ' read -r p_dest p_file; do
-    if [[ ! "${p_dest}" =~ '^#' ]] && [[ "${p_dest}" =~ $2 ]] && \
-      ! patch --dry-run -Rd "${p_dest}" -r - -s -p1 < \
-        "/root/fuel/mcp/patches/${p_file}" > /dev/null; then
-          patch -d "${p_dest}" -p1 < "/root/fuel/mcp/patches/${p_file}"
+    if ! patch --dry-run -Rd "${p_dest}" -r - -s -p1 < \
+      "/root/fuel/mcp/patches/${p_file}" > /dev/null; then
+        patch -d "${p_dest}" -p1 < "/root/fuel/mcp/patches/${p_file}"
     fi
-  done < "$1"
+  done < <(grep -vE '^#' "${1}" | grep -E "^.*${2}.*: ")
 fi
diff --git a/mcp/patches/patches.list b/mcp/patches/patches.list
index 2eb45b272..703bd0587 100644
--- a/mcp/patches/patches.list
+++ b/mcp/patches/patches.list
@@ -16,5 +16,6 @@
 /usr/share/salt-formulas/env: 0009-controller-Use-keystoneclient-to-check-project-ID.patch
 /usr/share/salt-formulas/env: 0010-maas-region-allow-timeout-override.patch
 /usr/share/salt-formulas/env: 0011-system.repo-Debian-Add-keyserver-proxy-support.patch
+/usr/share/salt-formulas/env: 0011-routes-Skip-network-restart-on-noifupdown.patch
 /usr/share/salt-formulas/env: 0012-linux.storage.lvm-Disable-filter.patch
 /usr/share/salt-formulas/env: 0013-dpdk-Support-ovs-bridge-tagging.patch
diff --git a/mcp/patches/patches_init.list b/mcp/patches/patches_init.list
new file mode 100644
index 000000000..94488cf9a
--- /dev/null
+++ b/mcp/patches/patches_init.list
@@ -0,0 +1,8 @@
+##############################################################################
+# Copyright (c) 2017 Mirantis Inc., Enea AB and others.
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Apache License, Version 2.0
+# which accompanies this distribution, and is available at
+# http://www.apache.org/licenses/LICENSE-2.0
+##############################################################################
+/usr/share/salt-formulas/env: 0014-reclass.storage.node-Merge-duplicate-nodes.patch
diff --git a/mcp/patches/scripts/0002-salt-master-init.sh-Apply-OPNFV-Fuel-patches.patch b/mcp/patches/scripts/0002-salt-master-init.sh-Apply-OPNFV-Fuel-patches.patch
new file mode 100644
index 000000000..978815020
--- /dev/null
+++ b/mcp/patches/scripts/0002-salt-master-init.sh-Apply-OPNFV-Fuel-patches.patch
@@ -0,0 +1,35 @@
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+: Copyright (c) 2017 Mirantis Inc., Enea AB and others.
+:
+: All rights reserved. This program and the accompanying materials
+: are made available under the terms of the Apache License, Version 2.0
+: which accompanies this distribution, and is available at
+: http://www.apache.org/licenses/LICENSE-2.0
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+From: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+Date: Fri, 8 Dec 2017 20:30:46 +0100
+Subject: [PATCH] salt-master-init.sh: Apply OPNFV Fuel patches
+
+Some of Fuel@OPNFV patches need to be applied before the reclass
+storage.node state is ran for Salt Master, i.e. between installing
+salt-formula-* packages and configuring the Salt Master salt services.
+
+JIRA: FUEL-310
+
+Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+---
+ salt-master-init.sh | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/salt-master-init.sh b/salt-master-init.sh
+index 343324c..a4ec138 100755
+--- a/salt-master-init.sh
++++ b/salt-master-init.sh
+@@ -284,6 +284,7 @@ options
+     system_config
+
+     saltmaster_bootstrap &&\
++    /root/fuel/mcp/patches/patch.sh /root/fuel/mcp/patches/patches_init.list formulas &&\
+     saltmaster_init &&\
+
+     verify_salt_minions