MaaS: Add support for dynamic fabric numbering

Previously, we hardcoded the fabric name for our 3rd interface
(which serves PXE/DHCP for the target nodes) to "fabric-2",
relying on predictable index numbers to be provided by MaaS based
on the interfaces defined in /etc/network/interfaces.

However, the fabric IDs/names generated by MaaS are not predictable,
and therefore cannot be hardcoded in our reclass model / scripts.

Work around this by:
- adding support for fabric ID deduction based on CIDR matching
  during subnet create/update operation in MaaS py module;
- adding support for VLAN DHCP enablement to MaaS py module,
  which was previously handled via shell MaaS API operations
  from maas/region.sls;

While at it, revert previous commit that disabled network discovery
("MaaS: Disable network discovery"), since it turns out that network
discovery was not the culprit for subnet creation failure, but wrong
fabric numbering.

This reverts commit 8cdf22d1a1bae4694a373873cab4feb6251069b7.

Change-Id: I15fa059004356cb4aaabb38999ea378dd3c0e0bb
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>

6 files changed, 139 insertions, 33 deletions

diff --git a/mcp/patches/0005-maas-module-Obtain-fabric-ID-from-CIDR.patch b/mcp/patches/0005-maas-module-Obtain-fabric-ID-from-CIDR.patch
new file mode 100644
index 000000000..5c77a63da
--- /dev/null
+++ b/mcp/patches/0005-maas-module-Obtain-fabric-ID-from-CIDR.patch
@@ -0,0 +1,49 @@
+From: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+Date: Sat, 19 Aug 2017 02:03:01 +0200
+Subject: [PATCH] maas: module: Obtain fabric ID from CIDR
+
+MaaS subnet update requires specifying the correct fabric via reclass,
+which we used to hardcode in our OPNFV reclass model to fabric-2.
+However, fabric index numbers are not deterministic, so the old
+method is unreliable.
+
+Update MaaS custom py module to determine fabric name/ID on the
+fly, based on CIDR matching (assuming we don't have CIDR conflicts).
+
+This change maintains backwards compatibility:
+- if fabric is specified via reclass model, it will be used as-is;
+- if fabric is not specified via reclass model, we try to deduce it
+  based on CIDR; if no match is found, the old default ('') is used;
+
+Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+Signed-off-by: Guillermo Herrero <Guillermo.Herrero@enea.com>
+---
+
+diff --git a/_modules/maas.py b/_modules/maas.py
+index d3227ca..8a2243d 100644
+--- a/_modules/maas.py
++++ b/_modules/maas.py
+@@ -198,7 +198,8 @@
+     def fill_data(self, name, subnet, fabrics):
+         data = {
+             'name': name,
+-            'fabric': str(fabrics[subnet.get('fabric', '')]),
++            'fabric': str(fabrics[subnet.get('fabric',
++                self._get_fabric_from_cidr(subnet.get('cidr')))]),
+             'cidr': subnet.get('cidr'),
+             'gateway_ip': subnet['gateway_ip'],
+         }
+@@ -215,6 +216,13 @@
+         self._process_iprange(res_json['id'])
+         return response
+
++    def _get_fabric_from_cidr(self, cidr):
++        subnets = json.loads(self._maas.get(u'api/2.0/subnets/').read())
++        for subnet in subnets:
++            if subnet['cidr'] == cidr:
++                return subnet['vlan']['fabric']
++        return ''
++
+     def _process_iprange(self, subnet_id):
+         ipranges = json.loads(self._maas.get(u'api/2.0/ipranges/').read())
+         LOG.warn('all %s ipranges %s', subnet_id, ipranges)
diff --git a/mcp/patches/0005-maas-vlan-DHCP-enable-on-fabric-2.patch b/mcp/patches/0005-maas-vlan-DHCP-enable-on-fabric-2.patch
deleted file mode 100644
index 777850446..000000000
--- a/mcp/patches/0005-maas-vlan-DHCP-enable-on-fabric-2.patch
+++ /dev/null
@@ -1,28 +0,0 @@
-From: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
-Date: Sat, 5 Aug 2017 02:03:01 +0200
-Subject: [PATCH] maas: vlan DHCP enable on fabric-2
-
-MaaS custom py modules does not implement vlan API support,
-so use MaaS CLI directly to enable DHCP on vlan 0 on fabric-2.
-
-Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
-Signed-off-by: Guillermo Herrero <Guillermo.Herrero@enea.com>
----
-
-diff --git a/maas/region.sls b/maas/region.sls
-index d3227ca..8a2243d 100644
---- a/maas/region.sls
-+++ b/maas/region.sls
-@@ -190,4 +190,12 @@
-   - require:
-     - module: maas_config
-
-+maas_vlan_dhcp:
-+# NOTE(armband): To be implemented via _modules/maas.py later
-+  cmd.run:
-+  - name: "maas login {{ region.admin.username }} http://{{ region.bind.host }}:5240/MAAS/api/2.0 - < /var/lib/maas/.maas_credentials && maas opnfv vlan update 2 0 dhcp_on=True primary_rack={{ region.maas_config.maas_name }}"
-+  - require:
-+    - module: maas_subnets
-+    - module: maas_dhcp_snippets
-+
- {%- endif %}
diff --git a/mcp/patches/0006-maas-module-Add-VLAN-DHCP-enable-support.patch b/mcp/patches/0006-maas-module-Add-VLAN-DHCP-enable-support.patch
new file mode 100644
index 000000000..aa829e076
--- /dev/null
+++ b/mcp/patches/0006-maas-module-Add-VLAN-DHCP-enable-support.patch
@@ -0,0 +1,80 @@
+From: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+Date: Sat, 19 Aug 2017 02:03:01 +0200
+Subject: [PATCH] maas: module: Add VLAN DHCP enable support
+
+MaaS custom py module does not support VLAN configuration.
+This should be implemented by adding a dedicated class for VLAN.
+However, we are only interested in updating an existign VLAN to
+enable DHCP on an existing IP range (set up via subnet configuration),
+so extend existing subnet handling to include basic VLAN update.
+
+NOTE: Design-wise, this is hacky, and its only purpose is to allow
+setting 'dhcp_on=True' for an existing VLAN.
+
+Example reclass model usage:
+maas:
+  region:
+    subnets:
+      192.168.11.0/24:
+        # ...
+        vlans:
+          untagged:
+            vid: 0
+            dhcp_on: true
+            primary_rack: mas01
+
+Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+---
+
+diff --git a/_modules/maas.py b/_modules/maas.py
+index d3227ca..8a2243d 100644
+--- a/_modules/maas.py
++++ b/_modules/maas.py
+             'gateway_ip': subnet['gateway_ip'],
+         }
+         self._iprange = subnet['iprange']
++        self._vlans = subnet['vlans']
+         return data
+
+     def update(self, new, old):
+@@ -214,6 +215,7 @@
+         response = super(Subnet, self).send(data)
+         res_json = json.loads(response)
+         self._process_iprange(res_json['id'])
++        self._process_dhcp_vlans_update(data)
+         return response
+
+     def _get_fabric_from_cidr(self, cidr):
+@@ -248,6 +250,32 @@
+         else:
+             self._maas.post(u'api/2.0/ipranges/', None, **data)
+
++    def _process_dhcp_vlans_update(self, subnet_data):
++        fabric_vlans = json.loads(self._maas.get(u'api/2.0/fabrics/{0}/vlans/'
++                                  .format(subnet_data['fabric'])).read())
++        LOG.warn('all fabric %s vlans %s', subnet_data['fabric'], fabric_vlans)
++        for vlan_name, vlan_data in self._vlans.iteritems():
++            update = False
++            old_data = None
++            for fabric_vlan in fabric_vlans:
++                if fabric_vlan['vid'] == vlan_data['vid']:
++                    update = True
++                    old_data = fabric_vlan
++                    break
++            data = {
++                'mtu': str(vlan_data.get('mtu', 1500)),
++                'dhcp_on': str(vlan_data.get('dhcp_on')),
++                'primary_rack': str(vlan_data.get('primary_rack')),
++                'secondary_rack': str(vlan_data.get('secondary_rack', ''))
++            }
++            if update:
++                LOG.warn('UPDATING %s %s', data, old_data)
++                self._maas.put(u'api/2.0/fabrics/{0}/vlans/{1}/'
++                               .format(subnet_data['fabric'], old_data['vid']),
++                               **data)
++            else:
++                LOG.warn('MISSING vlan %s, not doing anything', data)
++
+
+ class DHCPSnippet(MaasObject):
+     def __init__(self):
diff --git a/mcp/patches/0006-linux.network.interface-noifupdown-support.patch b/mcp/patches/0007-linux.network.interface-noifupdown-support.patch
index 5d9ce7be5..5d9ce7be5 100644
--- a/mcp/patches/0006-linux.network.interface-noifupdown-support.patch
+++ b/mcp/patches/0007-linux.network.interface-noifupdown-support.patch
diff --git a/mcp/patches/patches.list b/mcp/patches/patches.list
index acacb47e0..1a651cfe0 100644
--- a/mcp/patches/patches.list
+++ b/mcp/patches/patches.list
@@ -2,5 +2,6 @@
 /usr/share/salt-formulas/env: 0002-maas-region-skip-credentials-update.patch
 /usr/share/salt-formulas/env: 0003-maas-region-force-artifact-download.patch
 /usr/share/salt-formulas/env: 0004-maas-region-use-authorized_keys-1st-entry.patch
-/usr/share/salt-formulas/env: 0005-maas-vlan-DHCP-enable-on-fabric-2.patch
-/usr/share/salt-formulas/env: 0006-linux.network.interface-noifupdown-support.patch
+/usr/share/salt-formulas/env: 0005-maas-module-Obtain-fabric-ID-from-CIDR.patch
+/usr/share/salt-formulas/env: 0006-maas-module-Add-VLAN-DHCP-enable-support.patch
+/usr/share/salt-formulas/env: 0007-linux.network.interface-noifupdown-support.patch
diff --git a/mcp/reclass/classes/cluster/baremetal-mcp-ocata-ovs-ha/infra/maas.yml b/mcp/reclass/classes/cluster/baremetal-mcp-ocata-ovs-ha/infra/maas.yml
index cebcb90ae..d193469fc 100644
--- a/mcp/reclass/classes/cluster/baremetal-mcp-ocata-ovs-ha/infra/maas.yml
+++ b/mcp/reclass/classes/cluster/baremetal-mcp-ocata-ovs-ha/infra/maas.yml
@@ -29,18 +29,22 @@ parameters:
         disk_erase_with_secure_erase: false
         dnssec_validation: 'no'
         enable_third_party_drivers: true
-        network_discovery: 'disabled'
+        network_discovery: 'enabled'
         default_min_hwe_kernel: 'hwe-16.04'
       subnets:
-# TODO: parametrize address root (192.168.11), fabric-2, dhcp start/end?
+        # TODO: parametrize address root (192.168.11), dhcp start/end?
         192.168.11.0/24:
-          fabric: fabric-2
           cidr: 192.168.11.0/24
           gateway_ip: ${_param:single_address}
           iprange:
             start: 192.168.11.5
             end: 192.168.11.250
             type: dynamic
+          vlans:
+            untagged:
+              vid: 0
+              dhcp_on: true
+              primary_rack: ${_param:infra_maas_node01_hostname}
       machines:
         kvm01:
           interface: