Uplift Armband to Fuel Newton

[ Dan Andresan ]
- puppet: database: Fix Percona XtraBackup sync

[ Alexandru Avadanii ]
- disable all plugins;
- re-enable remote tracking;
- remove "Revert: Point to specific snapshot ...";
- patch context adjustments;
- obsolete "kernel-bump", linux-image-lts-xenial is now the default;
- network-checker iface state check now fixed upstream [1];
- fuel-nailgun-agent hugepage size should also check /proc/meminfo;
- fuel-nailgun-agent CPU detection for AArch64;
- nova AArch64 hugepage support is now upstream;
- obsolete Cirros Test VM direct kernel boot (by switch to AAVMF):
  * f-l/0001-upload_cirros-Add-direct-kernel-boot-support.patch
  * f-w/0001-direct-kernel-boot-for-cirros.patch
- rework m1.micro RAM size patch after puppet manifest split upstream;
- re-enable arch-agnostic plugins which were rebased in Fuel@OPNFV:
  * f_yardstick-pluginbuild
  * f_congress-pluginbuild
- do NOT retire MySQL SST provider patch series (nack: ARMBAND-186),
  rebase (and keep for now) MySQL SST provider patches, as trying to
  use xtrabackup-v2 revelead a regression since Colorado.3.0, and
  these patches simplify troubleshooting a lot;
- AArch64: nova: libvirt: Use host-model cpu (ARMBAND-193);
- AArch64: nova: libvirt: Use pointer_model instead of use_usb_tablet;
- m1.micro RAM size insufficient for TestVM with AAVMF (s/128/256/)
- switch Cirros TestVM to AAVMF from direct kernel boot;
- backport nova libvirt driver fix for deleting instances booted with
  AAVMF firmware from [2];

TODO (later):
- Include ISO build time fixes for cirros_testvm in Armband package;

TODO (ODL, later):
- test & revise leveldb patching;
- bring back Qugga patching for arm64;
- configure systemd service to automatically respawn;

[1] https://review.openstack.org/#/c/417373/
[2] https://review.openstack.org/#/c/357190/

JIRA: ARMBAND-29
JIRA: ARMBAND-32
JIRA: ARMBAND-63
JIRA: ARMBAND-88
JIRA: ARMBAND-116
JIRA: ARMBAND-118
JIRA: ARMBAND-186
JIRA: ARMBAND-193
JIRA: ARMBAND-194
JIRA: ARMBAND-195
JIRA: ARMBAND-196
JIRA: ARMBAND-197

Change-Id: Ia99022e364e61245d109cabab9d0ed7157b4d2f5
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
Signed-off-by: Dan Andresan <dan.andresan@enea.com>

6 files changed, 176 insertions, 112 deletions

diff --git a/patches/fuel-library/arm64-bug-fixes/0001-nova-config-Disable-usb-tablet-for-aarch64.patch b/patches/fuel-library/arm64-bug-fixes/0001-nova-config-Disable-usb-tablet-for-aarch64.patch
index f4e19229..eb611bfc 100644
--- a/patches/fuel-library/arm64-bug-fixes/0001-nova-config-Disable-usb-tablet-for-aarch64.patch
+++ b/patches/fuel-library/arm64-bug-fixes/0001-nova-config-Disable-usb-tablet-for-aarch64.patch
@@ -2,22 +2,29 @@ From: Stanislaw Kardach <stanislaw.kardach@cavium.com>
 Date: Tue, 15 Mar 2016 11:33:52 +0100
 Subject: [PATCH] nova config: Disable usb tablet for aarch64
 
+[ Alexandru Avadanii ]
+Rebased for Newton: reworked use_usb_tablet using pointer_model.
+
+JIRA: ARMBAND-194
+
+Signed-off-by: Stanislaw Kardach <stanislaw.kardach@cavium.com>
+Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
 ---
  deployment/puppet/openstack_tasks/manifests/roles/compute.pp | 4 ++++
  1 file changed, 4 insertions(+)
 
 diff --git a/deployment/puppet/openstack_tasks/manifests/roles/compute.pp b/deployment/puppet/openstack_tasks/manifests/roles/compute.pp
-index 211fefa..4240126 100644
+index 5b247b0..0a1a1c7 100644
 --- a/deployment/puppet/openstack_tasks/manifests/roles/compute.pp
 +++ b/deployment/puppet/openstack_tasks/manifests/roles/compute.pp
-@@ -379,6 +379,10 @@ class openstack_tasks::roles::compute {
-   }
-
-   nova_config {
-+    'libvirt/use_usb_tablet':       value => $::architecture ? {
-+        /(arm64|aarch64)/ => false,
-+        default           => true,
-+    };
+@@ -391,6 +391,10 @@ class openstack_tasks::roles::compute {
      'libvirt/live_migration_flag':  value => 'VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST';
      'libvirt/block_migration_flag': value => 'VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_NON_SHARED_INC';
      'DEFAULT/connection_type':      value => 'libvirt';
++    'DEFAULT/pointer_model':        value => $::architecture ? {
++        /(arm64|aarch64)/ => ps2mouse,
++        default           => usbtablet,
++    };
+   }
+
+   # TODO (iberezovskiy): rework this option management once it's available in puppet-nova module
diff --git a/patches/fuel-library/arm64-bug-fixes/0002-Install-vgabios-and-link-for-aarch64.patch b/patches/fuel-library/arm64-bug-fixes/0002-Install-vgabios-and-link-for-aarch64.patch
index da152ead..9b59349e 100644
--- a/patches/fuel-library/arm64-bug-fixes/0002-Install-vgabios-and-link-for-aarch64.patch
+++ b/patches/fuel-library/arm64-bug-fixes/0002-Install-vgabios-and-link-for-aarch64.patch
@@ -17,9 +17,9 @@ index 4240126..b18fa7b 100644
 --- a/deployment/puppet/openstack_tasks/manifests/roles/compute.pp
 +++ b/deployment/puppet/openstack_tasks/manifests/roles/compute.pp
 @@ -409,6 +409,14 @@ class openstack_tasks::roles::compute {
-     vncserver_listen                           => '0.0.0.0',
-     remove_unused_original_minimum_age_seconds => pick($nova_hash_real['remove_unused_original_minimum_age_seconds'], '86400'),
-     libvirt_service_name                       => $::nova::params::libvirt_service_name,
+     libvirt_service_name                       => 'libvirt-bin',
+     virtlock_service_name                      => 'virtlockd',
+     virtlog_service_name                       => 'virtlogd',
 +  } ->
 +  package { 'vgabios':
 +    ensure => present;
diff --git a/patches/fuel-library/arm64-bug-fixes/0003-nova-Update-vga-console-defaults-for-armv7-aarch64.patch b/patches/fuel-library/arm64-bug-fixes/0003-nova-Update-vga-console-defaults-for-armv7-aarch64.patch
index 9e9c3d53..be481a78 100644
--- a/patches/fuel-library/arm64-bug-fixes/0003-nova-Update-vga-console-defaults-for-armv7-aarch64.patch
+++ b/patches/fuel-library/arm64-bug-fixes/0003-nova-Update-vga-console-defaults-for-armv7-aarch64.patch
@@ -75,18 +75,17 @@ index b18fa7b..2f0f2a8 100644
    case $::osfamily {
      'RedHat': {
        # From legacy libvirt.pp
-@@ -290,6 +296,7 @@ class openstack_tasks::roles::compute {
+@@ -290,5 +296,6 @@ class openstack_tasks::roles::compute {
    }
 
    $notify_on_state_change = 'vm_and_task_state'
 +  $nova_path = '/usr/lib/python2.7/dist-packages/nova'
 
    class { '::nova':
-     rpc_backend                        => $rpc_backend_real,
-@@ -311,6 +318,18 @@ class openstack_tasks::roles::compute {
-     memcached_servers                  => $memcached_addresses,
-     cinder_catalog_info                => pick($nova_hash_real['cinder_catalog_info'], 'volumev2:cinderv2:internalURL'),
-     rabbit_heartbeat_timeout_threshold => $::os_service_default,
+@@ -311,5 +318,17 @@ class openstack_tasks::roles::compute {
+     rabbit_heartbeat_timeout_threshold     => $rabbit_heartbeat_timeout_threshold,
+     rabbit_heartbeat_rate                  => $rabbit_heartbeat_rate,
+     os_region_name                         => $region_name,
 +  } ->
 +  # FIXME(armband): Workaround for missing arm defaults in nova libvirt driver
 +  file { "${nova_path}/libvirt-vga-console.patch":
@@ -101,4 +100,3 @@ index b18fa7b..2f0f2a8 100644
 +    require => [Package['patch']],
    }
 
-   class { '::nova::availability_zone':
diff --git a/patches/fuel-library/arm64-bug-fixes/0004-AArch64-nova-libvirt-Use-host-model-cpu.patch b/patches/fuel-library/arm64-bug-fixes/0004-AArch64-nova-libvirt-Use-host-model-cpu.patch
new file mode 100644
index 00000000..aff89365
--- /dev/null
+++ b/patches/fuel-library/arm64-bug-fixes/0004-AArch64-nova-libvirt-Use-host-model-cpu.patch
@@ -0,0 +1,35 @@
+From: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+Date: Thu, 19 Jan 2017 19:19:28 +0100
+Subject: [PATCH] AArch64: nova: libvirt: Use host-model cpu
+
+Closes-bug: https://jira.opnfv.org/browse/ARMBAND-193
+
+Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+---
+ deployment/puppet/openstack_tasks/manifests/roles/compute.pp | 11 ++++++++++-
+ 1 file changed, 10 insertions(+), 1 deletion(-)
+
+diff --git a/deployment/puppet/openstack_tasks/manifests/roles/compute.pp b/deployment/puppet/openstack_tasks/manifests/roles/compute.pp
+index a6f4729..f6e6698 100644
+--- a/deployment/puppet/openstack_tasks/manifests/roles/compute.pp
++++ b/deployment/puppet/openstack_tasks/manifests/roles/compute.pp
+@@ -319,9 +319,18 @@ class openstack_tasks::roles::compute {
+   # guest OS by using a combination of CPU features and other parameters (such
+   # as CPUID level) that don't work. Until these issues are fixed, it's a good
+   # idea to avoid using host-model
++  # NOTE(armband): AArch64: Use host-model cpu
+   # http://libvirt.org/formatdomain.html#elementsCPU
+   # https://bugs.launchpad.net/mos/+bug/1618473
+-  $libvirt_cpu_mode = 'none'
++  # https://jira.opnfv.org/browse/ARMBAND-193
++  if str2bool($::is_virtual) {
++    $libvirt_cpu_mode = 'none'
++  } else {
++    $libvirt_cpu_mode = $::architecture ? {
++      /(arm64|aarch64)/ => 'host-model',
++      default           => 'none',
++    }
++  }
+
+   # Install / configure nova-compute
+
diff --git a/patches/fuel-library/arm64-bug-fixes/0004-nova-AArch64-enable-hugepage-support.patch b/patches/fuel-library/arm64-bug-fixes/0004-nova-AArch64-enable-hugepage-support.patch
deleted file mode 100644
index ab0fb664..00000000
--- a/patches/fuel-library/arm64-bug-fixes/0004-nova-AArch64-enable-hugepage-support.patch
+++ /dev/null
@@ -1,93 +0,0 @@
-From: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
-Date: Fri, 16 Sep 2016 21:31:56 +0200
-Subject: [PATCH] nova: AArch64: enable hugepage support
-
-This patch adds AArch64 to the hardcoded lists of valid
-architectures for hugepage support.
-
-Closes-bug: ARMBAND-90
-
-Signed-off-by: Veena Lingadahalli <vlingadahalli@mvista.com>
-Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
----
- .../nova-AArch64-enable-hugepage-support.patch     | 47 ++++++++++++++++++++++
- .../openstack_tasks/manifests/roles/compute.pp     | 12 ++++++
- 2 files changed, 59 insertions(+)
- create mode 100644 deployment/puppet/openstack/files/nova-AArch64-enable-hugepage-support.patch
-
-diff --git a/deployment/puppet/openstack/files/nova-AArch64-enable-hugepage-support.patch b/deployment/puppet/openstack/files/nova-AArch64-enable-hugepage-support.patch
-new file mode 100644
-index 0000000..72d6bfc
---- /dev/null
-+++ b/deployment/puppet/openstack/files/nova-AArch64-enable-hugepage-support.patch
-@@ -0,0 +1,47 @@
-+From: Veena Lingadahalli <vlingadahalli@mvista.com>
-+Date: Fri Sep 16 17:52:02 CEST 2016
-+Subject: [PATCH] nova: AArch64: enable hugepage support
-+
-+This patch adds AArch64 to the hardcoded lists of valid
-+architectures for hugepage support.
-+
-+Signed-off-by: Veena Lingadahalli <vlingadahalli@mvista.com>
-+Acked-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
-+---
-+
-+diff --git a/compute/arch.py a/compute/arch.py
-+index 265f853..6a4b4a3 100644
-+--- a/compute/arch.py
-++++ a/compute/arch.py
-+@@ -156,6 +156,9 @@ def canonicalize(name):
-+     if newname == "amd64":
-+         newname = X86_64
-+
-++    if newname == "aarch64":
-++        newname = AARCH64
-++
-+     if not is_valid(newname):
-+         raise exception.InvalidArchitectureName(arch=name)
-+
-+diff --git a/virt/libvirt/driver.py a/virt/libvirt/driver.py
-+index bac17cf..0e0bcae 100644
-+--- a/virt/libvirt/driver.py
-++++ a/virt/libvirt/driver.py
-+@@ -5206,7 +5206,7 @@ class LibvirtDriver(driver.ComputeDriver):
-+                     self._bad_libvirt_numa_version_warn = True
-+                 return False
-+
-+-        support_matrix = {(arch.I686, arch.X86_64): MIN_LIBVIRT_NUMA_VERSION,
-++        support_matrix = {(arch.I686, arch.X86_64, arch.AARCH64): MIN_LIBVIRT_NUMA_VERSION,
-+                           (arch.PPC64,
-+                            arch.PPC64LE): MIN_LIBVIRT_NUMA_VERSION_PPC}
-+         caps = self._host.get_capabilities()
-+@@ -5222,7 +5222,7 @@ class LibvirtDriver(driver.ComputeDriver):
-+     def _has_hugepage_support(self):
-+         # This means that the host can support multiple values for the size
-+         # field in LibvirtConfigGuestMemoryBackingPage
-+-        supported_archs = [arch.I686, arch.X86_64]
-++        supported_archs = [arch.I686, arch.X86_64, arch.AARCH64]
-+         caps = self._host.get_capabilities()
-+         return ((caps.host.cpu.arch in supported_archs) and
-+                 self._host.has_min_version(MIN_LIBVIRT_HUGEPAGE_VERSION,
-diff --git a/deployment/puppet/openstack_tasks/manifests/roles/compute.pp b/deployment/puppet/openstack_tasks/manifests/roles/compute.pp
-index 2f0f2a8..dc61033 100644
---- a/deployment/puppet/openstack_tasks/manifests/roles/compute.pp
-+++ b/deployment/puppet/openstack_tasks/manifests/roles/compute.pp
-@@ -330,6 +330,18 @@ class openstack_tasks::roles::compute {
-     unless  => "patch -p1 -R -N --dry-run < ${nova_path}/libvirt-vga-console.patch",
-     cwd     => $nova_path,
-     require => [Package['patch']],
-+  } ->
-+  # FIXME(armband): Add AArch64 to arch list for nova hugepage support
-+  file { "${nova_path}/nova-AArch64-enable-hugepage-support.patch":
-+    ensure => "file",
-+    source => "puppet:///modules/openstack/nova-AArch64-enable-hugepage-support.patch",
-+  } ->
-+  exec { 'nova AArch64 enable hugepage support':
-+    path    => ['/usr/bin'],
-+    command => "patch -p1 < ${nova_path}/nova-AArch64-enable-hugepage-support.patch",
-+    unless  => "patch -p1 -R -N --dry-run < ${nova_path}/nova-AArch64-enable-hugepage-support.patch",
-+    cwd     => $nova_path,
-+    require => [Package['patch']],
-   }
-
-   class { '::nova::availability_zone':
diff --git a/patches/fuel-library/arm64-bug-fixes/0005-nova-libvirt-fix-delete-instance-with-nvram.patch b/patches/fuel-library/arm64-bug-fixes/0005-nova-libvirt-fix-delete-instance-with-nvram.patch
new file mode 100644
index 00000000..9bf9a5c4
--- /dev/null
+++ b/patches/fuel-library/arm64-bug-fixes/0005-nova-libvirt-fix-delete-instance-with-nvram.patch
@@ -0,0 +1,117 @@
+From: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+Date: Thu, 19 Jan 2017 23:03:54 +0100
+Subject: [PATCH] nova: libvirt: fix delete instance with nvram
+
+Backported from [1].
+
+Closes-bug: 1567807
+
+[1] https://review.openstack.org/#/q/
+    539d381434ccadcdc3f5d58c2705c35558a3a065
+
+Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+---
+ .../files/nova-libvirt-delete-with-nvram.patch     | 70 ++++++++++++++++++++++
+ .../openstack_tasks/manifests/roles/compute.pp     | 12 ++++
+ 2 files changed, 82 insertions(+)
+ create mode 100644 deployment/puppet/openstack/files/nova-libvirt-delete-with-nvram.patch
+
+diff --git a/deployment/puppet/openstack/files/nova-libvirt-delete-with-nvram.patch b/deployment/puppet/openstack/files/nova-libvirt-delete-with-nvram.patch
+new file mode 100644
+index 0000000..5d4f67c
+--- /dev/null
++++ b/deployment/puppet/openstack/files/nova-libvirt-delete-with-nvram.patch
+@@ -0,0 +1,70 @@
++From: Kevin Zhao <kevin.zhao@linaro.org>
++Date: Thu, 5 Jan 2017 21:32:41 +0000
++Subject: [PATCH] libvirt: fix nova can't delete the instance with nvram
++
++Currently libvirt needs a flag when deleting an VM with a nvram file,
++without which nova can't delete an instance booted with UEFI. Add
++deletion flag for NVRAM. Also add a test case.
++
++[ Alexandru Avadanii ]
++Removed chunks affecting tests and adapted for OPNFV Armband.
++
++Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
++Co-authored-by: Derek Higgins <derekh@redhat.com>
++Change-Id: I46baa952b6c3a1a4c5cf2660931f317cafb5757d
++Closes-Bug: #1567807
++---
++
++diff --git a/virt/libvirt/driver.py b/virt/libvirt/driver.py
++--- a/virt/libvirt/driver.py
+++++ b/virt/libvirt/driver.py
++@@ -903,7 +903,8 @@ class LibvirtDriver(driver.ComputeDriver):
++         try:
++             guest = self._host.get_guest(instance)
++             try:
++-                guest.delete_configuration()
+++                support_uefi = self._has_uefi_support()
+++                guest.delete_configuration(support_uefi)
++             except libvirt.libvirtError as e:
++                 with excutils.save_and_reraise_exception():
++                     errcode = e.get_error_code()
++@@ -1241,7 +1242,8 @@ class LibvirtDriver(driver.ComputeDriver):
++             #             If any part of this block fails, the domain is
++             #             re-defined regardless.
++             if guest.has_persistent_configuration():
++-                guest.delete_configuration()
+++                support_uefi = self._has_uefi_support()
+++                guest.delete_configuration(support_uefi)
++
++             # Start copy with VIR_DOMAIN_REBASE_REUSE_EXT flag to
++             # allow writing to existing external volume file
++@@ -1760,7 +1762,8 @@ class LibvirtDriver(driver.ComputeDriver):
++             #             If any part of this block fails, the domain is
++             #             re-defined regardless.
++             if guest.has_persistent_configuration():
++-                guest.delete_configuration()
+++                support_uefi = self._has_uefi_support()
+++                guest.delete_configuration(support_uefi)
++
++             # NOTE (rmk): Establish a temporary mirror of our root disk and
++             #             issue an abort once we have a complete copy.
++diff --git a/virt/libvirt/guest.py b/virt/libvirt/guest.py
++--- a/virt/libvirt/guest.py
+++++ b/virt/libvirt/guest.py
++@@ -262,11 +262,13 @@ class Guest(object):
++                 yield VCPUInfo(
++                     id=vcpu[0], cpu=vcpu[3], state=vcpu[1], time=vcpu[2])
++
++-    def delete_configuration(self):
+++    def delete_configuration(self, support_uefi=False):
++         """Undefines a domain from hypervisor."""
++         try:
++-            self._domain.undefineFlags(
++-                libvirt.VIR_DOMAIN_UNDEFINE_MANAGED_SAVE)
+++            flags = libvirt.VIR_DOMAIN_UNDEFINE_MANAGED_SAVE
+++            if support_uefi:
+++                flags |= libvirt.VIR_DOMAIN_UNDEFINE_NVRAM
+++            self._domain.undefineFlags(flags)
++         except libvirt.libvirtError:
++             LOG.debug("Error from libvirt during undefineFlags. %d"
++                       "Retrying with undefine", self.id)
+diff --git a/deployment/puppet/openstack_tasks/manifests/roles/compute.pp b/deployment/puppet/openstack_tasks/manifests/roles/compute.pp
+index f6e6698..dd0c034 100644
+--- a/deployment/puppet/openstack_tasks/manifests/roles/compute.pp
++++ b/deployment/puppet/openstack_tasks/manifests/roles/compute.pp
+@@ -301,6 +301,18 @@ class openstack_tasks::roles::compute {
+     unless  => "patch -p1 -R -N --dry-run < ${nova_path}/libvirt-vga-console.patch",
+     cwd     => $nova_path,
+     require => [Package['patch']],
++  } ->
++  # FIXME(armband): Backport fix: nova delete instance with nvram
++  file { "${nova_path}/nova-libvirt-delete-with-nvram.patch":
++    ensure => "file",
++    source => "puppet:///modules/openstack/nova-libvirt-delete-with-nvram.patch",
++  } ->
++  exec { 'nova libvirt delete instance with nvram':
++    path    => ['/usr/bin'],
++    command => "patch -p1 < ${nova_path}/nova-libvirt-delete-with-nvram.patch",
++    unless  => "patch -p1 -R -N --dry-run < ${nova_path}/nova-libvirt-delete-with-nvram.patch",
++    cwd     => $nova_path,
++    require => [Package['patch']],
+   }
+
+   class { '::nova::cache':