Add fixes and improvements for arm64 deployment

[ Florin Dumitrascu <florin.dumitrascu@enea.com> ]

* arm64 support for OpenDaylight Fuel Plugin

[ Stanislaw Kardach <kda@semihalf.com> ]

* Limit ESP to first drive
* Remove default+timeout+tr from cobbler profile
* Fix puppet syntax errors
* Disable usb tablet on aarch64
* Support direct kernel boot for CirrOS TestVM on aarch64
* Remove git version signature not to confuse patches-export
* Fix VGA support for CirrOS TestVM
* Makefile: Add clean-{docker,build}.

[ Alexandru Avadanii <Alexandru.Avadanii@enea.com> ]

* Performance: Use gzip instead of xz compression.
* Switch mirror proto from https to http.
* Update TODO with remaining tasks.
* Disable amd64 Liberty fixture (no multi-arch support yet).
* m1.micro: Increase RAM size to 128MB for aarch64 images.

Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
Signed-off-by: Florin Dumitrascu <florin.dumitrascu@enea.com>
Signed-off-by: Stanislaw Kardach <kda@semihalf.com>

7 files changed, 241 insertions, 32 deletions

diff --git a/patches/fuel-library/0001-Add-arm64-templates-to-cobbler-and-nailgun.patch b/patches/fuel-library/0001-Add-arm64-templates-to-cobbler-and-nailgun.patch
index 56796d48..d8a6a03d 100644
--- a/patches/fuel-library/0001-Add-arm64-templates-to-cobbler-and-nailgun.patch
+++ b/patches/fuel-library/0001-Add-arm64-templates-to-cobbler-and-nailgun.patch
@@ -9,13 +9,13 @@ Add arm64 templates to cobbler and nailgun
  deployment/puppet/cobbler/manifests/server.pp      | 27 +++++++++++++++
  .../puppet/cobbler/templates/dnsmasq.template.erb  |  2 ++
  .../cobbler/templates/efidefault.template.erb      |  5 +++
- .../cobbler/templates/grubprofile.template.erb     | 16 +++++++++
+ .../cobbler/templates/grubprofile.template.erb     | 12 +++++++
  .../cobbler/templates/grubsystem.template.erb      | 16 +++++++++
  .../templates/preseed/ubuntu-1404.preseed.erb      |  2 ++
  .../puppet/nailgun/manifests/auxiliaryrepos.pp     | 12 ++++---
  deployment/puppet/nailgun/manifests/cobbler.pp     | 38 +++++++++++++++++-----
  .../puppet/nailgun/manifests/cobbler/preseed.pp    | 13 ++++++++
- 9 files changed, 119 insertions(+), 12 deletions(-)
+ 9 files changed, 115 insertions(+), 12 deletions(-)
  create mode 100644 deployment/puppet/cobbler/templates/efidefault.template.erb
  create mode 100644 deployment/puppet/cobbler/templates/grubprofile.template.erb
  create mode 100644 deployment/puppet/cobbler/templates/grubsystem.template.erb
@@ -84,14 +84,10 @@ index 0000000..af09eb8
 +$grub_menu_items
 diff --git a/deployment/puppet/cobbler/templates/grubprofile.template.erb b/deployment/puppet/cobbler/templates/grubprofile.template.erb
 new file mode 100644
-index 0000000..9a191bc
+index 0000000..eb7557c
 --- /dev/null
 +++ b/deployment/puppet/cobbler/templates/grubprofile.template.erb
-@@ -0,0 +1,16 @@
-+set default=1
-+set timeout=5
-+tr -s ipxe_mac ':' '-' $net_default_mac
-+
+@@ -0,0 +1,12 @@
 +menuentry '$profile_name' {
 +    echo 'Network status:'
 +    net_ls_cards
@@ -268,6 +264,3 @@ index 0000000..93c71ba
 +    require => Class['::cobbler::server'],
 +  }
 +}
--- 
-1.9.1
-
diff --git a/patches/fuel-library/0002-Add-arm64-for-auxiliary-repos.patch b/patches/fuel-library/0002-Add-arm64-for-auxiliary-repos.patch
index 0bf54a2c..591d3968 100644
--- a/patches/fuel-library/0002-Add-arm64-for-auxiliary-repos.patch
+++ b/patches/fuel-library/0002-Add-arm64-for-auxiliary-repos.patch
@@ -34,6 +34,3 @@ index 720943d..91adb3e 100644
  
    Exec['create_ubuntu_repo_dirs'] ->
      Exec['create_ubuntu_repo_Packages'] ->
--- 
-1.9.1
-
diff --git a/patches/fuel-library/0003-Make-qemu-kvm-architecture-aware.patch b/patches/fuel-library/0003-Make-qemu-kvm-architecture-aware.patch
index 00cd3c7a..01cbdf83 100644
--- a/patches/fuel-library/0003-Make-qemu-kvm-architecture-aware.patch
+++ b/patches/fuel-library/0003-Make-qemu-kvm-architecture-aware.patch
@@ -7,7 +7,7 @@ Subject: [PATCH] Make qemu-kvm architecture aware
  1 file changed, 7 insertions(+), 2 deletions(-)
 
 diff --git a/deployment/puppet/openstack/manifests/compute.pp b/deployment/puppet/openstack/manifests/compute.pp
-index b2339bc..4f380b8 100644
+index b2339bc..ca1b2c4 100644
 --- a/deployment/puppet/openstack/manifests/compute.pp
 +++ b/deployment/puppet/openstack/manifests/compute.pp
 @@ -169,10 +169,15 @@ class openstack::compute (
@@ -16,8 +16,8 @@ index b2339bc..4f380b8 100644
  
 +      # Guard against some exotic distros with their `uname -m`
 +      $arch = $::architecture ? {
-+        'arm64', 'aarch64' => 'aarch64',
-+        default => 'x86_64',
++        /(arm64|aarch64)/ => 'aarch64',
++        default           => 'x86_64',
 +      }
        # From legacy libvirt.pp
        exec { 'symlink-qemu-kvm':
@@ -28,6 +28,3 @@ index b2339bc..4f380b8 100644
        }
  
        package { 'avahi':
--- 
-1.9.1
-
diff --git a/patches/fuel-library/0004-Make-TestVM-creation-architecture-aware.patch b/patches/fuel-library/0004-Make-TestVM-creation-architecture-aware.patch
index 0ea66f66..b9a8ed10 100644
--- a/patches/fuel-library/0004-Make-TestVM-creation-architecture-aware.patch
+++ b/patches/fuel-library/0004-Make-TestVM-creation-architecture-aware.patch
@@ -7,12 +7,12 @@ images.
 ---
  deployment/puppet/openstack/manifests/img/cirros.pp           |  8 ++++++--
  .../puppet/osnailyfacter/modular/generate_vms/generate_vms.pp | 11 +++++++++++
- deployment/puppet/osnailyfacter/templates/vm_libvirt.erb      |  5 ++++-
+ deployment/puppet/osnailyfacter/templates/vm_libvirt.erb      |  9 ++++++++-
  files/fuel-migrate/fuel-migrate                               |  5 ++++-
- 4 files changed, 25 insertions(+), 4 deletions(-)
+ 4 files changed, 29 insertions(+), 4 deletions(-)
 
 diff --git a/deployment/puppet/openstack/manifests/img/cirros.pp b/deployment/puppet/openstack/manifests/img/cirros.pp
-index 84e202e..57ae256 100644
+index 84e202e..cff06e0 100644
 --- a/deployment/puppet/openstack/manifests/img/cirros.pp
 +++ b/deployment/puppet/openstack/manifests/img/cirros.pp
 @@ -9,6 +9,10 @@ class openstack::img::cirros (
@@ -20,8 +20,8 @@ index 84e202e..57ae256 100644
    $os_name = 'cirros',
  ) {
 +  $arch = $::architecture ? {
-+      'arm64', 'aarch64' => 'aarch64',
-+      default => 'x86_64',
++      /(arm64|aarch64)/ => 'aarch64',
++      default           => 'x86_64',
 +  }
  
    package { 'cirros-testvm':
@@ -66,7 +66,7 @@ index 6d5426a..da5954e 100644
      owner   => 'root',
      group   => 'root',
 diff --git a/deployment/puppet/osnailyfacter/templates/vm_libvirt.erb b/deployment/puppet/osnailyfacter/templates/vm_libvirt.erb
-index ae2ba5b..decf3ab 100644
+index ae2ba5b..2d030d9 100644
 --- a/deployment/puppet/osnailyfacter/templates/vm_libvirt.erb
 +++ b/deployment/puppet/osnailyfacter/templates/vm_libvirt.erb
 @@ -3,7 +3,7 @@
@@ -82,12 +82,24 @@ index ae2ba5b..decf3ab 100644
      <acpi/>
      <apic/>
      <pae/>
-+  <% if $machine_arch == 'arm64' { %>
-+    <gic version='3' />
-+  <% } %>
++  <% if $machine_arch == 'aarch64' %>
++    <gic version='host' />
++  <% end %>
    </features>
    <clock offset='utc'/>
    <on_poweroff>destroy</on_poweroff>
+@@ -57,7 +60,11 @@
+     <input type='keyboard' bus='ps2'/>
+     <graphics type='vnc' port='-1' autoport='yes'/>
+     <video>
++<% if $machine_arch == 'aarch64' %>
++      <model type='vga' vram='9216' heads='1'/>
++<% else %>
+       <model type='cirrus' vram='9216' heads='1'/>
++<% end %>
+     </video>
+     <memballoon model='virtio'>
+     </memballoon>
 diff --git a/files/fuel-migrate/fuel-migrate b/files/fuel-migrate/fuel-migrate
 index fad9436..667d4a7 100755
 --- a/files/fuel-migrate/fuel-migrate
@@ -117,6 +129,3 @@ index fad9436..667d4a7 100755
    </features>
    <clock offset='utc'/>
    <on_poweroff>destroy</on_poweroff>
--- 
-1.9.1
-
diff --git a/patches/fuel-library/0005-Disable-usb-tablet-for-aarch64.patch b/patches/fuel-library/0005-Disable-usb-tablet-for-aarch64.patch
new file mode 100644
index 00000000..37b42341
--- /dev/null
+++ b/patches/fuel-library/0005-Disable-usb-tablet-for-aarch64.patch
@@ -0,0 +1,38 @@
+From: Stanislaw Kardach <stanislaw.kardach@caviumnetworks.com>
+Date: Tue, 15 Mar 2016 11:33:52 +0100
+Subject: [PATCH] Disable usb tablet for aarch64
+
+---
+ deployment/puppet/openstack/manifests/compute.pp | 12 ++++++++++++
+ 1 file changed, 12 insertions(+)
+
+diff --git a/deployment/puppet/openstack/manifests/compute.pp b/deployment/puppet/openstack/manifests/compute.pp
+index ca1b2c4..c0349dc 100644
+--- a/deployment/puppet/openstack/manifests/compute.pp
++++ b/deployment/puppet/openstack/manifests/compute.pp
+@@ -316,6 +316,10 @@ class openstack::compute (
+   nova_config {
+     'libvirt/live_migration_flag':  value => 'VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST';
+     'libvirt/block_migration_flag': value => 'VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_NON_SHARED_INC';
++    'libvirt/use_usb_tablet': value => $::architecture ? {
++        /(arm64|aarch64)/ => false,
++        default           => true,
++    }
+   }
+ 
+   nova_config {
+@@ -364,6 +368,14 @@ class openstack::compute (
+     # Workaround for bug LP #1469308
+     # also service name for Ubuntu and Centos is the same.
+     libvirt_service_name     => "libvirtd",
++  } ->
++  package { 'vgabios':
++    ensure => present;
++  } ->
++  file { '/usr/share/qemu/vgabios-stdvga.bin':
++    ensure  => link,
++    target  => '/usr/share/vgabios/vgabios.bin',
++    replace => false,
+   }
+ 
+   # From legacy libvirt.pp
diff --git a/patches/fuel-library/0006-direct-kernel-boot-for-cirros.patch b/patches/fuel-library/0006-direct-kernel-boot-for-cirros.patch
new file mode 100644
index 00000000..0c110430
--- /dev/null
+++ b/patches/fuel-library/0006-direct-kernel-boot-for-cirros.patch
@@ -0,0 +1,138 @@
+From: Stanislaw Kardach <stanislaw.kardach@caviumnetworks.com>
+Date: Tue, 15 Mar 2016 15:01:34 +0100
+Subject: [PATCH] direct kernel boot for cirros
+
+---
+ .../osnailyfacter/modular/astute/upload_cirros.rb  | 58 +++++++++++++++++++++-
+ .../puppet/osnailyfacter/templates/vm_libvirt.erb  |  8 ++-
+ 2 files changed, 63 insertions(+), 3 deletions(-)
+
+diff --git a/deployment/puppet/osnailyfacter/modular/astute/upload_cirros.rb b/deployment/puppet/osnailyfacter/modular/astute/upload_cirros.rb
+index 04d597c..c342830 100755
+--- a/deployment/puppet/osnailyfacter/modular/astute/upload_cirros.rb
++++ b/deployment/puppet/osnailyfacter/modular/astute/upload_cirros.rb
+@@ -1,6 +1,7 @@
+ #!/usr/bin/env ruby
+ require 'hiera'
+ 
++
+ ENV['LANG'] = 'C'
+ 
+ hiera = Hiera.new(:config => '/etc/hiera.yaml')
+@@ -56,11 +57,17 @@ def image_list
+     fields = line.split('|').map { |f| f.chomp.strip }
+     next if fields[1] == 'ID'
+     next unless fields[2]
+-    images << fields[2]
++    images << [fields[1], fields[2]]
+   end
+   {:images => images, :exit_code => return_code}
+ end
+ 
++def image_name_list
++  images = image_list
++  images[:images].map! { |i| i[1] }
++  images
++end
++
+ # TODO degorenko: remove --os-image-api-version after liberty (fuel-8.0) release
+ def image_create(image_hash)
+   command = <<-EOF
+@@ -79,6 +86,15 @@ EOF
+   [ stdout, return_code ]
+ end
+ 
++# Calls glance update-image with a given property and value
++def update_image(image_id, property, value)
++  command = "/usr/bin/glance image-update --#{property} #{value} #{image_id}"
++  puts command
++  stdout = `#{command}`
++  return_code = $?.exitstatus
++  [ stdout, return_code ]
++end
++
+ # check if Glance is online
+ # waited until the glance is started because when vCenter used as a glance
+ # backend launch may takes up to 1 minute.
+@@ -93,7 +109,7 @@ end
+ # upload image to Glance
+ # if it have not been already uploaded
+ def upload_image(image)
+-  list_of_images = image_list
++  list_of_images = image_name_list
+   if list_of_images[:images].include?(image['img_name']) && list_of_images[:exit_code] == 0
+     puts "Image '#{image['img_name']}' is already present!"
+     return 0
+@@ -109,6 +125,43 @@ def upload_image(image)
+   return return_code
+ end
+ 
++# For each disk image try to find a kernel and initramfs images and
++# attach then to it via kernel_id and ramdisk_id glance properties.
++def connect_dependant_images(images)
++  # for each image
++  # get image id from glance
++  img_list = image_list
++  return_code = img_list[:exit_code]
++  if return_code == 0
++    images.each do |image|
++      img_list[:images].each do |il|
++        if il[1] == image['img_name']
++          image['id'] = il[0]
++        end
++      end
++    end
++    # for each image that is not in [aki, ari]
++    images.each do |image|
++      next if ['aki', 'ari'].include?(image['disk_format'])
++      images.each do |i|
++        # find aki/ari image whose name starts with this image's name
++        if i['img_name'].start_with?(image['img_name'])
++          ret = 0
++          if i['disk_format'] == 'aki'
++            _, ret = update_image(image['id'], 'property',
++                                        "kernel_id=#{i['id']}")
++          elsif i['disk_format'] == 'ari'
++            _, ret = update_image(image['id'], 'property',
++                                        "ramdisk_id=#{i['id']}")
++          end
++          return_code += ret
++        end
++      end
++    end
++  end
++  return return_code
++end
++
+ ########################
+ 
+ wait_for_glance
+@@ -117,6 +170,7 @@ errors = 0
+ test_vm_images.each do |image|
+   errors += upload_image(image)
+ end
++errors = connect_dependant_images(test_vm_images) unless errors != 0
+ 
+ exit 1 unless errors == 0
+ 
+diff --git a/deployment/puppet/osnailyfacter/templates/vm_libvirt.erb b/deployment/puppet/osnailyfacter/templates/vm_libvirt.erb
+index 88b2c73..65e9400 100644
+--- a/deployment/puppet/osnailyfacter/templates/vm_libvirt.erb
++++ b/deployment/puppet/osnailyfacter/templates/vm_libvirt.erb
+@@ -3,9 +3,15 @@
+   <memory unit='GiB'><%= @details['mem'] %></memory>
+   <vcpu placement='static'><%= @details['cpu'] %></vcpu>
+   <os>
+-    <type arch='<%= @machine_arch %>' machine='<%= @machine_type %>'>hvm</type>
++  <type arch='<%= @machine_arch %>' machine='<%= @machine_type %>'>hvm</type>
++  <% if $machine_arch == 'aarch64' %>
++    <kernel>/var/lib/nova/<%= @details['id'] %>_vm.kernel</kernel>
++    <initrd>/var/lib/nova/<%= @details['id'] %>_vm.initramfs</initrd>
++    <cmdline>root=/dev/vda1 rw rootwait console=tty0 console=ttyS0 console=ttyAMA0</cmdline>
++  <% else %>
+     <boot dev='network'/>
+     <boot dev='hd'/>
++  <% end %>
+   </os>
+   <features>
+     <acpi/>
diff --git a/patches/fuel-library/0007-m1.micro-Increase-profile-RAM-size-to-128MB.patch b/patches/fuel-library/0007-m1.micro-Increase-profile-RAM-size-to-128MB.patch
new file mode 100644
index 00000000..8253acad
--- /dev/null
+++ b/patches/fuel-library/0007-m1.micro-Increase-profile-RAM-size-to-128MB.patch
@@ -0,0 +1,37 @@
+From: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+Date: Sun, 20 Mar 2016 18:24:04 +0100
+Subject: [PATCH] m1.micro: Increase profile RAM size to 128MB.
+
+TestVM (cirros) on aarch64 requires more than 64MB RAM.
+Keep profiles uniform across all archs.
+---
+ .../osnailyfacter/modular/openstack-controller/openstack-controller.pp  | 2 +-
+ tests/noop/spec/hosts/openstack-controller/openstack-controller_spec.rb | 2 +-
+ 2 files changed, 2 insertions(+), 2 deletions(-)
+
+diff --git a/deployment/puppet/osnailyfacter/modular/openstack-controller/openstack-controller.pp b/deployment/puppet/osnailyfacter/modular/openstack-controller/openstack-controller.pp
+index b1d8795..88e0ab3 100644
+--- a/deployment/puppet/osnailyfacter/modular/openstack-controller/openstack-controller.pp
++++ b/deployment/puppet/osnailyfacter/modular/openstack-controller/openstack-controller.pp
+@@ -233,7 +233,7 @@ if $primary_controller {
+       "OS_REGION_NAME=${region}",
+       "NOVA_ENDPOINT_TYPE=internalURL",
+     ],
+-    command   => 'bash -c "nova flavor-create --is-public true m1.micro auto 64 0 1"',
++    command   => 'bash -c "nova flavor-create --is-public true m1.micro auto 128 0 1"',
+     #FIXME(mattymo): Upstream bug PUP-2299 for retries in unless/onlyif
+     # Retry nova-flavor list until it exits 0, then exit with grep status,
+     # finally exit 1 if tries exceeded
+diff --git a/tests/noop/spec/hosts/openstack-controller/openstack-controller_spec.rb b/tests/noop/spec/hosts/openstack-controller/openstack-controller_spec.rb
+index 36161e6..b15da07 100644
+--- a/tests/noop/spec/hosts/openstack-controller/openstack-controller_spec.rb
++++ b/tests/noop/spec/hosts/openstack-controller/openstack-controller_spec.rb
+@@ -171,7 +171,7 @@ describe manifest do
+     if primary_controller
+       it 'should retry unless when creating m1.micro flavor' do
+         should contain_exec('create-m1.micro-flavor').with(
+-           'command' => 'bash -c "nova flavor-create --is-public true m1.micro auto 64 0 1"',
++           'command' => 'bash -c "nova flavor-create --is-public true m1.micro auto 128 0 1"',
+            'unless'  => 'bash -c \'for tries in {1..10}; do
+                     nova flavor-list | grep m1.micro;
+                     status=("${PIPESTATUS[@]}");