fuel-plugin-opendaylight: Prebuilt leveldbjni DEB

Switch from deploy time building of leveldb + leveldbjni to
using a prebuilt artifact, shipped as a DEB package from Armband
repos.

JIRA: ARMBAND-114

Change-Id: Ic7a52c78e803c1a6ebda1f127774959a0880a5a3
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>

3 files changed, 52 insertions, 254 deletions

diff --git a/patches/fuel-mirror/0002-Add-fuel-agent-dependencies-to-Ubuntu-mirror.patch b/patches/fuel-mirror/0002-Add-fuel-agent-dependencies-to-Ubuntu-mirror.patch
index 5df52fa1..d18c90a7 100644
--- a/patches/fuel-mirror/0002-Add-fuel-agent-dependencies-to-Ubuntu-mirror.patch
+++ b/patches/fuel-mirror/0002-Add-fuel-agent-dependencies-to-Ubuntu-mirror.patch
@@ -17,16 +17,26 @@ fixes persistent device names in the udev it ships out of the box.
 FIXME: Remove lshw package from this patch when Ubuntu Trusty on arm64
 fixes bug [1]: "AArch64: slow cpuinfo due to redundant loop".
 
+Added armband-odl-leveldb-fix for patching ODL's leveldbjni without
+building it for each deploy.
+
+JIRA: ARMBAND-114
+
 [1] https://bugs.launchpad.net/ubuntu/+source/lshw/+bug/1582181
 ---
- contrib/fuel_mirror/data/ubuntu.yaml | 4 ++++
- 1 file changed, 4 insertions(+)
+ contrib/fuel_mirror/data/ubuntu.yaml | 5 +++++
+ 1 file changed, 5 insertions(+)
 
 diff --git a/contrib/fuel_mirror/data/ubuntu.yaml b/contrib/fuel_mirror/data/ubuntu.yaml
-index cd86ec9..2ec4f1d 100644
+index cd86ec9..18fb1ee 100644
 --- a/contrib/fuel_mirror/data/ubuntu.yaml
 +++ b/contrib/fuel_mirror/data/ubuntu.yaml
-@@ -77,6 +77,7 @@ packages: &packages
+@@ -73,10 +73,12 @@ packages: &packages
+      - "acpi-support"
+      - "anacron"
+      - "aptitude"
++     - "armband-odl-leveldb-fix"
+      - "atop"
       - "acct"
       - "bash-completion"
       - "bc"
@@ -34,7 +44,7 @@ index cd86ec9..2ec4f1d 100644
       - "build-essential"
       - "cloud-init"
       - "conntrackd"
-@@ -101,6 +102,7 @@ packages: &packages
+@@ -101,6 +103,7 @@ packages: &packages
       - "linux-image-generic-lts-trusty"
       - "live-boot"
       - "livecd-rootfs"
@@ -42,7 +52,7 @@ index cd86ec9..2ec4f1d 100644
       - "mc"
       - "memcached"
       - "monit"
-@@ -116,6 +118,7 @@ packages: &packages
+@@ -116,6 +119,7 @@ packages: &packages
       - "puppet"
       - "python-lesscpy"
       - "python-pip"
@@ -50,7 +60,7 @@ index cd86ec9..2ec4f1d 100644
       - "rsyslog-gnutls"
       - "rsyslog-relp"
       - "screen"
-@@ -127,6 +130,7 @@ packages: &packages
+@@ -127,6 +131,7 @@ packages: &packages
       - "tmux"
       - "traceroute"
       - "ubuntu-standard"
diff --git a/patches/fuel-plugin-opendaylight/0001-ODL-leveldb-leveldbjni-Enable-arm64-support.patch b/patches/fuel-plugin-opendaylight/0001-ODL-leveldb-leveldbjni-Enable-arm64-support.patch
index f1a1d412..5cfa99b5 100644
--- a/patches/fuel-plugin-opendaylight/0001-ODL-leveldb-leveldbjni-Enable-arm64-support.patch
+++ b/patches/fuel-plugin-opendaylight/0001-ODL-leveldb-leveldbjni-Enable-arm64-support.patch
@@ -13,96 +13,23 @@ This is done when running odl-install puppet manifest.
 
 For more info about this ODL limitation, see [1].
 
-[1] https://bugs.opendaylight.org/show_bug.cgi?id=3973
-
 [ Alexandru.Avadanii@enea.com ]
 Remove JAVA8 changes after upstream rework.
+Convert puppet patching to prebuilt DEB package, as part of [2].
+
+[1] https://bugs.opendaylight.org/show_bug.cgi?id=3973
+[2] https://jira.opnfv.org/browse/ARMBAND-114
 
 Signed-off-by: Florin Dumitrascu <florin.dumitrascu@enea.com>
 Signed-off-by: Stanislaw Kardach <stanislaw.kardach@cavium.com>
 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
 ---
- .../modules/opendaylight/files/leveldb-arm64.patch |  35 +++++
- .../opendaylight/files/leveldbjni-native.patch     |  23 ++++
- .../puppet/modules/opendaylight/manifests/init.pp  |   8 ++
- .../modules/opendaylight/manifests/install.pp      |  12 +-
- .../modules/opendaylight/manifests/leveldbjni.pp   | 150 +++++++++++++++++++++
- deployment_tasks.yaml                              |   2 +-
- 6 files changed, 227 insertions(+), 3 deletions(-)
- create mode 100644 deployment_scripts/puppet/modules/opendaylight/files/leveldb-arm64.patch
- create mode 100644 deployment_scripts/puppet/modules/opendaylight/files/leveldbjni-native.patch
+ .../puppet/modules/opendaylight/manifests/init.pp  |  8 ++++++++
+ .../modules/opendaylight/manifests/install.pp      | 12 +++++++++--
+ .../modules/opendaylight/manifests/leveldbjni.pp   | 24 ++++++++++++++++++++++
+ 3 files changed, 42 insertions(+), 2 deletions(-)
  create mode 100644 deployment_scripts/puppet/modules/opendaylight/manifests/leveldbjni.pp
 
-diff --git a/deployment_scripts/puppet/modules/opendaylight/files/leveldb-arm64.patch b/deployment_scripts/puppet/modules/opendaylight/files/leveldb-arm64.patch
-new file mode 100644
-index 0000000..e0824ca
---- /dev/null
-+++ b/deployment_scripts/puppet/modules/opendaylight/files/leveldb-arm64.patch
-@@ -0,0 +1,35 @@
-+diff --git a/port/atomic_pointer.h b/port/atomic_pointer.h
-+index e17bf43..78cb6b3 100644
-+--- a/port/atomic_pointer.h
-++++ b/port/atomic_pointer.h
-+@@ -36,6 +36,8 @@
-+ #define ARCH_CPU_X86_FAMILY 1
-+ #elif defined(__ARMEL__)
-+ #define ARCH_CPU_ARM_FAMILY 1
-++#elif defined(__aarch64__)
-++#define ARCH_CPU_ARM64_FAMILY 1
-+ #elif defined(__ppc__) || defined(__powerpc__) || defined(__powerpc64__)
-+ #define ARCH_CPU_PPC_FAMILY 1
-+ #endif
-+@@ -93,6 +95,13 @@ inline void MemoryBarrier() {
-+ }
-+ #define LEVELDB_HAVE_MEMORY_BARRIER
-+
-++// ARM64
-++#elif defined(ARCH_CPU_ARM64_FAMILY)
-++inline void MemoryBarrier() {
-++ asm volatile("dmb sy" : : : "memory");
-++}
-++#define LEVELDB_HAVE_MEMORY_BARRIER
-++
-+ // PPC
-+ #elif defined(ARCH_CPU_PPC_FAMILY) && defined(__GNUC__)
-+ inline void MemoryBarrier() {
-+@@ -216,6 +225,7 @@ class AtomicPointer {
-+ #undef LEVELDB_HAVE_MEMORY_BARRIER
-+ #undef ARCH_CPU_X86_FAMILY
-+ #undef ARCH_CPU_ARM_FAMILY
-++#undef ARCH_CPU_ARM64_FAMILY
-+ #undef ARCH_CPU_PPC_FAMILY
-+
-+ }  // namespace port
-diff --git a/deployment_scripts/puppet/modules/opendaylight/files/leveldbjni-native.patch b/deployment_scripts/puppet/modules/opendaylight/files/leveldbjni-native.patch
-new file mode 100644
-index 0000000..08a528b
---- /dev/null
-+++ b/deployment_scripts/puppet/modules/opendaylight/files/leveldbjni-native.patch
-@@ -0,0 +1,23 @@
-+diff --git a/leveldbjni-all/pom.xml b/leveldbjni-all/pom.xml
-+index 426f2a6..c4a2a16 100755
-+--- a/leveldbjni-all/pom.xml
-++++ b/leveldbjni-all/pom.xml
-+@@ -41,7 +41,7 @@
-+
-+   <groupId>org.fusesource.leveldbjni</groupId>
-+   <artifactId>leveldbjni-all</artifactId>
-+-  <version>1.8</version>
-++  <version>1.8-odl</version>
-+   <packaging>bundle</packaging>
-+
-+   <name>${project.artifactId}</name>
-+@@ -119,7 +119,8 @@
-+               META-INF/native/osx/libleveldbjni.jnilib;osname=macosx;processor=x86,
-+               META-INF/native/osx/libleveldbjni.jnilib;osname=macosx;processor=x86-64,
-+               META-INF/native/linux32/libleveldbjni.so;osname=Linux;processor=x86,
-+-              META-INF/native/linux64/libleveldbjni.so;osname=Linux;processor=x86-64
-++              META-INF/native/linux64/libleveldbjni.so;osname=Linux;processor=x86-64,
-++              META-INF/native/linux64/libleveldbjni.so;osname=Linux;processor=aarch64
-+             </Bundle-NativeCode>
-+          </instructions>
-+         </configuration>
 diff --git a/deployment_scripts/puppet/modules/opendaylight/manifests/init.pp b/deployment_scripts/puppet/modules/opendaylight/manifests/init.pp
 index 69cc23a..791ceba 100644
 --- a/deployment_scripts/puppet/modules/opendaylight/manifests/init.pp
@@ -158,22 +85,22 @@ index 1962609..ca98532 100644
  }
 diff --git a/deployment_scripts/puppet/modules/opendaylight/manifests/leveldbjni.pp b/deployment_scripts/puppet/modules/opendaylight/manifests/leveldbjni.pp
 new file mode 100644
-index 0000000..5125fc7
+index 0000000..ed97c5d
 --- /dev/null
 +++ b/deployment_scripts/puppet/modules/opendaylight/manifests/leveldbjni.pp
-@@ -0,0 +1,150 @@
+@@ -0,0 +1,25 @@
 +# == Class opendaylight::leveldbjni
 +#
-+# Manages compilation of JNI for leveldb.
++# Manages patching of JNI for leveldb on arm64.
++#
++# It fetches a prebuilt DEB package containing leveldbjni,
++# patched so that it's properly detected by maven on arm64 systems.
 +#
-+# It compiles the leveldbjni from sources and patches it so that it's properly
-+# detected by maven on arm64 system.
++# DEB: http://linux.enea.com/mos-repos/ubuntu/9.0/ \
++#      pool/main/a/armband-odl-leveldb-fix/
++# JIRA: https://jira.opnfv.org/browse/ARMBAND-114
 +#
-+class opendaylight::leveldbjni(
-+  $target = '/opt/opendaylight',
-+  $user    = 'odl',
-+  $branch  = 'leveldbjni-1.8',
-+) {
++class opendaylight::leveldbjni {
 +  case $::osfamily {
 +    'Debian', 'Ubuntu': {}
 +    default: {
@@ -181,147 +108,9 @@ index 0000000..5125fc7
 +    }
 +  }
 +
-+  $libsnappy_dir = '/usr/lib'
-+  $leveldb_dir = "${target}/leveldb"
-+  $leveldbjni_dir = "${target}/leveldbjni"
-+  $environment = ["SNAPPY_HOME=${libsnappy_dir}", "LEVELDB_HOME=${leveldb_dir}", "LEVELDBJNI_HOME=${leveldbjni_dir}", "LIBRARY_PATH=${libsnappy_dir}", "C_INCLUDE_PATH=${libsnappy_dir}", "CPLUS_INCLUDE_PATH=${libsnappy_dir}","JAVA_HOME=/usr/lib/jvm/java-8-openjdk-${::opendaylight::arch}"]
-+  $parallel_jobs = ($processorcount + 1)/2
-+
-+  if ! defined(Package['git']) {
-+    package { 'git':
-+      ensure => 'present',
-+    }
-+  }
-+  if ! defined(Package['build-essential']) {
-+    package { 'build-essential':
-+      ensure => 'present',
-+    }
-+  }
-+  if ! defined(Package['automake']) {
-+    package { 'automake':
-+      ensure => 'present',
-+    }
-+  }
-+  if ! defined(Package['make']) {
-+    package { 'make':
-+      ensure => 'present',
-+    }
-+  }
-+  if ! defined(Package['libsnappy-dev']) {
-+    package { 'libsnappy-dev':
-+      ensure => 'present',
-+    }
-+  }
-+  if ! defined(Package['patch']) {
-+    package { 'patch':
++  if ! defined(Package['armband-odl-leveldb-fix']) {
++    package { 'armband-odl-leveldb-fix':
 +      ensure => 'present',
 +    }
 +  }
-+  if ! defined(Package['maven']) {
-+    package { 'maven':
-+      ensure => 'present',
-+    }
-+  }
-+  if ! defined(Package['openjdk-8-jdk']) {
-+    package { 'openjdk-8-jdk':
-+      ensure => 'present',
-+    }
-+  }
-+
-+  exec { 'leveldb-fetch':
-+    path    => ['/usr/bin'],
-+    command => "git clone git://github.com/chirino/leveldb.git ${leveldb_dir}",
-+    creates => "${leveldb_dir}/.git",
-+    user    => $user,
-+    require => [Package['git']],
-+  } ->
-+  exec { 'leveldbjni-fetch':
-+    path    => ['/usr/bin'],
-+    command => "git clone https://github.com/fusesource/leveldbjni.git -b ${branch} ${leveldbjni_dir}",
-+    creates => "${leveldbjni_dir}/.git",
-+    user    => $user,
-+    require => [Package['git']],
-+  } ->
-+  exec { 'leveldb-merge-google':
-+    path    => ['/usr/bin', '/bin'],
-+    command => "git config user.name odl && git config user.email 'o@d.l' && git remote add google https://github.com/google/leveldb.git && git fetch google master && git fetch --tags google && git merge --no-edit v1.18",
-+    unless  => "git remote | grep google",
-+    cwd     => $leveldb_dir,
-+    user    => $user,
-+    require => [Package['git']],
-+  } ->
-+  exec { 'leveldb-patch':
-+    path    => ['/usr/bin'],
-+    command => "patch -p1 < ${leveldbjni_dir}/leveldb.patch",
-+    unless  => "patch -p1 -R -N --dry-run < ${leveldbjni_dir}/leveldb.patch",
-+    cwd     => $leveldb_dir,
-+    user    => $user,
-+    require => [Package['patch']],
-+  } ->
-+  file { "${leveldb_dir}/leveldb-arm64.patch":
-+    ensure => "file",
-+    source => "puppet:///modules/opendaylight/leveldb-arm64.patch",
-+    owner  => $user,
-+  } ->
-+  exec { 'leveldb-patch-for-arm64':
-+    path    => ['/usr/bin'],
-+    command => "patch -p1 < ${leveldb_dir}/leveldb-arm64.patch",
-+    unless  => "patch -p1 -R -N --dry-run < ${leveldb_dir}/leveldb-arm64.patch",
-+    cwd     => $leveldb_dir,
-+    user    => $user,
-+    require => [Package['patch']],
-+  } ->
-+  exec { 'leveldb-build':
-+    command => "/usr/bin/make libleveldb.a -j${parallel_jobs}",
-+    creates => "${leveldb_dir}/libleveldb.a",
-+    environment => $environment,
-+    cwd     => $leveldb_dir,
-+    user    => $user,
-+    require => [Package['git'], Package['make'], Package['build-essential']],
-+  } ->
-+  file { "${leveldbjni_dir}/leveldbjni.patch":
-+    ensure => "file",
-+    source => "puppet:///modules/opendaylight/leveldbjni-native.patch",
-+    owner  => $user,
-+  } ->
-+  exec { 'leveldbjni-patch':
-+    path    => ['/usr/bin'],
-+    command => "patch -p1 < ${leveldbjni_dir}/leveldbjni.patch",
-+    unless  => "patch -p1 -R -N --dry-run < ${leveldbjni_dir}/leveldbjni.patch",
-+    cwd     => $leveldbjni_dir,
-+    user    => $user,
-+    require => [Package['patch']],
-+  } ->
-+  exec { 'leveldbjni-build':
-+    command     => "/usr/bin/mvn clean install -P download -P linux64 -P all",
-+    # Creates requires a fully qualified path which we don't have
-+    unless      => "/usr/bin/test -d ~${user}/.m2/repository/org/fusesource/leveldbjni/leveldbjni-all",
-+    environment => $environment,
-+    cwd         => $leveldbjni_dir,
-+    user        => $user,
-+    timeout     => 1200,
-+    require     => [Package['maven']],
-+  } ->
-+  # Can't do this with 'file' type because we need to purge existing package
-+  # and at the same time don't perform the copy if we've already copied our
-+  # package.
-+  exec { 'leveldbjni-copy':
-+    path    => ['/bin'],
-+    command => "rm -rf ${target}/system/org/fusesource/leveldbjni && cp -r ~${user}/.m2/repository/org/fusesource/leveldbjni $target/system/org/fusesource/",
-+    creates => "${target}/system/org/fusesource/leveldbjni/leveldbjni-linux64",
-+    user    => $user,
-+  }
 +}
-diff --git a/deployment_tasks.yaml b/deployment_tasks.yaml
-index 20bd24f..e324ff5 100644
---- a/deployment_tasks.yaml
-+++ b/deployment_tasks.yaml
-@@ -20,7 +20,7 @@
-   parameters:
-     puppet_manifest: puppet/manifests/odl-install.pp
-     puppet_modules: puppet/modules:/etc/puppet/modules
--    timeout: 720
-+    timeout: 3600
-
- - id: netconfig
-   type: puppet
diff --git a/patches/opnfv-fuel/0000-f_odlpluginbuild-Armband-prereq-in-packages.yaml.patch b/patches/opnfv-fuel/0000-f_odlpluginbuild-Armband-prereq-in-packages.yaml.patch
index 3f98e1cd..8a42f66e 100644
--- a/patches/opnfv-fuel/0000-f_odlpluginbuild-Armband-prereq-in-packages.yaml.patch
+++ b/patches/opnfv-fuel/0000-f_odlpluginbuild-Armband-prereq-in-packages.yaml.patch
@@ -2,14 +2,19 @@ From: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
 Date: Sat, 19 Nov 2016 20:39:35 +0100
 Subject: [PATCH] f_odlpluginbuild: Armband prereq in packages.yaml
 
-We need to explicitily add our Armband leveldb patching prereq
-tools, like maven, git, make etc. to the ODL plugin's packages.yaml,
-so they are included in the offline Ubuntu mirror.
+We need to explicitily add our Armband leveldb patching prereq to
+the ODL plugin's packages.yaml, so they are included in the
+offline Ubuntu/MOS mirrors.
 
-While at it, also add openjdk-8-jre amos version additional dep:
-- libasound2
+After ARMBAND-114 was implemented, the dependency list is only:
+- armband-odl-leveldb-fix (for patching leveldbjni);
+- libasound2 (openjdk-8-jre amos version additional dep);
+
+Note: armband-odl-leveldb-fix MUST be also added to MOS mirror
+package list, since it's not present in Ubuntu repos.
 
 JIRA: ARMBAND-167
+JIRA: ARMBAND-114
 
 FIXME: These are only required for arm64, so maybe later we could
 filter out other archs by extending the packages.yaml with an <arch>
@@ -17,8 +22,8 @@ field.
 
 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
 ---
- build/f_isoroot/f_odlpluginbuild/packages.yaml | 11 +++++++++++
- 1 file changed, 11 insertions(+)
+ build/f_isoroot/f_odlpluginbuild/packages.yaml | 5 +++++
+ 1 file changed, 5 insertions(+)
  create mode 100644 build/f_isoroot/f_odlpluginbuild/packages.yaml
 
 diff --git a/build/f_isoroot/f_odlpluginbuild/packages.yaml b/build/f_isoroot/f_odlpluginbuild/packages.yaml
@@ -26,15 +31,9 @@ new file mode 100644
 index 0000000..b1bf086
 --- /dev/null
 +++ b/build/f_isoroot/f_odlpluginbuild/packages.yaml
-@@ -0,0 +1,11 @@
+@@ -0,0 +1,5 @@
 +# Armband: openjdk-8-jre depends on libasound2
-+# Armband: leveldb patching requires the other
++# Armband: leveldb patching requires armband-odl-leveldb-fix
 +packages:
-+  - git
-+  - build-essential
-+  - automake
-+  - make
-+  - libsnappy-dev
-+  - patch
-+  - maven
++  - armband-odl-leveldb-fix
 +  - libasound2