build: Use git submodules for Fuel git repos

This change reworks support for cloning (fetching) and patching all
Fuel components (fuel-library ... fuel-ui, see [1] for full list),
both outside the ISO build process (for development purposes, like
testing OPNFV patches apply cleanly to each component), and during
the ISO build.

The implementation relies heavily on git submodules and patches,
so backporting pending upstream gerrit changes, as well as adding new
out-of-upstream-tree patches, should be trivial.

Also, since without tracking remotes the repos rarely change, leverage
the in-place OPNFV build caching mechanism to drastically reduce the
number of git clones during each build.

The mechanism is detailed more in-depth in the JIRA ticket [3], and
it's merely a rehash of the submodule handling in Armband, implemented
initially by Stanislaw Kardach <stanislaw.kardach@cavium.com>.

Pros (+ new features):
 - OPNFV patches for Fuel components can be applied (imported) or
   developed (exported) outside of the ISO build process;
 - git repo caching reduces the number of external pulls;
 - reuse the same fuel-* componets in ALL targets
   e.g. fuel-mirror used to employ 2 slightly different git trees;
 - one step closer to Fuel/Armband merge (lots of steps left though);

Cons:
 - adds a lot complexity to solve a non-problem (handling the repos
   is fine in the current form, for the current goals);
   However, patching <fuel-astute> or <fuel-agent> seems to be
   quite complicated with the current mechanism, as env vars
   override our locally patched repos for these components).
   The proposed change should eliminate this issue completely.

CHANGE:
 Aligned divergent fuel-mirror in ISO vs local repo build by
 applying the 302 redirect fix ("Fixed handling http redirects") [2]
 on top of 9.0.1 tag in fuel-mirror repo, and using that for both.
 This obsoleted the followking mk file:
 - build/f_isoroot/f_repobuild/config.mk

CHANGE:
 Removed unused make target `get-fuel-repo` from build/config.mk.

FIXME:
 Remote tracking is currently deactivated for all branches.
 Change this when remote HEAD should be tracked instead,
 e.g. during Newton rebase.

[1] https://github.com/openstack/fuel-main/blob/stable/mitaka/repos.mk#L32
[2] https://review.openstack.org/#/c/334882/
[3] https://jira.opnfv.org/browse/FUEL-200

TODO: Extend build/f_repos/README.md with info about:
 - branch changes;
 - tag bumps;
 - patching now supports subdirs;

v5 -> v6:
 * Added support for nesting patches in subdirs, to be used for feature
   groups and related changes across submodules.
   E.g. Upcoming multiarch EFI support could go in:
    - .../f_repos/patch/fuel-agent/multiarch-efi/...;
    - .../f_repos/patch/fuel-astute/multiarch-efi/...;
   While Armband patches could go in:
    - .../f_repos/patch/fuel-agent/armband/...;
    - .../f_repos/patch/fuel-astute/armband/...;
   etc.
v6 -> v13:
 * fix wrong tag used for VERSION dump;
 * fix wrong tag used for patches-export;
 * move OPNFV_TAG to main config.mk and use it in FUEL_*_COMMIT;
 * only run `patches-import` when patches changed / not yet applied;
 * only run `clean` for stale `.cacheid` (avoid re-applying patches);
 * allow git submodule path/name to be different (FIXED);
 * put-cache should only run when artifact is not already cached;
 * `git am`: use 3-way merge so already applied patches are gently
   skipped (useful for upstreaming patches from Armband);
 * Properly support nested patch dirs;
 * Colorize the output a little;

JIRA: FUEL-200

Change-Id: I9dbdd9b3022896d4497d21be8dacc859730db489
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>

24 files changed, 630 insertions, 0 deletions

diff --git a/build/f_repos/Makefile b/build/f_repos/Makefile
new file mode 100644
index 000000000..66abf2d71
--- /dev/null
+++ b/build/f_repos/Makefile
@@ -0,0 +1,165 @@
+##############################################################################
+# Copyright (c) 2015,2016 Ericsson AB, Enea AB and others.
+# stefan.k.berg@ericsson.com
+# jonas.bjurel@ericsson.com
+# Alexandru.Avadanii@enea.com
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Apache License, Version 2.0
+# which accompanies this distribution, and is available at
+# http://www.apache.org/licenses/LICENSE-2.0
+##############################################################################
+
+SHELL = /bin/bash
+REPOINFO = $(shell readlink -e ../repo_info.sh)
+FREPODIR = $(shell pwd)
+FPATCHES = $(shell find ${F_PATCH_DIR} -name '*.patch')
+
+include ../config.mk
+
+# NOTE: Mechanism overview is presented in ./README.md.
+
+# Submodule consistent states:
+#   - NOT initialized (submodule trees are not populated at all);
+#   - initialized, bound to saved commits;
+#   - initialized, tracking remote origin (only for FUEL_TRACK_REMOTES);
+#   - patched (local patches are applied);
+
+# In order to keep things sort of separate, we should only pass up (to main
+# Makefile) the fully-pactched repos, and gather any fingerprinting info here.
+
+# Fuel@OPNFV relies on upstream git repos (one per component) in 1 of 2 ways:
+#   - pinned down to tag objects (e.g. "9.0.1")
+#   - tracking upstream remote HEAD on a stable or master branch
+# FIXME(alav): Should we support mixed cases? (e.g. pin down only fuel-main)
+
+# To enable remote tracking, set the following var to any non-empty string.
+# Leaving this var empty will bind each git submodule to its saved commit.
+# NOTE: For non-stable branches, unless overriden, always track remotes
+FUEL_TRACK_REMOTES ?= $(shell git symbolic-ref --short HEAD | grep -v stable)
+
+.PHONY: all
+all: release
+
+##############################################################################
+# git submodule operations - to be used stand-alone or from parent Makefile
+##############################################################################
+
+# Fetch & update git submodules, checkout remote HEAD or saved commit
+# Also gather fingerprints for parent gitinfo_fuel.txt.
+.PHONY: sub
+sub: .cachefuelinfo
+
+.cachefuelinfo:
+	@if [ -n "${FUEL_TRACK_REMOTES}" ]; then \
+		git submodule update --init --remote; \
+	else \
+		git submodule update --init; \
+	fi
+	@rm -f $@
+	@git submodule -q foreach '${REPOINFO} . >> ${FREPODIR}/$@'
+
+# Generate patches from submodules
+.PHONY: patches-export
+patches-export: sub
+	@git submodule -q foreach ' \
+		SUB_DIR=${F_PATCH_DIR}/$$name; \
+		git tag | awk "!/root/ && /${F_OPNFV_TAG}-fuel/" | while read F_TAG; do \
+			SUB_FEATURE=`dirname $${F_TAG#${F_OPNFV_TAG}-fuel/}`; \
+			echo "`tput setaf 2`-- exporting $$name ($$F_TAG)`tput sgr0`"; \
+			mkdir -p $$SUB_DIR/$${SUB_FEATURE} && \
+			git format-patch --no-signature --ignore-space-at-eol \
+				-o $$SUB_DIR/$$SUB_FEATURE -N $$F_TAG-root..$$F_TAG; \
+		done'
+	@sed -i -e '1d' -e 's/[[:space:]]*$$//' ${FPATCHES}
+
+# Apply patches from patch/* to respective submodules
+# We rely on `make sub` and/or `make clean` to checkout correct base
+.PHONY: patches-import
+patches-import: sub .cachepatched
+
+.cachepatched: ${FPATCHES}
+	@$(MAKE) clean
+	@git submodule -q foreach ' \
+		SUB_DIR=${F_PATCH_DIR}/$$name; mkdir -p $$SUB_DIR && \
+		git tag ${F_OPNFV_TAG}-root && \
+		git checkout -q -b opnfv-fuel && \
+		find $$SUB_DIR -type d | sort | while read p_dir; do \
+			SUB_PATCHES=$$(ls $$p_dir/*.patch 2>/dev/null); \
+			if [ -n "$$SUB_PATCHES" ]; then \
+				SUB_FEATURE=$${p_dir#$$SUB_DIR}; \
+				SUB_TAG=${F_OPNFV_TAG}-fuel$$SUB_FEATURE/patch; \
+				echo "`tput setaf 2`-- patching $$name ($$SUB_TAG)`tput sgr0`";\
+				git tag $$SUB_TAG-root && git am -3 --whitespace=nowarn \
+					--committer-date-is-author-date $$SUB_PATCHES && \
+				git tag $$SUB_TAG || exit 1; \
+			fi \
+		done && \
+		git tag ${F_OPNFV_TAG}'
+	@touch $@
+
+# Clean any changes made to submodules, checkout upstream Fuel root commit
+.PHONY: clean
+clean:
+	@git submodule -q foreach ' \
+		git am -q --abort > /dev/null 2>&1; \
+		git checkout -q ${F_OPNFV_TAG}-root > /dev/null 2>&1; \
+		git branch -q -D opnfv-fuel > /dev/null 2>&1; \
+		git tag | grep ${F_OPNFV_TAG} | xargs git tag -d > /dev/null 2>&1; \
+		git reset -q --hard HEAD; \
+		git clean -xdff'
+	@rm -f .cachepatched
+
+.PHONY: deepclean
+deepclean: clean clean-cache
+	@git submodule deinit -f .
+	@rm -f .cache*
+
+.PHONY: release
+release: sub
+	# Store artifact in cache straight away if caching is enabled
+	# (no .cacheid will be present unless this is a cached build)
+	test -f .cacheid && $(MAKE) -f Makefile put-cache || exit 0
+	# NOTE: Patches are not included in cache
+	$(MAKE) -f Makefile patches-import
+
+##############################################################################
+# Cache operations - only used when building through ci/build.sh
+##############################################################################
+
+# NOTE: Current method of collecting submodule refs requires submodules to be
+#       NOT initialized <OR> NO patches applied.
+# NOTE: Querying `git submodule status` from parent will show the patched tree.
+# Create a unique hash to be used for getting and putting cache, based on:
+#   - git submodule SHAs, collected with `git submodule status`
+#   - The contents of this Makefile
+.cacheid:
+	@$(MAKE) clean
+	sha1sum Makefile > .cachedata
+	git submodule status | cut -c2-41 >> .cachedata
+	cat .cachedata | $(CACHETOOL) getid > .cacheid
+
+# Clean local data related to caching - called prior to ordinary build
+.PHONY: clean-cache
+clean-cache: clean
+	rm -f .cachedata .cacheid
+
+# Try to download cache - called prior to ordinary build
+.PHONY: get-cache
+get-cache: .cacheid
+	@if $(CACHETOOL) check $(shell cat .cacheid); then \
+		$(MAKE) clean && \
+		$(CACHETOOL) get $(shell cat .cacheid) | \
+			tar xf - -C ${F_GIT_ROOT}/.git; \
+	else \
+		echo "No cache item found for $(shell cat .cacheid)" ;\
+		exit 0;\
+	fi
+
+# Store cache if not already stored - called after ordinary build
+.PHONY: put-cache
+put-cache: .cacheid
+	@if ! $(CACHETOOL) check $(shell cat .cacheid); then \
+		$(MAKE) clean && \
+		tar cf - -C ${F_GIT_ROOT}/.git modules | \
+			$(CACHETOOL) put $(shell cat .cacheid); \
+	fi
diff --git a/build/f_repos/README.md b/build/f_repos/README.md
new file mode 100644
index 000000000..6cc825e05
--- /dev/null
+++ b/build/f_repos/README.md
@@ -0,0 +1,94 @@
+Fuel@OPNFV submodule fetching and patching
+==========================================
+
+This directory holds submodule fetching/patching scripts, intended for
+working with upstream Fuel components (fuel-library, ... , fuel-ui) in
+developing/applying OPNFV patches (backports, custom fixes etc.).
+
+The scripts should be friendly to the following 2 use-cases:
+  - development work: easily cloning, binding repos to specific commits,
+    remote tracking, patch development etc.;
+  - to provide parent build scripts an easy method of tracking upstream
+    references and applying OPNFV patches on top;
+
+Also, we need to support at least the following modes of operations:
+  - submodule bind - each submodule patches will be based on the commit ID
+    saved in the .gitmodules config file;
+  - remote tracking - each submodule will sync with the upstream remote
+    and patches will be applied on top of <sub_remote>/<sub_branch>/HEAD;
+
+Workflow (development)
+----------------------
+The standard development workflow should look as follows:
+
+1. Decide whether remote tracking should be active or not:
+   NOTE: Setting the following var to any non-empty str enables remote track.
+   NOTE: Leaving unset will enable remote track for anything but stable branch.
+
+   $ export FUEL_TRACK_REMOTES=""
+
+2. All Fuel sub-projects are registered as submodules. To initialize them, call:
+   If remote tracking is active, upstream remote is queried and latest remote
+   branch HEAD is fetched. Otherwise, checkout commit IDs from .gitmodules.
+
+   $ make sub
+
+3. Apply patches from `patches/<sub-project>/*` to respective submodules via:
+
+   $ make patches-import
+
+   This will result in creation of:
+   - a tag called `${FUEL_MAIN_TAG}-opnfv-root` at the same commit as Fuel@OPNFV
+     upstream reference (bound to git submodule OR tracking remote HEAD);
+   - a new branch `opnfv-fuel` which will hold all the OPNFV patches,
+     each patch is applied on this new branch with `git-am`;
+   - a tag called `${FUEL_MAIN_TAG}-opnfv` at `opnfv-fuel/HEAD`;
+
+4. Modify sub-projects for whatever you need.
+   Commit your changes when you want them taken into account in the build.
+
+5. Re-create patches via:
+
+   $ make patches-export
+
+   Each commit on `opnfv-fuel` branch of each subproject will be
+   exported to `patches/subproject/` via `git format-patch`.
+
+   NOTE: Only commit (-f) submodules when you need to bump upstream ref.
+   NOTE: DO NOT commit patched submodules!
+
+6. Clean workbench branches and tags with:
+
+   $ make clean
+
+7. De-initialize submodules and force a clean clone with:
+
+   $ make deepclean
+
+Workflow (ISO build)
+--------------------
+Parent build scripts require this mechanism to do some fingerprinting,
+so here is the intended flow for all artifacts to be generated right:
+
+1. (Optional) Cached submodules might be fetched from build cache.
+
+2. Submodules are updated
+   We also dump each submodule's git info using repo_info.sh, since
+   we want to collect git refs before patching (i.e. upstream refs).
+
+3. Make target `release` is built
+   This will make sure the modules are in a clean state, put them in cache,
+   then apply the patches.
+
+4. fuel-main's `${FUEL_MAIN_TAG}-opnfv-root` tag is used to determine VERSION info
+   It will accommodate both bound tags and remote tracking references.
+
+Sub-project maintenance
+-----------------------
+1. Adding a new submodule
+   If you need to add another subproject, you can do it with `git submodule`.
+   Make sure that you specify branch (with `-b`), short name (with `--name`)
+   and point it to `upstream/*` directory, i.e.:
+
+   $ git submodule -b stable/mitaka add --name fuel-web \
+     https://github.com/openstack/fuel-web.git upstream/fuel-web
diff --git a/build/f_repos/patch/fuel-library/0001-Reduce-ceilometer-memory-usage.patch b/build/f_repos/patch/fuel-library/0001-Reduce-ceilometer-memory-usage.patch
new file mode 100644
index 000000000..e51e70b64
--- /dev/null
+++ b/build/f_repos/patch/fuel-library/0001-Reduce-ceilometer-memory-usage.patch
@@ -0,0 +1,26 @@
+From: Michael Polenchuk <mpolenchuk@mirantis.com>
+Date: Thu, 8 Sep 2016 19:25:14 +0300
+Subject: [PATCH] Reduce ceilometer memory usage
+
+Change-Id: I8b1a97bd710c859a3543d1aed8226313f35f4f10
+---
+ .../puppet/openstack_tasks/manifests/ceilometer/controller.pp      | 7 +++++++
+ 1 file changed, 7 insertions(+)
+
+diff --git a/deployment/puppet/openstack_tasks/manifests/ceilometer/controller.pp b/deployment/puppet/openstack_tasks/manifests/ceilometer/controller.pp
+index 2c63ff0..1dda109 100644
+--- a/deployment/puppet/openstack_tasks/manifests/ceilometer/controller.pp
++++ b/deployment/puppet/openstack_tasks/manifests/ceilometer/controller.pp
+@@ -234,5 +234,12 @@ class openstack_tasks::ceilometer::controller {
+         Ceilometer_config<| title == 'oslo_messaging_rabbit/kombu_compression' |> { value => $kombu_compression }
+       }
+     }
++
++    # Reduce memory usage
++    ceilometer_config {
++      'DEFAULT/executor_thread_pool_size':               value => ceiling($service_workers*0.5);
++      'notification/pipeline_processing_queues':         value => ceiling($service_workers*1.5);
++      'oslo_messaging_rabbit/rabbit_qos_prefetch_count': value => 5;
++    }
+   }
+ }
diff --git a/build/f_repos/patch/fuel-library/0002-Disable-token-revoke-to-increase-keystone-performanc.patch b/build/f_repos/patch/fuel-library/0002-Disable-token-revoke-to-increase-keystone-performanc.patch
new file mode 100644
index 000000000..d4baa9c1e
--- /dev/null
+++ b/build/f_repos/patch/fuel-library/0002-Disable-token-revoke-to-increase-keystone-performanc.patch
@@ -0,0 +1,40 @@
+From: iberezovskiy <iberezovskiy@mirantis.com>
+Date: Mon, 19 Sep 2016 12:35:05 +0300
+Subject: [PATCH] Disable token revoke to increase keystone performance
+
+Change-Id: I4df816369093908ad1eac372f38c92155efbe8e0
+Closes-bug: #1625077
+---
+ deployment/puppet/openstack_tasks/manifests/keystone/keystone.pp | 2 ++
+ tests/noop/spec/hosts/keystone/keystone_spec.rb                  | 5 +++++
+ 2 files changed, 7 insertions(+)
+
+diff --git a/deployment/puppet/openstack_tasks/manifests/keystone/keystone.pp b/deployment/puppet/openstack_tasks/manifests/keystone/keystone.pp
+index 3162287..ba9d7df 100644
+--- a/deployment/puppet/openstack_tasks/manifests/keystone/keystone.pp
++++ b/deployment/puppet/openstack_tasks/manifests/keystone/keystone.pp
+@@ -327,6 +327,8 @@ class openstack_tasks::keystone::keystone {
+       memcache_pool_unused_timeout => '60',
+       cache_memcache_servers       => $memcache_servers,
+       policy_driver                => 'keystone.policy.backends.sql.Policy',
++      # Set revoke_by_id to false according to LP #1625077
++      revoke_by_id                 => false,
+     }
+
+     Package<| title == 'keystone'|> ~> Service<| title == 'keystone'|>
+diff --git a/tests/noop/spec/hosts/keystone/keystone_spec.rb b/tests/noop/spec/hosts/keystone/keystone_spec.rb
+index b29e691..3c0f847 100644
+--- a/tests/noop/spec/hosts/keystone/keystone_spec.rb
++++ b/tests/noop/spec/hosts/keystone/keystone_spec.rb
+@@ -176,6 +176,11 @@ describe manifest do
+         should contain_class('keystone').with('sync_db' => primary_controller)
+     end
+
++    it 'should declare keystone class with revoke_by_id set to false' do
++      # Set revoke_by_id to false according to LP #1625077
++      should contain_class('keystone').with('revoke_by_id' => false)
++    end
++
+     it 'should configure keystone with paramters' do
+       should contain_keystone_config('token/caching').with(:value => 'false')
+       should contain_keystone_config('cache/enabled').with(:value => 'true')
diff --git a/build/f_repos/patch/fuel-main/0001-OPNFV-Additions-to-bootstrap_admin_node.sh.patch b/build/f_repos/patch/fuel-main/0001-OPNFV-Additions-to-bootstrap_admin_node.sh.patch
new file mode 100644
index 000000000..446d0b6bd
--- /dev/null
+++ b/build/f_repos/patch/fuel-main/0001-OPNFV-Additions-to-bootstrap_admin_node.sh.patch
@@ -0,0 +1,88 @@
+From: Fuel OPNFV <fuel@opnfv.org>
+Date: Mon, 13 Jun 2016 22:23:57 +0200
+Subject: [PATCH] OPNFV: Additions to bootstrap_admin_node.sh
+
+---
+ iso/bootstrap_admin_node.sh | 39 +++++++++++++++++++++++++++++++--------
+ 1 file changed, 31 insertions(+), 8 deletions(-)
+
+diff --git a/iso/bootstrap_admin_node.sh b/iso/bootstrap_admin_node.sh
+index 3197c91..db3123d 100755
+--- a/iso/bootstrap_admin_node.sh
++++ b/iso/bootstrap_admin_node.sh
+@@ -339,8 +339,22 @@ fuelmenu --save-only --iface=$ADMIN_INTERFACE || fail
+ set +x
+ echo "Done!"
+
++### OPNFV addition BEGIN
++shopt -s nullglob
++for script in /opt/opnfv/bootstrap/pre.d/*.sh
++do
++  echo "Pre script: $script" >> /root/pre.log 2>&1
++  $script >> /root/pre.log 2>&1
++done
++shopt -u nullglob
++### OPNFV addition END
++
++# Enable sshd
++systemctl enable sshd
++systemctl start sshd
++
+ if [[ "$showmenu" == "yes" || "$showmenu" == "YES" ]]; then
+-  fuelmenu || fail
++  fuelmenu
+ else
+   # Give user 15 seconds to enter fuelmenu or else continue
+   echo
+@@ -360,9 +374,10 @@ else
+   fi
+ fi
+
++# OPNFV: Disabled to speedup installation in offline env.
+ # Enable online base MOS repos (security, updates) if we run an ISO installation
+-[ -f /etc/fuel_build_id ] && \
+-  yum-config-manager --enable mos${FUEL_RELEASE}-security mos${FUEL_RELEASE}-updates --save
++#[ -f /etc/fuel_build_id ] && \
++#  yum-config-manager --enable mos${FUEL_RELEASE}-security mos${FUEL_RELEASE}-updates --save
+
+ if [ ! -f "${ASTUTE_YAML}" ]; then
+   echo ${fuelmenu_fail_message}
+@@ -377,9 +392,7 @@ if [ ! -f /etc/fuel_build_id ]; then
+   ln -s ${wwwdir}/${OPENSTACK_VERSION}/ubuntu ${wwwdir}/ubuntu
+ fi
+
+-# Enable sshd
+-systemctl enable sshd
+-systemctl start sshd
++systemctl reload sshd
+
+ # Enable iptables
+ systemctl enable iptables.service
+@@ -532,6 +545,16 @@ systemctl start ntpd
+
+ bash /etc/rc.local
+
++### OPNFV addition BEGIN
++shopt -s nullglob
++for script in /opt/opnfv/bootstrap/post.d/*.sh
++do
++  echo "Post script: $script" >> /root/post.log 2>&1
++  $script >> /root/post.log 2>&1
++done
++shopt -u nullglob
++### OPNFV addition END
++
+ if [ "`get_bootstrap_skip`" = "False" ]; then
+   build_ubuntu_bootstrap bs_status || true
+ else
+@@ -540,8 +563,8 @@ else
+ fi
+
+ #Check if repo is accessible
+-echo "Checking for access to updates repository..."
+-repourl=$(yum repolist all -v | awk '{if ($1 ~ "baseurl" && $3 ~ "updates") print $3}' | head -1)
++echo "Checking for access to updates repository/mirrorlist..."
++repourl=$(yum repolist all -v | awk '{if ($1 ~ "baseurl|mirrors" && $3 ~ "updates") print $3}' | head -1)
+ if urlaccesscheck check "$repourl" ; then
+   UPDATE_ISSUES=0
+ else
diff --git a/build/f_repos/patch/fuel-main/0002-OPNFV-showmenu-yes-in-isolinux.cfg.patch b/build/f_repos/patch/fuel-main/0002-OPNFV-showmenu-yes-in-isolinux.cfg.patch
new file mode 100644
index 000000000..dbff8c7d4
--- /dev/null
+++ b/build/f_repos/patch/fuel-main/0002-OPNFV-showmenu-yes-in-isolinux.cfg.patch
@@ -0,0 +1,24 @@
+From: Fuel OPNFV <fuel@opnfv.org>
+Date: Mon, 13 Jun 2016 22:23:57 +0200
+Subject: [PATCH] OPNFV: showmenu=yes in isolinux.cfg
+
+---
+ iso/isolinux/isolinux.cfg | 4 ++--
+ 1 file changed, 2 insertions(+), 2 deletions(-)
+
+diff --git a/iso/isolinux/isolinux.cfg b/iso/isolinux/isolinux.cfg
+index c6b1ed9..77a4b18 100644
+--- a/iso/isolinux/isolinux.cfg
++++ b/iso/isolinux/isolinux.cfg
+@@ -19,9 +19,9 @@ label nailgun
+   menu label ^1. Fuel Install (Static IP)
+   menu default
+   kernel vmlinuz
+-  append initrd=initrd.img net.ifnames=0 biosdevname=0 inst.repo=cdrom:LABEL=will_be_substituted_with_ISO_VOLUME_ID:/ inst.ks=cdrom:LABEL=will_be_substituted_with_ISO_VOLUME_ID:/ks.cfg ip=10.20.0.2::10.20.0.1:255.255.255.0:fuel.domain.tld:eth0:off::: nameserver=10.20.0.1
++  append initrd=initrd.img net.ifnames=0 biosdevname=0 inst.repo=cdrom:LABEL=will_be_substituted_with_ISO_VOLUME_ID:/ inst.ks=cdrom:LABEL=will_be_substituted_with_ISO_VOLUME_ID:/ks.cfg ip=10.20.0.2::10.20.0.1:255.255.255.0:fuel.domain.tld:eth0:off::: nameserver=10.20.0.1 showmenu=yes
+
+ label nailgunifname
+   menu label ^2. Fuel Advanced Install (Static IP)
+   kernel vmlinuz
+-  append initrd=initrd.img inst.repo=cdrom:LABEL=will_be_substituted_with_ISO_VOLUME_ID:/ inst.ks=cdrom:LABEL=will_be_substituted_with_ISO_VOLUME_ID:/ks.cfg ip=10.20.0.2::10.20.0.1:255.255.255.0:fuel.domain.tld:adminif:off::: nameserver=10.20.0.1 ifname=adminif:XX:XX:XX:XX:XX:XX
++  append initrd=initrd.img inst.repo=cdrom:LABEL=will_be_substituted_with_ISO_VOLUME_ID:/ inst.ks=cdrom:LABEL=will_be_substituted_with_ISO_VOLUME_ID:/ks.cfg ip=10.20.0.2::10.20.0.1:255.255.255.0:fuel.domain.tld:adminif:off::: nameserver=10.20.0.1 ifname=adminif:XX:XX:XX:XX:XX:XX showmenu=yes
diff --git a/build/f_repos/patch/fuel-main/0003-repo-mirror-Allow-multi-arch-local-mirrors.patch b/build/f_repos/patch/fuel-main/0003-repo-mirror-Allow-multi-arch-local-mirrors.patch
new file mode 100644
index 000000000..9469643c3
--- /dev/null
+++ b/build/f_repos/patch/fuel-main/0003-repo-mirror-Allow-multi-arch-local-mirrors.patch
@@ -0,0 +1,59 @@
+From: Stanislaw Kardach <stanislaw.kardach@cavium.com>
+Date: Thu, 25 Feb 2016 13:31:19 +0100
+Subject: [PATCH] repo mirror: Allow multi-arch local mirrors
+
+This patch allows specifying multiple architectures via UBUNTU_ARCH in
+form of a list of space separated architectures. The first architecture
+in the list is considered primary and will be used for building all the
+deb packages by fuel-main. Additional architectures are added to allow
+targets of other architectures to use the mirror.
+NOTE: this imposes a requirement that all packages built are arch
+independent (which is true so far).
+
+Signed-off-by: Stanislaw Kardach <stanislaw.kardach@cavium.com>
+Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+---
+ config.mk               | 2 +-
+ mirror/ubuntu/module.mk | 2 +-
+ sandbox.mk              | 2 +-
+ 3 files changed, 3 insertions(+), 3 deletions(-)
+
+diff --git a/config.mk b/config.mk
+index 74ee039..45a3b30 100644
+--- a/config.mk
++++ b/config.mk
+@@ -49,7 +49,7 @@ UBUNTU_MINOR:=04
+ UBUNTU_RELEASE_NUMBER:=$(UBUNTU_MAJOR).$(UBUNTU_MINOR)
+ UBUNTU_KERNEL_FLAVOR?=lts-trusty
+ UBUNTU_NETBOOT_FLAVOR?=netboot
+-UBUNTU_ARCH:=amd64
++UBUNTU_ARCH?=amd64
+ UBUNTU_IMAGE_RELEASE:=$(UBUNTU_MAJOR)$(UBUNTU_MINOR)
+ SEPARATE_IMAGES?=/boot,ext2 /,ext4
+
+diff --git a/mirror/ubuntu/module.mk b/mirror/ubuntu/module.mk
+index 7a9466e..fe1ada2 100644
+--- a/mirror/ubuntu/module.mk
++++ b/mirror/ubuntu/module.mk
+@@ -81,7 +81,7 @@ $(BUILD_DIR)/mirror/ubuntu/mirror.done:
+ 	--root=$(MIRROR_MOS_UBUNTU_ROOT) \
+ 	--dist=$(MIRROR_MOS_UBUNTU_SUITE) \
+ 	--section=$(subst $(space),$(comma),$(MIRROR_MOS_UBUNTU_SECTION)) \
+-	--arch=$(UBUNTU_ARCH) \
++	--arch=$(shell echo $(UBUNTU_ARCH) | tr ' ' ',') \
+ 	$(LOCAL_MIRROR_UBUNTU)/
+ 	rm -rf $(LOCAL_MIRROR_UBUNTU)/.temp $(LOCAL_MIRROR_UBUNTU)/project
+ 	$(ACTION.TOUCH)
+diff --git a/sandbox.mk b/sandbox.mk
+index 4bc3962..5ffddbd 100644
+--- a/sandbox.mk
++++ b/sandbox.mk
+@@ -188,7 +188,7 @@ touch $(SANDBOX_UBUNTU)/etc/init.d/.legacy-bootordering
+ mkdir -p $(SANDBOX_UBUNTU)/usr/sbin
+ cp -a $(BUILD_DIR)/policy-rc.d $(SANDBOX_UBUNTU)/usr/sbin
+ echo "Running debootstrap"
+-sudo debootstrap --no-check-gpg --include=ca-certificates --arch=$(UBUNTU_ARCH) $(MIRROR_UBUNTU_SUITE) $(SANDBOX_UBUNTU) $(MIRROR_UBUNTU_METHOD)://$(MIRROR_UBUNTU)$(MIRROR_UBUNTU_ROOT)
++sudo debootstrap --no-check-gpg --include=ca-certificates --arch=$(word 1,$(UBUNTU_ARCH)) $(MIRROR_UBUNTU_SUITE) $(SANDBOX_UBUNTU) $(MIRROR_UBUNTU_METHOD)://$(MIRROR_UBUNTU)$(MIRROR_UBUNTU_ROOT)
+ if [ -e $(SANDBOX_UBUNTU)/etc/resolv.conf ]; then sudo cp -a $(SANDBOX_UBUNTU)/etc/resolv.conf $(SANDBOX_UBUNTU)/etc/resolv.conf.orig; fi
+ sudo cp /etc/resolv.conf $(SANDBOX_UBUNTU)/etc/resolv.conf
+ if [ -e $(SANDBOX_UBUNTU)/etc/hosts ]; then sudo cp -a $(SANDBOX_UBUNTU)/etc/hosts $(SANDBOX_UBUNTU)/etc/hosts.orig; fi
diff --git a/build/f_repos/patch/fuel-main/0004-xorriso-fails-to-add-files-with-path-longer-then-240.patch b/build/f_repos/patch/fuel-main/0004-xorriso-fails-to-add-files-with-path-longer-then-240.patch
new file mode 100644
index 000000000..734523040
--- /dev/null
+++ b/build/f_repos/patch/fuel-main/0004-xorriso-fails-to-add-files-with-path-longer-then-240.patch
@@ -0,0 +1,24 @@
+From: Artem Silenkov <asilenkov@mirantis.com>
+Date: Tue, 22 Sep 2015 21:54:44 +0300
+Subject: [PATCH] xorriso fails to add files with path longer then 240
+
+joliet-long option added to xorriso command line
+
+Change-Id: I97016cf3e6a554f44d839740ba7d210337435cdc
+Related-Bug: #1498619
+---
+ iso/module.mk | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/iso/module.mk b/iso/module.mk
+index 8995378..0073068 100644
+--- a/iso/module.mk
++++ b/iso/module.mk
+@@ -174,6 +174,7 @@ $(ISO_PATH): $(BUILD_DIR)/iso/isoroot.done
+ 		-V $(ISO_VOLUME_ID) -p $(ISO_VOLUME_PREP) \
+ 		-J -R \
+ 		-graft-points \
++		-joliet-long \
+ 		-b isolinux/isolinux.bin -no-emul-boot -boot-load-size 4 -boot-info-table \
+ 		-isohybrid-mbr /usr/lib/syslinux/isohdpfx.bin \
+ 		-eltorito-alt-boot -e images/efiboot.img -no-emul-boot \
diff --git a/build/f_repos/patch/fuel-mirror/0001-Fixed-handling-http-redirects.patch b/build/f_repos/patch/fuel-mirror/0001-Fixed-handling-http-redirects.patch
new file mode 100644
index 000000000..b82be1128
--- /dev/null
+++ b/build/f_repos/patch/fuel-mirror/0001-Fixed-handling-http-redirects.patch
@@ -0,0 +1,87 @@
+From: Bulat Gaifullin <bgaifullin@mirantis.com>
+Date: Fri, 24 Jun 2016 15:52:41 +0200
+Subject: [PATCH] Fixed handling http redirects
+
+Override method 'redirect_request' for patching
+new request, that has been created on handling
+http redirect.
+
+Change-Id: I40db406e2377bebec1113639b91a0b5262e2e9ad
+Closes-Bug: 1593674
+(cherry picked from commit 192a3d9f8f993afb12c5108dd9339c6688c23e11)
+---
+ packetary/library/connections.py    | 21 ++++++++++++++++++++-
+ packetary/tests/test_connections.py | 19 +++++++++++++++++++
+ 2 files changed, 39 insertions(+), 1 deletion(-)
+
+diff --git a/packetary/library/connections.py b/packetary/library/connections.py
+index 36a7a84..49b6c9b 100644
+--- a/packetary/library/connections.py
++++ b/packetary/library/connections.py
+@@ -93,9 +93,23 @@ class ResumableResponse(StreamWrapper):
+                 self.stream = response.stream
+
+
+-class RetryHandler(urllib.BaseHandler):
++class RetryHandler(urllib.HTTPRedirectHandler):
+     """urllib Handler to add ability for retrying on server errors."""
+
++    def redirect_request(self, req, fp, code, msg, headers, newurl):
++        new_req = urllib.HTTPRedirectHandler.redirect_request(
++            self, req, fp, code, msg, headers, newurl
++        )
++        if new_req is not None:
++            # We use class assignment for casting new request to type
++            # RetryableRequest
++            new_req.__class__ = RetryableRequest
++            new_req.retries_left = req.retries_left
++            new_req.offset = req.offset
++            new_req.start_time = req.start_time
++            new_req.retry_interval = req.retry_interval
++        return new_req
++
+     @staticmethod
+     def http_request(request):
+         """Initialises http request.
+@@ -118,6 +132,11 @@ class RetryHandler(urllib.BaseHandler):
+         :return: ResumableResponse if success otherwise same response
+         """
+         code, msg = response.getcode(), response.msg
++
++        if 300 <= code < 400:
++            # the redirect group, pass to next handler as is
++            return response
++
+         # the server should response partial content if range is specified
+         if request.offset > 0 and code != 206:
+             raise RangeError(msg)
+diff --git a/packetary/tests/test_connections.py b/packetary/tests/test_connections.py
+index a2621c8..c80b03d 100644
+--- a/packetary/tests/test_connections.py
++++ b/packetary/tests/test_connections.py
+@@ -268,6 +268,25 @@ class TestRetryHandler(base.TestCase):
+         self.handler.http_response(request, response_mock)
+         self.handler.parent.open.assert_called_once_with(request)
+
++    @mock.patch(
++        'packetary.library.connections.urllib.'
++        'HTTPRedirectHandler.redirect_request'
++    )
++    def test_redirect_request(self, redirect_mock, _):
++        redirect_mock.return_value = connections.urllib.Request(
++            'http://localhost/'
++        )
++        req = mock.MagicMock(retries_left=10, retry_interval=5, offset=100)
++        new_req = self.handler.redirect_request(req, -1, 301, "", {}, "")
++        self.assertIsInstance(new_req, connections.RetryableRequest)
++        self.assertEqual(req.retries_left, new_req.retries_left)
++        self.assertEqual(req.retry_interval, new_req.retry_interval)
++        self.assertEqual(req.offset, new_req.offset)
++        redirect_mock.return_value = None
++        self.assertIsNone(
++            self.handler.redirect_request(req, -1, 301, "", {}, "")
++        )
++
+
+ class TestResumeableResponse(base.TestCase):
+     def setUp(self):
diff --git a/build/f_repos/patch/fuel-web/0001-Mark-Intel-82599-10-Gigabit-NIC-as-DPDK-capable.patch b/build/f_repos/patch/fuel-web/0001-Mark-Intel-82599-10-Gigabit-NIC-as-DPDK-capable.patch
new file mode 100644
index 000000000..6fa1e436c
--- /dev/null
+++ b/build/f_repos/patch/fuel-web/0001-Mark-Intel-82599-10-Gigabit-NIC-as-DPDK-capable.patch
@@ -0,0 +1,23 @@
+From: Fuel OPNFV <fuel@opnfv.org>
+Date: Wed, 3 Aug 2016 09:41:21 +0200
+Subject: [PATCH] Mark Intel 82599 10 Gigabit NIC as DPDK capable
+
+This NIC is used on Ericsson POD2 but Fuel doesn't allow to activate
+DPDK on it.
+---
+ nailgun/nailgun/fixtures/openstack.yaml | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/nailgun/nailgun/fixtures/openstack.yaml b/nailgun/nailgun/fixtures/openstack.yaml
+index 74fa509..d08a4fc 100644
+--- a/nailgun/nailgun/fixtures/openstack.yaml
++++ b/nailgun/nailgun/fixtures/openstack.yaml
+@@ -2213,7 +2213,7 @@
+           "15ad:07b0", "8086:15a5", "1137:0043", "1137:0071", "14e4:168a",
+           "14e4:16a9", "14e4:164f", "14e4:168e", "14e4:16af", "14e4:163d",
+           "14e4:163f", "14e4:168d", "14e4:16a1", "14e4:16a2", "14e4:16ad",
+-          "14e4:16ae", "14e4:163e", "14e4:16a4"
++          "14e4:16ae", "14e4:163e", "14e4:16a4", "8086:10f8"
+         ]
+ - pk: 3
+   extend: *ubuntu_release
diff --git a/build/f_repos/sub/fuel-agent b/build/f_repos/sub/fuel-agent
new file mode 160000
+Subproject 7ffbf39caf5845bd82b8ce20a7766cf24aa803f
diff --git a/build/f_repos/sub/fuel-astute b/build/f_repos/sub/fuel-astute
new file mode 160000
+Subproject 390b257240d49cc5e94ed5c4fcd940b5f2f6ec6
diff --git a/build/f_repos/sub/fuel-library b/build/f_repos/sub/fuel-library
new file mode 160000
+Subproject e283b62750d9e26355981b3ad3be7c880944ae0
diff --git a/build/f_repos/sub/fuel-main b/build/f_repos/sub/fuel-main
new file mode 160000
+Subproject d6a22557d132c592b18c6bac90f5f4b8d1aa3ad
diff --git a/build/f_repos/sub/fuel-menu b/build/f_repos/sub/fuel-menu
new file mode 160000
+Subproject 0ed9e206ed1c6271121d3acf52a6bf757411286
diff --git a/build/f_repos/sub/fuel-mirror b/build/f_repos/sub/fuel-mirror
new file mode 160000
+Subproject d1ef06b530ce2149230953bb3810a88ecaff870
diff --git a/build/f_repos/sub/fuel-nailgun-agent b/build/f_repos/sub/fuel-nailgun-agent
new file mode 160000
+Subproject 46fa0db0f8944f9e67699d281d462678aaf4db2
diff --git a/build/f_repos/sub/fuel-ostf b/build/f_repos/sub/fuel-ostf
new file mode 160000
+Subproject f09c98ff7cc71ee612b2450f68a19f2f9c64345
diff --git a/build/f_repos/sub/fuel-ui b/build/f_repos/sub/fuel-ui
new file mode 160000
+Subproject 90de7ef4477230cb7335453ed26ed4306ca6f04
diff --git a/build/f_repos/sub/fuel-upgrade b/build/f_repos/sub/fuel-upgrade
new file mode 160000
+Subproject c1c4bac6a467145ac4fac73e4a7dd2b00380ecf
diff --git a/build/f_repos/sub/fuel-web b/build/f_repos/sub/fuel-web
new file mode 160000
+Subproject e2b85bafb68c348f25cb7cceda81edc668ba2e6
diff --git a/build/f_repos/sub/network-checker b/build/f_repos/sub/network-checker
new file mode 160000
+Subproject fcb47dd095a76288aacf924de574e39709e1f3c
diff --git a/build/f_repos/sub/python-fuelclient b/build/f_repos/sub/python-fuelclient
new file mode 160000
+Subproject 67d8c693a670d27c239d5d175f3ea2a0512c498
diff --git a/build/f_repos/sub/shotgun b/build/f_repos/sub/shotgun
new file mode 160000
+Subproject 781a8cfa0b6eb290e730429fe2792f2b6f5e0c1