diff options
author | Alexandru Avadanii <Alexandru.Avadanii@enea.com> | 2016-09-24 17:27:16 +0200 |
---|---|---|
committer | Alexandru Avadanii <Alexandru.Avadanii@enea.com> | 2016-10-05 21:44:58 +0200 |
commit | 6de5bac371844b70b96b9bd75b62b8d06e5f51fc (patch) | |
tree | 7105db6f62d236c91130d47283842aa4a6f16156 /build | |
parent | 49f066e78e52fde9ce0d467a838211270f518591 (diff) |
build: Use git submodules for Fuel git repos
This change reworks support for cloning (fetching) and patching all
Fuel components (fuel-library ... fuel-ui, see [1] for full list),
both outside the ISO build process (for development purposes, like
testing OPNFV patches apply cleanly to each component), and during
the ISO build.
The implementation relies heavily on git submodules and patches,
so backporting pending upstream gerrit changes, as well as adding new
out-of-upstream-tree patches, should be trivial.
Also, since without tracking remotes the repos rarely change, leverage
the in-place OPNFV build caching mechanism to drastically reduce the
number of git clones during each build.
The mechanism is detailed more in-depth in the JIRA ticket [3], and
it's merely a rehash of the submodule handling in Armband, implemented
initially by Stanislaw Kardach <stanislaw.kardach@cavium.com>.
Pros (+ new features):
- OPNFV patches for Fuel components can be applied (imported) or
developed (exported) outside of the ISO build process;
- git repo caching reduces the number of external pulls;
- reuse the same fuel-* componets in ALL targets
e.g. fuel-mirror used to employ 2 slightly different git trees;
- one step closer to Fuel/Armband merge (lots of steps left though);
Cons:
- adds a lot complexity to solve a non-problem (handling the repos
is fine in the current form, for the current goals);
However, patching <fuel-astute> or <fuel-agent> seems to be
quite complicated with the current mechanism, as env vars
override our locally patched repos for these components).
The proposed change should eliminate this issue completely.
CHANGE:
Aligned divergent fuel-mirror in ISO vs local repo build by
applying the 302 redirect fix ("Fixed handling http redirects") [2]
on top of 9.0.1 tag in fuel-mirror repo, and using that for both.
This obsoleted the followking mk file:
- build/f_isoroot/f_repobuild/config.mk
CHANGE:
Removed unused make target `get-fuel-repo` from build/config.mk.
FIXME:
Remote tracking is currently deactivated for all branches.
Change this when remote HEAD should be tracked instead,
e.g. during Newton rebase.
[1] https://github.com/openstack/fuel-main/blob/stable/mitaka/repos.mk#L32
[2] https://review.openstack.org/#/c/334882/
[3] https://jira.opnfv.org/browse/FUEL-200
TODO: Extend build/f_repos/README.md with info about:
- branch changes;
- tag bumps;
- patching now supports subdirs;
v5 -> v6:
* Added support for nesting patches in subdirs, to be used for feature
groups and related changes across submodules.
E.g. Upcoming multiarch EFI support could go in:
- .../f_repos/patch/fuel-agent/multiarch-efi/...;
- .../f_repos/patch/fuel-astute/multiarch-efi/...;
While Armband patches could go in:
- .../f_repos/patch/fuel-agent/armband/...;
- .../f_repos/patch/fuel-astute/armband/...;
etc.
v6 -> v13:
* fix wrong tag used for VERSION dump;
* fix wrong tag used for patches-export;
* move OPNFV_TAG to main config.mk and use it in FUEL_*_COMMIT;
* only run `patches-import` when patches changed / not yet applied;
* only run `clean` for stale `.cacheid` (avoid re-applying patches);
* allow git submodule path/name to be different (FIXED);
* put-cache should only run when artifact is not already cached;
* `git am`: use 3-way merge so already applied patches are gently
skipped (useful for upstreaming patches from Armband);
* Properly support nested patch dirs;
* Colorize the output a little;
JIRA: FUEL-200
Change-Id: I9dbdd9b3022896d4497d21be8dacc859730db489
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
Diffstat (limited to 'build')
28 files changed, 455 insertions, 102 deletions
diff --git a/build/Makefile b/build/Makefile index 59d9096c6..56acb40fe 100644 --- a/build/Makefile +++ b/build/Makefile @@ -80,10 +80,7 @@ ORIGDIR := $(TOPDIR)/origiso # END of variables to customize ############################################################################# -# Fuel-main destination path and fuel-* submodule patching, for full list check: -# https://github.com/openstack/fuel-main/blob/stable/mitaka/repos.mk#L32-L44 FUEL_MAIN_DIR := /tmp/fuel-main -FUEL_PATCHES := $(shell find $(BUILD_BASE)/patch-repos -name '*.patch' | sort) SUBCLEAN = $(addsuffix .clean,$(SUBDIRS)) @@ -111,9 +108,12 @@ include cache.mk $(ISOCACHE): # Clone Fuel to non-persistent location and build sudo rm -rf $(FUEL_MAIN_DIR) + $(MAKE) -C f_repos -f Makefile release git clone $(FUEL_MAIN_REPO) $(FUEL_MAIN_DIR) - git -C $(FUEL_MAIN_DIR) checkout $(FUEL_MAIN_TAG) - @echo "fuel" `git -C $(FUEL_MAIN_DIR) rev-parse HEAD` >> $(VERSION_FILE) + # Save upstream ref, checkout OPNFV tag (upstream ref + patches) + @echo "fuel" `git -C $(FUEL_MAIN_DIR) \ + rev-parse $(F_OPNFV_TAG)-root` >> $(VERSION_FILE) + git -C $(FUEL_MAIN_DIR) checkout $(F_OPNFV_TAG) # Remove Docker optimizations, otherwise multistrap will fail during # Fuel build. sudo rm -f /etc/apt/apt.conf.d/docker* @@ -134,14 +134,7 @@ $(ISOCACHE): sudo docker info # fuel-main Makefiles do not like `make -C` cd $(FUEL_MAIN_DIR) && make repos - $(REPOINFO) -r $(FUEL_MAIN_DIR) > gitinfo_fuel.txt - # OPNFV patches at Fuel build time - # Need to be commited in order for them to be considered by the Fuel - # build system - $(foreach patch,$(FUEL_PATCHES),git \ - -C $(subst $(BUILD_BASE)/patch-repos,$(FUEL_MAIN_DIR),$(dir $(patch))) \ - am --whitespace=nowarn --committer-date-is-author-date $(patch) || \ - (echo 'Error: Failed patching Fuel repos!' ; exit 1);) + cp f_repos/.cachefuelinfo gitinfo_fuel.txt # Repeat build up to three times sudo -E ./fuel_build_loop @@ -172,6 +165,7 @@ patch-packages: .PHONY: clean $(SUBCLEAN) clean: $(SUBCLEAN) + $(MAKE) -C f_repos -f Makefile clean $(MAKE) -C patch-packages -f Makefile clean @rm -f *.iso @rm -Rf release @@ -182,7 +176,8 @@ clean: $(SUBCLEAN) .PHONY: deepclean deepclean: clean clean-cache - make -C docker clean + $(MAKE) -C f_repos -f Makefile deepclean + $(MAKE) -C docker clean docker rmi opnfv.org/ubuntu-builder:14.04 &>/dev/null || exit 0 docker rmi opnfv.org/ubuntu-builder:latest &>/dev/null || exit 0 docker rmi ubuntu:14.04 &>/dev/null || exit 0 @@ -224,24 +219,26 @@ debug: ############################################################################# # Create a unique hash to be used for getting and putting cache, based on: -# - The commit ID of the full Fuel repo structre +# - The commit ID of the full Fuel repo structure # - The contents of all local Fuel patches +# - Makefile, config and sensitive build scripts fingerprints +# NOTE: Patching is deterministic, so we can fingerprint (submodule root commit +# info + OPNFV patches) at once, after patches-import. +# NOTE: When git submodule remote tracking is active, `git submodule status` +# will point to the latest commits (remote/branch/HEAD) + OPNFV patches. .cacheid: - git clone $(FUEL_MAIN_REPO) $(FUEL_MAIN_DIR) - git -C $(FUEL_MAIN_DIR) checkout $(FUEL_MAIN_TAG) - make -C $(FUEL_MAIN_DIR) repos - $(REPOINFO) -r $(FUEL_MAIN_DIR) > .cachedata - $(foreach patch,$(FUEL_PATCHES),sha1sum $(patch) >> .cachedata;) + $(MAKE) -C f_repos -f Makefile get-cache release + git submodule status | cut -c2-41 > .cachedata sha1sum fuel_build_loop >> .cachedata sha1sum config.mk >> .cachedata sha1sum Makefile >> .cachedata $(CACHETOOL) getbiweek >> .cachedata cat .cachedata | $(CACHETOOL) getid > .cacheid - # Not removing fuel-main as it is re-used in build # Clean local data related to caching - called prior to ordinary build .PHONY: clean-cache clean-cache: $(SUBCLEANCACHE) + $(MAKE) -C f_repos -f Makefile clean-cache rm -f .cachedata .cacheid # Try to download cache - called prior to ordinary build diff --git a/build/config.mk b/build/config.mk index f78043e18..ab30518b5 100644 --- a/build/config.mk +++ b/build/config.mk @@ -1,5 +1,5 @@ ############################################################################## -# Copyright (c) 2015 Ericsson AB and others. +# Copyright (c) 2015,2016 Ericsson AB and others. # stefan.k.berg@ericsson.com # jonas.bjurel@ericsson.com # All rights reserved. This program and the accompanying materials @@ -8,34 +8,59 @@ # http://www.apache.org/licenses/LICENSE-2.0 ############################################################################## -FUEL_MAIN_REPO := https://github.com/openstack/fuel-main -FUEL_MAIN_TAG := 9.0.1 -MOS_VERSION = 9.0 +# fuel-main tag checked out from upstream `fuel-main` repo before patching it +FUEL_MAIN_TAG = 9.0.1 +MOS_VERSION = 9.0 OPENSTACK_VERSION = mitaka-9.0 -# Pinning down exact Fuel repo versions for Fuel 9.0.1 -export FUELLIB_COMMIT?=e283b62750d9e26355981b3ad3be7c880944ae0f -export NAILGUN_COMMIT?=e2b85bafb68c348f25cb7cceda81edc668ba2e64 -export PYTHON_FUELCLIENT_COMMIT?=67d8c693a670d27c239d5d175f3ea2a0512c498c -export FUEL_AGENT_COMMIT?=7ffbf39caf5845bd82b8ce20a7766cf24aa803fb -export FUEL_NAILGUN_AGENT_COMMIT?=46fa0db0f8944f9e67699d281d462678aaf4db26 -export ASTUTE_COMMIT?=390b257240d49cc5e94ed5c4fcd940b5f2f6ec64 -export OSTF_COMMIT?=f09c98ff7cc71ee612b2450f68a19f2f9c64345a -export FUEL_MIRROR_COMMIT?=d1ef06b530ce2149230953bb3810a88ecaff870c -export FUELMENU_COMMIT?=0ed9e206ed1c6271121d3acf52a6bf757411286b -export SHOTGUN_COMMIT?=781a8cfa0b6eb290e730429fe2792f2b6f5e0c11 -export NETWORKCHECKER_COMMIT?=fcb47dd095a76288aacf924de574e39709e1f3ca -export FUELUPGRADE_COMMIT?=c1c4bac6a467145ac4fac73e4a7dd2b00380ecfb -export FUEL_UI_COMMIT?=90de7ef4477230cb7335453ed26ed4306ca6f04f - -# for the patches applying purposes +# FIXME(alav): Disable remote tracking for now, stick to submodule commits +FUEL_TRACK_REMOTES = + +############################################################################## +# Fuel components pinning / remote tracking; use submodules from f_repos +############################################################################## + +# git submodule & patch locations for Fuel components +F_GIT_ROOT := $(shell git rev-parse --show-toplevel) +F_SUBMOD_DIR := ${F_GIT_ROOT}/build/f_repos/sub +F_PATCH_DIR := ${F_GIT_ROOT}/build/f_repos/patch +F_OPNFV_TAG := ${FUEL_MAIN_TAG}-opnfv + +# fuel-main repo location used by main Makefile ISO building, use submodule +FUEL_MAIN_REPO := ${F_SUBMOD_DIR}/fuel-main + +export FUELLIB_REPO?=${F_SUBMOD_DIR}/fuel-library +export NAILGUN_REPO?=${F_SUBMOD_DIR}/fuel-web +export PYTHON_FUELCLIENT_REPO?=${F_SUBMOD_DIR}/python-fuelclient +export FUEL_AGENT_REPO?=${F_SUBMOD_DIR}/fuel-agent +export FUEL_NAILGUN_AGENT_REPO?=${F_SUBMOD_DIR}/fuel-nailgun-agent +export ASTUTE_REPO?=${F_SUBMOD_DIR}/fuel-astute +export OSTF_REPO?=${F_SUBMOD_DIR}/fuel-ostf +export FUEL_MIRROR_REPO?=${F_SUBMOD_DIR}/fuel-mirror +export FUELMENU_REPO?=${F_SUBMOD_DIR}/fuel-menu +export SHOTGUN_REPO?=${F_SUBMOD_DIR}/shotgun +export NETWORKCHECKER_REPO?=${F_SUBMOD_DIR}/network-checker +export FUELUPGRADE_REPO?=${F_SUBMOD_DIR}/fuel-upgrade +export FUEL_UI_REPO?=${F_SUBMOD_DIR}/fuel-ui + +# OPNFV tags are automatically applied by `make -C f_repos patches-import` +export FUELLIB_COMMIT?=${F_OPNFV_TAG} +export NAILGUN_COMMIT?=${F_OPNFV_TAG} +export PYTHON_FUELCLIENT_COMMIT?=${F_OPNFV_TAG} +export FUEL_AGENT_COMMIT?=${F_OPNFV_TAG} +export FUEL_NAILGUN_AGENT_COMMIT?=${F_OPNFV_TAG} +export ASTUTE_COMMIT?=${F_OPNFV_TAG} +export OSTF_COMMIT?=${F_OPNFV_TAG} +export FUEL_MIRROR_COMMIT?=${F_OPNFV_TAG} +export FUELMENU_COMMIT?=${F_OPNFV_TAG} +export SHOTGUN_COMMIT?=${F_OPNFV_TAG} +export NETWORKCHECKER_COMMIT?=${F_OPNFV_TAG} +export FUELUPGRADE_COMMIT?=${F_OPNFV_TAG} +export FUEL_UI_COMMIT?=${F_OPNFV_TAG} + +# for the patches applying purposes (empty git config in docker build container) export GIT_COMMITTER_NAME?=Fuel OPNFV export GIT_COMMITTER_EMAIL?=fuel@opnfv.org DOCKER_REPO := http://get.docker.com/builds/Linux/x86_64 DOCKER_TAG := docker-latest - -.PHONY: get-fuel-repo -get-fuel-repo: - @echo $(FUEL_MAIN_REPO) $(FUEL_MAIN_TAG) - diff --git a/build/f_isoroot/f_repobuild/Makefile b/build/f_isoroot/f_repobuild/Makefile index 7c6b16304..ad1a934fa 100644 --- a/build/f_isoroot/f_repobuild/Makefile +++ b/build/f_isoroot/f_repobuild/Makefile @@ -13,7 +13,6 @@ TOP := $(shell pwd) TMP_ROOT_DIR := $(shell echo "$(MIRROR_UBUNTU_ROOT)" | cut -d "/" -f2) include ../../config.mk -include config.mk export MOS_VERSION export OPENSTACK_VERSION @@ -25,8 +24,7 @@ nailgun: sudo apt-get install -y git libxml2-dev libxslt-dev python-dev python-pip libz-dev libyaml-dev createrepo python-yaml rm -Rf nailgun sudo mkdir -p /var/www/nailgun - git clone $(FUEL_MIRROR_REPO) - cd fuel-mirror && git checkout -q $(FUEL_MIRROR_COMMIT) + ln -sf ${F_SUBMOD_DIR}/fuel-mirror fuel-mirror sudo pip install -U -r ./fuel-mirror/requirements.txt sudo pip install ./fuel-mirror sudo pip install ./fuel-mirror/contrib/fuel_mirror @@ -34,7 +32,7 @@ nailgun: sudo fuel-mirror --debug --config ./opnfv-config.yaml create --group ubuntu --pattern=ubuntu sudo chmod -R 755 /var/www/nailgun cp -Rp /var/www/nailgun . - # On the end we want to have ubuntu repository in mirrors/ubuntu directory + # In the end we want to have ubuntu repository in mirrors/ubuntu directory -if [ "$(MIRROR_UBUNTU_ROOT)" != "/ubuntu/" ]; then \ mkdir -p nailgun/mirrors/ubuntu;\ mv nailgun/mirrors$(MIRROR_UBUNTU_ROOT)* nailgun/mirrors/ubuntu;\ @@ -65,7 +63,6 @@ release:nailgun .cacheid: date +"Repocache %G%V" > .cachedata sha1sum Makefile >> .cachedata - sha1sum config.mk >> .cachedata $(CACHETOOL) packages >> .cachedata echo -n $(UBUNTU_ARCH) | sha1sum | awk {'print $$1'} >> .cachedata cat .cachedata | $(CACHETOOL) getid > .cacheid diff --git a/build/f_isoroot/f_repobuild/config.mk b/build/f_isoroot/f_repobuild/config.mk deleted file mode 100644 index d81732849..000000000 --- a/build/f_isoroot/f_repobuild/config.mk +++ /dev/null @@ -1,13 +0,0 @@ -############################################################################## -# Copyright (c) 2016 Ericsson AB and others. -# mskalski@mirantis.com -# All rights reserved. This program and the accompanying materials -# are made available under the terms of the Apache License, Version 2.0 -# which accompanies this distribution, and is available at -# http://www.apache.org/licenses/LICENSE-2.0 -############################################################################## - -FUEL_MIRROR_REPO?=https://github.com/openstack/fuel-mirror -# Point to the commit where 302 redirects are handled -# https://bugs.launchpad.net/fuel/mitaka/+bug/1593674 -FUEL_MIRROR_COMMIT?=192a3d9f8f993afb12c5108dd9339c6688c23e11 diff --git a/build/f_repos/Makefile b/build/f_repos/Makefile new file mode 100644 index 000000000..66abf2d71 --- /dev/null +++ b/build/f_repos/Makefile @@ -0,0 +1,165 @@ +############################################################################## +# Copyright (c) 2015,2016 Ericsson AB, Enea AB and others. +# stefan.k.berg@ericsson.com +# jonas.bjurel@ericsson.com +# Alexandru.Avadanii@enea.com +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Apache License, Version 2.0 +# which accompanies this distribution, and is available at +# http://www.apache.org/licenses/LICENSE-2.0 +############################################################################## + +SHELL = /bin/bash +REPOINFO = $(shell readlink -e ../repo_info.sh) +FREPODIR = $(shell pwd) +FPATCHES = $(shell find ${F_PATCH_DIR} -name '*.patch') + +include ../config.mk + +# NOTE: Mechanism overview is presented in ./README.md. + +# Submodule consistent states: +# - NOT initialized (submodule trees are not populated at all); +# - initialized, bound to saved commits; +# - initialized, tracking remote origin (only for FUEL_TRACK_REMOTES); +# - patched (local patches are applied); + +# In order to keep things sort of separate, we should only pass up (to main +# Makefile) the fully-pactched repos, and gather any fingerprinting info here. + +# Fuel@OPNFV relies on upstream git repos (one per component) in 1 of 2 ways: +# - pinned down to tag objects (e.g. "9.0.1") +# - tracking upstream remote HEAD on a stable or master branch +# FIXME(alav): Should we support mixed cases? (e.g. pin down only fuel-main) + +# To enable remote tracking, set the following var to any non-empty string. +# Leaving this var empty will bind each git submodule to its saved commit. +# NOTE: For non-stable branches, unless overriden, always track remotes +FUEL_TRACK_REMOTES ?= $(shell git symbolic-ref --short HEAD | grep -v stable) + +.PHONY: all +all: release + +############################################################################## +# git submodule operations - to be used stand-alone or from parent Makefile +############################################################################## + +# Fetch & update git submodules, checkout remote HEAD or saved commit +# Also gather fingerprints for parent gitinfo_fuel.txt. +.PHONY: sub +sub: .cachefuelinfo + +.cachefuelinfo: + @if [ -n "${FUEL_TRACK_REMOTES}" ]; then \ + git submodule update --init --remote; \ + else \ + git submodule update --init; \ + fi + @rm -f $@ + @git submodule -q foreach '${REPOINFO} . >> ${FREPODIR}/$@' + +# Generate patches from submodules +.PHONY: patches-export +patches-export: sub + @git submodule -q foreach ' \ + SUB_DIR=${F_PATCH_DIR}/$$name; \ + git tag | awk "!/root/ && /${F_OPNFV_TAG}-fuel/" | while read F_TAG; do \ + SUB_FEATURE=`dirname $${F_TAG#${F_OPNFV_TAG}-fuel/}`; \ + echo "`tput setaf 2`-- exporting $$name ($$F_TAG)`tput sgr0`"; \ + mkdir -p $$SUB_DIR/$${SUB_FEATURE} && \ + git format-patch --no-signature --ignore-space-at-eol \ + -o $$SUB_DIR/$$SUB_FEATURE -N $$F_TAG-root..$$F_TAG; \ + done' + @sed -i -e '1d' -e 's/[[:space:]]*$$//' ${FPATCHES} + +# Apply patches from patch/* to respective submodules +# We rely on `make sub` and/or `make clean` to checkout correct base +.PHONY: patches-import +patches-import: sub .cachepatched + +.cachepatched: ${FPATCHES} + @$(MAKE) clean + @git submodule -q foreach ' \ + SUB_DIR=${F_PATCH_DIR}/$$name; mkdir -p $$SUB_DIR && \ + git tag ${F_OPNFV_TAG}-root && \ + git checkout -q -b opnfv-fuel && \ + find $$SUB_DIR -type d | sort | while read p_dir; do \ + SUB_PATCHES=$$(ls $$p_dir/*.patch 2>/dev/null); \ + if [ -n "$$SUB_PATCHES" ]; then \ + SUB_FEATURE=$${p_dir#$$SUB_DIR}; \ + SUB_TAG=${F_OPNFV_TAG}-fuel$$SUB_FEATURE/patch; \ + echo "`tput setaf 2`-- patching $$name ($$SUB_TAG)`tput sgr0`";\ + git tag $$SUB_TAG-root && git am -3 --whitespace=nowarn \ + --committer-date-is-author-date $$SUB_PATCHES && \ + git tag $$SUB_TAG || exit 1; \ + fi \ + done && \ + git tag ${F_OPNFV_TAG}' + @touch $@ + +# Clean any changes made to submodules, checkout upstream Fuel root commit +.PHONY: clean +clean: + @git submodule -q foreach ' \ + git am -q --abort > /dev/null 2>&1; \ + git checkout -q ${F_OPNFV_TAG}-root > /dev/null 2>&1; \ + git branch -q -D opnfv-fuel > /dev/null 2>&1; \ + git tag | grep ${F_OPNFV_TAG} | xargs git tag -d > /dev/null 2>&1; \ + git reset -q --hard HEAD; \ + git clean -xdff' + @rm -f .cachepatched + +.PHONY: deepclean +deepclean: clean clean-cache + @git submodule deinit -f . + @rm -f .cache* + +.PHONY: release +release: sub + # Store artifact in cache straight away if caching is enabled + # (no .cacheid will be present unless this is a cached build) + test -f .cacheid && $(MAKE) -f Makefile put-cache || exit 0 + # NOTE: Patches are not included in cache + $(MAKE) -f Makefile patches-import + +############################################################################## +# Cache operations - only used when building through ci/build.sh +############################################################################## + +# NOTE: Current method of collecting submodule refs requires submodules to be +# NOT initialized <OR> NO patches applied. +# NOTE: Querying `git submodule status` from parent will show the patched tree. +# Create a unique hash to be used for getting and putting cache, based on: +# - git submodule SHAs, collected with `git submodule status` +# - The contents of this Makefile +.cacheid: + @$(MAKE) clean + sha1sum Makefile > .cachedata + git submodule status | cut -c2-41 >> .cachedata + cat .cachedata | $(CACHETOOL) getid > .cacheid + +# Clean local data related to caching - called prior to ordinary build +.PHONY: clean-cache +clean-cache: clean + rm -f .cachedata .cacheid + +# Try to download cache - called prior to ordinary build +.PHONY: get-cache +get-cache: .cacheid + @if $(CACHETOOL) check $(shell cat .cacheid); then \ + $(MAKE) clean && \ + $(CACHETOOL) get $(shell cat .cacheid) | \ + tar xf - -C ${F_GIT_ROOT}/.git; \ + else \ + echo "No cache item found for $(shell cat .cacheid)" ;\ + exit 0;\ + fi + +# Store cache if not already stored - called after ordinary build +.PHONY: put-cache +put-cache: .cacheid + @if ! $(CACHETOOL) check $(shell cat .cacheid); then \ + $(MAKE) clean && \ + tar cf - -C ${F_GIT_ROOT}/.git modules | \ + $(CACHETOOL) put $(shell cat .cacheid); \ + fi diff --git a/build/f_repos/README.md b/build/f_repos/README.md new file mode 100644 index 000000000..6cc825e05 --- /dev/null +++ b/build/f_repos/README.md @@ -0,0 +1,94 @@ +Fuel@OPNFV submodule fetching and patching +========================================== + +This directory holds submodule fetching/patching scripts, intended for +working with upstream Fuel components (fuel-library, ... , fuel-ui) in +developing/applying OPNFV patches (backports, custom fixes etc.). + +The scripts should be friendly to the following 2 use-cases: + - development work: easily cloning, binding repos to specific commits, + remote tracking, patch development etc.; + - to provide parent build scripts an easy method of tracking upstream + references and applying OPNFV patches on top; + +Also, we need to support at least the following modes of operations: + - submodule bind - each submodule patches will be based on the commit ID + saved in the .gitmodules config file; + - remote tracking - each submodule will sync with the upstream remote + and patches will be applied on top of <sub_remote>/<sub_branch>/HEAD; + +Workflow (development) +---------------------- +The standard development workflow should look as follows: + +1. Decide whether remote tracking should be active or not: + NOTE: Setting the following var to any non-empty str enables remote track. + NOTE: Leaving unset will enable remote track for anything but stable branch. + + $ export FUEL_TRACK_REMOTES="" + +2. All Fuel sub-projects are registered as submodules. To initialize them, call: + If remote tracking is active, upstream remote is queried and latest remote + branch HEAD is fetched. Otherwise, checkout commit IDs from .gitmodules. + + $ make sub + +3. Apply patches from `patches/<sub-project>/*` to respective submodules via: + + $ make patches-import + + This will result in creation of: + - a tag called `${FUEL_MAIN_TAG}-opnfv-root` at the same commit as Fuel@OPNFV + upstream reference (bound to git submodule OR tracking remote HEAD); + - a new branch `opnfv-fuel` which will hold all the OPNFV patches, + each patch is applied on this new branch with `git-am`; + - a tag called `${FUEL_MAIN_TAG}-opnfv` at `opnfv-fuel/HEAD`; + +4. Modify sub-projects for whatever you need. + Commit your changes when you want them taken into account in the build. + +5. Re-create patches via: + + $ make patches-export + + Each commit on `opnfv-fuel` branch of each subproject will be + exported to `patches/subproject/` via `git format-patch`. + + NOTE: Only commit (-f) submodules when you need to bump upstream ref. + NOTE: DO NOT commit patched submodules! + +6. Clean workbench branches and tags with: + + $ make clean + +7. De-initialize submodules and force a clean clone with: + + $ make deepclean + +Workflow (ISO build) +-------------------- +Parent build scripts require this mechanism to do some fingerprinting, +so here is the intended flow for all artifacts to be generated right: + +1. (Optional) Cached submodules might be fetched from build cache. + +2. Submodules are updated + We also dump each submodule's git info using repo_info.sh, since + we want to collect git refs before patching (i.e. upstream refs). + +3. Make target `release` is built + This will make sure the modules are in a clean state, put them in cache, + then apply the patches. + +4. fuel-main's `${FUEL_MAIN_TAG}-opnfv-root` tag is used to determine VERSION info + It will accommodate both bound tags and remote tracking references. + +Sub-project maintenance +----------------------- +1. Adding a new submodule + If you need to add another subproject, you can do it with `git submodule`. + Make sure that you specify branch (with `-b`), short name (with `--name`) + and point it to `upstream/*` directory, i.e.: + + $ git submodule -b stable/mitaka add --name fuel-web \ + https://github.com/openstack/fuel-web.git upstream/fuel-web diff --git a/build/patch-repos/build/repos/fuel-library/0001-Reduce-ceilometer-memory-usage.patch b/build/f_repos/patch/fuel-library/0001-Reduce-ceilometer-memory-usage.patch index f7fe06884..e51e70b64 100644 --- a/build/patch-repos/build/repos/fuel-library/0001-Reduce-ceilometer-memory-usage.patch +++ b/build/f_repos/patch/fuel-library/0001-Reduce-ceilometer-memory-usage.patch @@ -1,11 +1,10 @@ -From 38315ee3945a5444de13918eb29e7771eb9927c7 Mon Sep 17 00:00:00 2001 From: Michael Polenchuk <mpolenchuk@mirantis.com> Date: Thu, 8 Sep 2016 19:25:14 +0300 -Subject: [PATCH 1/1] Reduce ceilometer memory usage +Subject: [PATCH] Reduce ceilometer memory usage Change-Id: I8b1a97bd710c859a3543d1aed8226313f35f4f10 --- - .../manifests/ceilometer/controller.pp | 7 +++++++ + .../puppet/openstack_tasks/manifests/ceilometer/controller.pp | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/deployment/puppet/openstack_tasks/manifests/ceilometer/controller.pp b/deployment/puppet/openstack_tasks/manifests/ceilometer/controller.pp @@ -25,6 +24,3 @@ index 2c63ff0..1dda109 100644 + } } } --- -1.7.9.5 - diff --git a/build/patch-repos/build/repos/fuel-library/0001-Disable-token-revoke-to-increase-keystone-performanc.patch b/build/f_repos/patch/fuel-library/0002-Disable-token-revoke-to-increase-keystone-performanc.patch index 4d5be493b..d4baa9c1e 100644 --- a/build/patch-repos/build/repos/fuel-library/0001-Disable-token-revoke-to-increase-keystone-performanc.patch +++ b/build/f_repos/patch/fuel-library/0002-Disable-token-revoke-to-increase-keystone-performanc.patch @@ -1,4 +1,3 @@ -From ba542dde34743b276547b7ca3c6591feb9000877 Mon Sep 17 00:00:00 2001 From: iberezovskiy <iberezovskiy@mirantis.com> Date: Mon, 19 Sep 2016 12:35:05 +0300 Subject: [PATCH] Disable token revoke to increase keystone performance @@ -6,8 +5,8 @@ Subject: [PATCH] Disable token revoke to increase keystone performance Change-Id: I4df816369093908ad1eac372f38c92155efbe8e0 Closes-bug: #1625077 --- - .../openstack_tasks/manifests/keystone/keystone.pp | 2 ++ - tests/noop/spec/hosts/keystone/keystone_spec.rb | 5 +++++ + deployment/puppet/openstack_tasks/manifests/keystone/keystone.pp | 2 ++ + tests/noop/spec/hosts/keystone/keystone_spec.rb | 5 +++++ 2 files changed, 7 insertions(+) diff --git a/deployment/puppet/openstack_tasks/manifests/keystone/keystone.pp b/deployment/puppet/openstack_tasks/manifests/keystone/keystone.pp @@ -21,7 +20,7 @@ index 3162287..ba9d7df 100644 + # Set revoke_by_id to false according to LP #1625077 + revoke_by_id => false, } - + Package<| title == 'keystone'|> ~> Service<| title == 'keystone'|> diff --git a/tests/noop/spec/hosts/keystone/keystone_spec.rb b/tests/noop/spec/hosts/keystone/keystone_spec.rb index b29e691..3c0f847 100644 @@ -30,7 +29,7 @@ index b29e691..3c0f847 100644 @@ -176,6 +176,11 @@ describe manifest do should contain_class('keystone').with('sync_db' => primary_controller) end - + + it 'should declare keystone class with revoke_by_id set to false' do + # Set revoke_by_id to false according to LP #1625077 + should contain_class('keystone').with('revoke_by_id' => false) @@ -39,6 +38,3 @@ index b29e691..3c0f847 100644 it 'should configure keystone with paramters' do should contain_keystone_config('token/caching').with(:value => 'false') should contain_keystone_config('cache/enabled').with(:value => 'true') --- -1.7.9.5 - diff --git a/build/patch-repos/0010-bootstrap_admin_node.sh.patch b/build/f_repos/patch/fuel-main/0001-OPNFV-Additions-to-bootstrap_admin_node.sh.patch index d3769adfb..446d0b6bd 100644 --- a/build/patch-repos/0010-bootstrap_admin_node.sh.patch +++ b/build/f_repos/patch/fuel-main/0001-OPNFV-Additions-to-bootstrap_admin_node.sh.patch @@ -1,10 +1,13 @@ From: Fuel OPNFV <fuel@opnfv.org> Date: Mon, 13 Jun 2016 22:23:57 +0200 -Subject: OPNFV: Additions to bootstrap_admin_node.sh +Subject: [PATCH] OPNFV: Additions to bootstrap_admin_node.sh --- + iso/bootstrap_admin_node.sh | 39 +++++++++++++++++++++++++++++++-------- + 1 file changed, 31 insertions(+), 8 deletions(-) + diff --git a/iso/bootstrap_admin_node.sh b/iso/bootstrap_admin_node.sh -index 3197c91..e035145 100755 +index 3197c91..db3123d 100755 --- a/iso/bootstrap_admin_node.sh +++ b/iso/bootstrap_admin_node.sh @@ -339,8 +339,22 @@ fuelmenu --save-only --iface=$ADMIN_INTERFACE || fail @@ -44,7 +47,9 @@ index 3197c91..e035145 100755 if [ ! -f "${ASTUTE_YAML}" ]; then echo ${fuelmenu_fail_message} -@@ -377,7 +392,5 @@ if [ ! -f /etc/fuel_build_id ]; then +@@ -377,9 +392,7 @@ if [ ! -f /etc/fuel_build_id ]; then + ln -s ${wwwdir}/${OPENSTACK_VERSION}/ubuntu ${wwwdir}/ubuntu + fi -# Enable sshd -systemctl enable sshd @@ -70,7 +75,7 @@ index 3197c91..e035145 100755 if [ "`get_bootstrap_skip`" = "False" ]; then build_ubuntu_bootstrap bs_status || true else -@@ -571,8 +571,8 @@ +@@ -540,8 +563,8 @@ else fi #Check if repo is accessible diff --git a/build/patch-repos/0020-isolinux.cfg.patch b/build/f_repos/patch/fuel-main/0002-OPNFV-showmenu-yes-in-isolinux.cfg.patch index f09be2a7f..dbff8c7d4 100644 --- a/build/patch-repos/0020-isolinux.cfg.patch +++ b/build/f_repos/patch/fuel-main/0002-OPNFV-showmenu-yes-in-isolinux.cfg.patch @@ -1,8 +1,11 @@ From: Fuel OPNFV <fuel@opnfv.org> Date: Mon, 13 Jun 2016 22:23:57 +0200 -Subject: OPNFV: showmenu=yes in isolinux.cfg +Subject: [PATCH] OPNFV: showmenu=yes in isolinux.cfg --- + iso/isolinux/isolinux.cfg | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + diff --git a/iso/isolinux/isolinux.cfg b/iso/isolinux/isolinux.cfg index c6b1ed9..77a4b18 100644 --- a/iso/isolinux/isolinux.cfg diff --git a/build/patch-repos/0030-repo-multi-arch-local-mirrors.patch b/build/f_repos/patch/fuel-main/0003-repo-mirror-Allow-multi-arch-local-mirrors.patch index d40428743..9469643c3 100644 --- a/build/patch-repos/0030-repo-multi-arch-local-mirrors.patch +++ b/build/f_repos/patch/fuel-main/0003-repo-mirror-Allow-multi-arch-local-mirrors.patch @@ -1,6 +1,6 @@ From: Stanislaw Kardach <stanislaw.kardach@cavium.com> Date: Thu, 25 Feb 2016 13:31:19 +0100 -Subject: repo mirror: Allow multi-arch local mirrors +Subject: [PATCH] repo mirror: Allow multi-arch local mirrors This patch allows specifying multiple architectures via UBUNTU_ARCH in form of a list of space separated architectures. The first architecture @@ -13,11 +13,24 @@ independent (which is true so far). Signed-off-by: Stanislaw Kardach <stanislaw.kardach@cavium.com> Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com> --- + config.mk | 2 +- mirror/ubuntu/module.mk | 2 +- sandbox.mk | 2 +- - config.mk | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) +diff --git a/config.mk b/config.mk +index 74ee039..45a3b30 100644 +--- a/config.mk ++++ b/config.mk +@@ -49,7 +49,7 @@ UBUNTU_MINOR:=04 + UBUNTU_RELEASE_NUMBER:=$(UBUNTU_MAJOR).$(UBUNTU_MINOR) + UBUNTU_KERNEL_FLAVOR?=lts-trusty + UBUNTU_NETBOOT_FLAVOR?=netboot +-UBUNTU_ARCH:=amd64 ++UBUNTU_ARCH?=amd64 + UBUNTU_IMAGE_RELEASE:=$(UBUNTU_MAJOR)$(UBUNTU_MINOR) + SEPARATE_IMAGES?=/boot,ext2 /,ext4 + diff --git a/mirror/ubuntu/module.mk b/mirror/ubuntu/module.mk index 7a9466e..fe1ada2 100644 --- a/mirror/ubuntu/module.mk @@ -44,16 +57,3 @@ index 4bc3962..5ffddbd 100644 if [ -e $(SANDBOX_UBUNTU)/etc/resolv.conf ]; then sudo cp -a $(SANDBOX_UBUNTU)/etc/resolv.conf $(SANDBOX_UBUNTU)/etc/resolv.conf.orig; fi sudo cp /etc/resolv.conf $(SANDBOX_UBUNTU)/etc/resolv.conf if [ -e $(SANDBOX_UBUNTU)/etc/hosts ]; then sudo cp -a $(SANDBOX_UBUNTU)/etc/hosts $(SANDBOX_UBUNTU)/etc/hosts.orig; fi -diff --git a/config.mk b/config.mk -index 74ee039..45a3b30 100644 ---- a/config.mk -+++ b/config.mk -@@ -49,7 +49,7 @@ UBUNTU_MINOR:=04 - UBUNTU_RELEASE_NUMBER:=$(UBUNTU_MAJOR).$(UBUNTU_MINOR) - UBUNTU_KERNEL_FLAVOR?=lts-trusty - UBUNTU_NETBOOT_FLAVOR?=netboot --UBUNTU_ARCH:=amd64 -+UBUNTU_ARCH?=amd64 - UBUNTU_IMAGE_RELEASE:=$(UBUNTU_MAJOR)$(UBUNTU_MINOR) - SEPARATE_IMAGES?=/boot,ext2 /,ext4 - diff --git a/build/patch-repos/0040-xorriso-fails-to-add-files-with-path-longer-then-240.patch b/build/f_repos/patch/fuel-main/0004-xorriso-fails-to-add-files-with-path-longer-then-240.patch index 734523040..734523040 100644 --- a/build/patch-repos/0040-xorriso-fails-to-add-files-with-path-longer-then-240.patch +++ b/build/f_repos/patch/fuel-main/0004-xorriso-fails-to-add-files-with-path-longer-then-240.patch diff --git a/build/f_repos/patch/fuel-mirror/0001-Fixed-handling-http-redirects.patch b/build/f_repos/patch/fuel-mirror/0001-Fixed-handling-http-redirects.patch new file mode 100644 index 000000000..b82be1128 --- /dev/null +++ b/build/f_repos/patch/fuel-mirror/0001-Fixed-handling-http-redirects.patch @@ -0,0 +1,87 @@ +From: Bulat Gaifullin <bgaifullin@mirantis.com> +Date: Fri, 24 Jun 2016 15:52:41 +0200 +Subject: [PATCH] Fixed handling http redirects + +Override method 'redirect_request' for patching +new request, that has been created on handling +http redirect. + +Change-Id: I40db406e2377bebec1113639b91a0b5262e2e9ad +Closes-Bug: 1593674 +(cherry picked from commit 192a3d9f8f993afb12c5108dd9339c6688c23e11) +--- + packetary/library/connections.py | 21 ++++++++++++++++++++- + packetary/tests/test_connections.py | 19 +++++++++++++++++++ + 2 files changed, 39 insertions(+), 1 deletion(-) + +diff --git a/packetary/library/connections.py b/packetary/library/connections.py +index 36a7a84..49b6c9b 100644 +--- a/packetary/library/connections.py ++++ b/packetary/library/connections.py +@@ -93,9 +93,23 @@ class ResumableResponse(StreamWrapper): + self.stream = response.stream + + +-class RetryHandler(urllib.BaseHandler): ++class RetryHandler(urllib.HTTPRedirectHandler): + """urllib Handler to add ability for retrying on server errors.""" + ++ def redirect_request(self, req, fp, code, msg, headers, newurl): ++ new_req = urllib.HTTPRedirectHandler.redirect_request( ++ self, req, fp, code, msg, headers, newurl ++ ) ++ if new_req is not None: ++ # We use class assignment for casting new request to type ++ # RetryableRequest ++ new_req.__class__ = RetryableRequest ++ new_req.retries_left = req.retries_left ++ new_req.offset = req.offset ++ new_req.start_time = req.start_time ++ new_req.retry_interval = req.retry_interval ++ return new_req ++ + @staticmethod + def http_request(request): + """Initialises http request. +@@ -118,6 +132,11 @@ class RetryHandler(urllib.BaseHandler): + :return: ResumableResponse if success otherwise same response + """ + code, msg = response.getcode(), response.msg ++ ++ if 300 <= code < 400: ++ # the redirect group, pass to next handler as is ++ return response ++ + # the server should response partial content if range is specified + if request.offset > 0 and code != 206: + raise RangeError(msg) +diff --git a/packetary/tests/test_connections.py b/packetary/tests/test_connections.py +index a2621c8..c80b03d 100644 +--- a/packetary/tests/test_connections.py ++++ b/packetary/tests/test_connections.py +@@ -268,6 +268,25 @@ class TestRetryHandler(base.TestCase): + self.handler.http_response(request, response_mock) + self.handler.parent.open.assert_called_once_with(request) + ++ @mock.patch( ++ 'packetary.library.connections.urllib.' ++ 'HTTPRedirectHandler.redirect_request' ++ ) ++ def test_redirect_request(self, redirect_mock, _): ++ redirect_mock.return_value = connections.urllib.Request( ++ 'http://localhost/' ++ ) ++ req = mock.MagicMock(retries_left=10, retry_interval=5, offset=100) ++ new_req = self.handler.redirect_request(req, -1, 301, "", {}, "") ++ self.assertIsInstance(new_req, connections.RetryableRequest) ++ self.assertEqual(req.retries_left, new_req.retries_left) ++ self.assertEqual(req.retry_interval, new_req.retry_interval) ++ self.assertEqual(req.offset, new_req.offset) ++ redirect_mock.return_value = None ++ self.assertIsNone( ++ self.handler.redirect_request(req, -1, 301, "", {}, "") ++ ) ++ + + class TestResumeableResponse(base.TestCase): + def setUp(self): diff --git a/build/patch-repos/build/repos/fuel-nailgun/0010-Mark-Intel-82599-10-Gigabit-NIC-as-DPDK-capable.patch b/build/f_repos/patch/fuel-web/0001-Mark-Intel-82599-10-Gigabit-NIC-as-DPDK-capable.patch index d486f2354..6fa1e436c 100644 --- a/build/patch-repos/build/repos/fuel-nailgun/0010-Mark-Intel-82599-10-Gigabit-NIC-as-DPDK-capable.patch +++ b/build/f_repos/patch/fuel-web/0001-Mark-Intel-82599-10-Gigabit-NIC-as-DPDK-capable.patch @@ -12,11 +12,12 @@ diff --git a/nailgun/nailgun/fixtures/openstack.yaml b/nailgun/nailgun/fixtures/ index 74fa509..d08a4fc 100644 --- a/nailgun/nailgun/fixtures/openstack.yaml +++ b/nailgun/nailgun/fixtures/openstack.yaml -@@ -2215,3 +2215,3 @@ +@@ -2213,7 +2213,7 @@ + "15ad:07b0", "8086:15a5", "1137:0043", "1137:0071", "14e4:168a", + "14e4:16a9", "14e4:164f", "14e4:168e", "14e4:16af", "14e4:163d", "14e4:163f", "14e4:168d", "14e4:16a1", "14e4:16a2", "14e4:16ad", - "14e4:16ae", "14e4:163e", "14e4:16a4" + "14e4:16ae", "14e4:163e", "14e4:16a4", "8086:10f8" ] --- -1.9.1 - + - pk: 3 + extend: *ubuntu_release diff --git a/build/f_repos/sub/fuel-agent b/build/f_repos/sub/fuel-agent new file mode 160000 +Subproject 7ffbf39caf5845bd82b8ce20a7766cf24aa803f diff --git a/build/f_repos/sub/fuel-astute b/build/f_repos/sub/fuel-astute new file mode 160000 +Subproject 390b257240d49cc5e94ed5c4fcd940b5f2f6ec6 diff --git a/build/f_repos/sub/fuel-library b/build/f_repos/sub/fuel-library new file mode 160000 +Subproject e283b62750d9e26355981b3ad3be7c880944ae0 diff --git a/build/f_repos/sub/fuel-main b/build/f_repos/sub/fuel-main new file mode 160000 +Subproject d6a22557d132c592b18c6bac90f5f4b8d1aa3ad diff --git a/build/f_repos/sub/fuel-menu b/build/f_repos/sub/fuel-menu new file mode 160000 +Subproject 0ed9e206ed1c6271121d3acf52a6bf757411286 diff --git a/build/f_repos/sub/fuel-mirror b/build/f_repos/sub/fuel-mirror new file mode 160000 +Subproject d1ef06b530ce2149230953bb3810a88ecaff870 diff --git a/build/f_repos/sub/fuel-nailgun-agent b/build/f_repos/sub/fuel-nailgun-agent new file mode 160000 +Subproject 46fa0db0f8944f9e67699d281d462678aaf4db2 diff --git a/build/f_repos/sub/fuel-ostf b/build/f_repos/sub/fuel-ostf new file mode 160000 +Subproject f09c98ff7cc71ee612b2450f68a19f2f9c64345 diff --git a/build/f_repos/sub/fuel-ui b/build/f_repos/sub/fuel-ui new file mode 160000 +Subproject 90de7ef4477230cb7335453ed26ed4306ca6f04 diff --git a/build/f_repos/sub/fuel-upgrade b/build/f_repos/sub/fuel-upgrade new file mode 160000 +Subproject c1c4bac6a467145ac4fac73e4a7dd2b00380ecf diff --git a/build/f_repos/sub/fuel-web b/build/f_repos/sub/fuel-web new file mode 160000 +Subproject e2b85bafb68c348f25cb7cceda81edc668ba2e6 diff --git a/build/f_repos/sub/network-checker b/build/f_repos/sub/network-checker new file mode 160000 +Subproject fcb47dd095a76288aacf924de574e39709e1f3c diff --git a/build/f_repos/sub/python-fuelclient b/build/f_repos/sub/python-fuelclient new file mode 160000 +Subproject 67d8c693a670d27c239d5d175f3ea2a0512c498 diff --git a/build/f_repos/sub/shotgun b/build/f_repos/sub/shotgun new file mode 160000 +Subproject 781a8cfa0b6eb290e730429fe2792f2b6f5e0c1 |