Improved caching functionality and tracability

In order to enable the caching of more build artifacts than the Fuel
iso during CI builds, the caching functionality and the CI build
frontend "build.sh" has been rewritten.

The build.sh script will now rely on the "make cache" functionality of
the top Makefile to make sure that the build is using cache handling.

The underlying cache logic is implemented in "cache.mk" for the top
makefile and those recursive levels that do not produce cachable
artifacts themselves in that they are only calling the SUBDIRS of
their lower level (like f_isoroot).

All "leaf" Makefiles will however need to implement three cache
targets in their top Makefile (for visibility):

clean-cache:
        Clean all files relating to the handling of caches.
get-cache:
        Attempt to fetch a cached artifact using a SHA1 key.
put-cache:
        Store a built artifact into the cache.

They can just implement a simple "no-op" functionality for thes
targets if they do not have any use of the caching functionality.

If they are to use the caching functionality, they must make sure to
implement a logic that produces a SHA1 hash based on the source of the
data they are to cache - for upstream repos one suggestion is to use
the commit ID of the used upstream HEAD. For examples, see the top
Makefile that implements this logic for the Fuel ISO build, taking
into consideration the commit IDs of all the upstream repositories
used.

To improve tracability, the root directory of the ISO will contain the
file gitinfo.txt, which is meant to detail the upstream repo and
commit ID used for all upstream dependencies of a build. If you are
adding additional upstream dependencies, make sure to use the
repo_info.sh tool to add this data.

The cache tool cache.sh has no notion of the cache data it is storing
- from the tools perspective, cache data is just a binary blob piped
in or out of the tool.

This blob is stored by the cache tool at the cache location as
<SHA1>.blob, together with an associated meta file <SHA1>.meta.

The cache meta file currently holds just one line:

    Expires: <epoch time>

This file is expected to be used to iterate through the cache objects
and retire those who's expiry date has passed. Currently objects will
always have an expiry date of two weeks into the future, but down the
road the "cache put" functionality may be amended with an optional
"age" argument that can set a different expiration time.

New tools in this commit:

  cache.sh - the cache logic
  repo_info.sh - the (optionally recursive) repo information logger

Change-Id: I8a40546c21febeecc9de6d82c0ceb6bc60b04205
Signed-off-by: Stefan K. Berg <stefan.k.berg@ericsson.com>

1 files changed, 35 insertions, 10 deletions

diff --git a/fuel/build/f_isoroot/f_repobuild/Makefile b/fuel/build/f_isoroot/f_repobuild/Makefile
index 03e4caecb..ad40d0b9a 100644
--- a/fuel/build/f_isoroot/f_repobuild/Makefile
+++ b/fuel/build/f_isoroot/f_repobuild/Makefile
@@ -10,17 +10,13 @@
 
 SHELL := /bin/bash
 TOP := $(shell pwd)
-DOCKNAME = fuelrepo
-DOCKVERSION = 1.0
 
 RSYNC_HOST := $(shell ./select_ubuntu_repo.sh)
 
 .PHONY: all
-all: .nailgun
+all: nailgun
 
-.nailgun:
-	sudo apt-get update
-	sudo apt-get upgrade -y
+nailgun:
 	sudo apt-get install -y rsync python python-yaml dpkg-dev openssl
 	rm -rf tmpiso tmpdir
 	mkdir tmpiso
@@ -37,16 +33,45 @@ all: .nailgun
 	sudo su - -c /opt/fuel-createmirror-6.1/fuel-createmirror
 	sudo chmod -R 755 /var/www/nailgun
 	cp -Rp /var/www/nailgun .
-	touch .nailgun
 
 .PHONY: clean
 clean:
-	# Deliberately not cleaning nailgun directory to speed up multiple builds
-	@rm -rf ../release/opnfv/nailgun fuel-createmirror_6.1*.deb
+	@rm -rf ../release/opnfv/nailgun nailgun fuel-createmirror_6.1*.deb
 
 .PHONY: release
-release:.nailgun
+release:nailgun
 	@rm -Rf ../release/opnfv/nailgun
 	@mkdir -p ../release/opnfv
 	@cp -Rp nailgun ../release/opnfv/nailgun
 
+#############################################################################
+# Cache operations - only used when building through ci/build.sh
+#############################################################################
+
+# Create a unique hash to be used for getting and putting cache, based on:
+#   - Year and week (causing the cache to be rebuilt weekly)
+#   - The contents of this Makefile
+.cacheid:
+	date +"Repocache %G%V" > .cachedata
+	sha1sum Makefile >> .cachedata
+	cat .cachedata | $(CACHETOOL) getid > .cacheid
+
+# Clean local data related to caching - called prior to ordinary build
+.PHONY: clean-cache
+clean-cache: clean
+	rm -f .cachedata .cacheid
+
+# Try to download cache - called prior to ordinary build
+.PHONY: get-cache
+get-cache: .cacheid
+	@if $(CACHETOOL) check $(shell cat .cacheid); then \
+		 $(CACHETOOL) get $(shell cat .cacheid) | tar xf -;\
+	else \
+		echo "No cache item found for $(shell cat .cacheid)" ;\
+		exit 0;\
+	fi
+
+# Store cache if not already stored - called after ordinary build
+.PHONY: put-cache
+put-cache: .cacheid
+	@tar cf - nailgun | $(CACHETOOL) put $(shell cat .cacheid)