diff options
| author |   Alexandru Avadanii <Alexandru.Avadanii@enea.com> | 2016-12-18 16:53:26 +0100 |
|---|---|---|
| committer | Alexandru Avadanii <Alexandru.Avadanii@enea.com> | 2017-01-21 18:07:28 +0100 |
| commit | e42a9b3011f96ad26f4a19db77ac44cad31a4290 (patch) | |
| tree | b6a1c7f918138dd3e4705d170dde85eb2e5aaa6a /patches/opnfv-fuel/multiarch-mirrors | |
| parent | 5c258a9ae96fd2a0c6fd56e41c4368467e931782 (diff) | |
Uplift Armband to Fuel Newton
[ Dan Andresan ]
- puppet: database: Fix Percona XtraBackup sync
[ Alexandru Avadanii ]
- disable all plugins;
- re-enable remote tracking;
- remove "Revert: Point to specific snapshot ...";
- patch context adjustments;
- obsolete "kernel-bump", linux-image-lts-xenial is now the default;
- network-checker iface state check now fixed upstream [1];
- fuel-nailgun-agent hugepage size should also check /proc/meminfo;
- fuel-nailgun-agent CPU detection for AArch64;
- nova AArch64 hugepage support is now upstream;
- obsolete Cirros Test VM direct kernel boot (by switch to AAVMF):
* f-l/0001-upload_cirros-Add-direct-kernel-boot-support.patch
* f-w/0001-direct-kernel-boot-for-cirros.patch
- rework m1.micro RAM size patch after puppet manifest split upstream;
- re-enable arch-agnostic plugins which were rebased in Fuel@OPNFV:
* f_yardstick-pluginbuild
* f_congress-pluginbuild
- do NOT retire MySQL SST provider patch series (nack: ARMBAND-186),
rebase (and keep for now) MySQL SST provider patches, as trying to
use xtrabackup-v2 revelead a regression since Colorado.3.0, and
these patches simplify troubleshooting a lot;
- AArch64: nova: libvirt: Use host-model cpu (ARMBAND-193);
- AArch64: nova: libvirt: Use pointer_model instead of use_usb_tablet;
- m1.micro RAM size insufficient for TestVM with AAVMF (s/128/256/)
- switch Cirros TestVM to AAVMF from direct kernel boot;
- backport nova libvirt driver fix for deleting instances booted with
AAVMF firmware from [2];
TODO (later):
- Include ISO build time fixes for cirros_testvm in Armband package;
TODO (ODL, later):
- test & revise leveldb patching;
- bring back Qugga patching for arm64;
- configure systemd service to automatically respawn;
[1] https://review.openstack.org/#/c/417373/
[2] https://review.openstack.org/#/c/357190/
JIRA: ARMBAND-29
JIRA: ARMBAND-32
JIRA: ARMBAND-63
JIRA: ARMBAND-88
JIRA: ARMBAND-116
JIRA: ARMBAND-118
JIRA: ARMBAND-186
JIRA: ARMBAND-193
JIRA: ARMBAND-194
JIRA: ARMBAND-195
JIRA: ARMBAND-196
JIRA: ARMBAND-197
Change-Id: Ia99022e364e61245d109cabab9d0ed7157b4d2f5
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
Signed-off-by: Dan Andresan <dan.andresan@enea.com>
Diffstat (limited to 'patches/opnfv-fuel/multiarch-mirrors')
2 files changed, 1248 insertions, 27 deletions
diff --git a/patches/opnfv-fuel/multiarch-mirrors/0000-f_repobuild-Use-packetary-to-build-partial-mirror.patch b/patches/opnfv-fuel/multiarch-mirrors/0000-f_repobuild-Use-packetary-to-build-partial-mirror.patch new file mode 100644 index 00000000..54045aa1 --- /dev/null +++ b/patches/opnfv-fuel/multiarch-mirrors/0000-f_repobuild-Use-packetary-to-build-partial-mirror.patch @@ -0,0 +1,1221 @@ +From: Alexandru Avadanii <Alexandru.Avadanii@enea.com> +Date: Wed, 6 Jul 2016 17:03:49 +0200 +Subject: [PATCH] f_repobuild: Use packetary to build partial mirror + +Packetary [1] will replace fuel-mirror in upstream Fuel. +fuel-mirror is using packetary under the hood already, so the +transition should not be too hard. + +Adapt OPNFV Ubuntu partial mirror build: +- obsolete opnfv-config.yaml (packetary uses CLI args for this); +- new opnfv_config.yaml will hold old ubuntu.yaml data, split at + build runtime into: + * mos_<arch>_mirror.yaml - MOS mirror cfg, only for deps analysis; + * ubuntu_<arch>_mirror.yaml - Ubuntu mirror cfg, for pkgs download; + * ubuntu_<arch>_packages.yaml - Ubuntu packages to download; + +While the consumers of our mirror still use `debootstrap`, which only +supports the "main" repo component, make sure the "main" package set +is complete, in the scope of: + * ubuntu_<arch>_mirror_main.yaml - Ubuntu [main only] mirror; + * ubuntu_<arch>_packages_main.yaml - Ubuntu packages for [main]; + +New workflow (executed for EACH architecture): +1. Collect bootstrap package deps from <fuel_bootstrap_cli.yaml>; +2. Collect all fixture release pkgs from fuel-web's <openstack.yaml>; +3. Add OPNFV extra pkg deps from "opnfv_config.yaml" + (based on old fuel-mirror) + + Currently, this is based on old fuel-mirror's + <fuel-mirror/contrib/fuel_mirror/data/ubuntu.yaml>, holding a + hardcoded list of package deps; plus eventual extra OPNFV packages, + like additional kernels, bootloaders etc. + Since we now inspect MOS repos for deps with packetary, this old + list should be easy to throw away as soon as we confirm each item + is directly required by another, already existing package in the + mirror. For now, keep the old list as safety net for missing deps. + + NOTE: Dependencies up to and including this group should also + be satisfiable from Ubuntu "main" repo component only. + +4. Append plugin dependencies to the package list; + +5. Run `packetary unresolved` for MOS repos, gathering MOS deps that + should be fetched from Ubuntu. The new packages will be appended + to the predefined list from old "ubuntu.yaml"; + +6. Run `packetary clone` to download all required pkgs for the + partial Ubuntu mirror. + +7. IF repo component merging is disabled, run `packetary clone` only + on Ubuntu[main] repo components to download any (still) missing + dependencies for `debootstrap`, which only looks at this component, + and not the whole mirror. + +8. IF repo component merging is enabled, run first `dpkg-scanpackages` + to filter out old duplicate versions of packages, then run + `packetary create` to merge all repo comps into a single "main". + +v1 -> v2: +- multiarch support (activated via UBUNTU_ARCH envvar); +- fixed debootstrap missing deps in "main" component of Ubuntu mirror; +- factored out some hardcodes into variables/functions; +- add .gitignore file; +- move generated config files to "opnfv_config" subdir; +- add arm64 Ubuntu repo URL (ports.ubuntu.com is separated from the + main Ubuntu mirror, archive.ubuntu.com, which only holds x86 pkgs); + +v3 -> v4: +- introduce env var "MIRROR_UBUNTU_MERGE" to control local mirror + repo-component merging into a single "main"; +- enable mirror repo-component merging by default, edit + fuel_bootstrap_cli.yaml accordingly; + +v4 -> v6: +- edit dea_base to use only main mirror; +- fix duplicated logs; +- remove `-q` flag for `create`; + +v6 -> v7: +- rebase for Newton fuel_bootstrap_cli.yaml (s/trusty/xenial/); +- include OPNFV plugin deps in local Ubuntu mirror; +- add `vgabios` dependency, required for arm64 guest firmware; +- remove all fuel-mirror references; +- merge old fuel-mirror pkglist to OPNFV pkglist; +- update packetary commit to include upstream bugfixes [2, 3]; + +v7 -> v11: +- re-enable f_repobuild after Newton rebase; +- fix leftover "trusty" ref to "xenial"; +- force f_repobuild rebuild to bypass cache (by editing Makefile); +- extend f_repobuild .cacheid to include scripts and OPNFV config; + +v11 -> v12: +- fix install/install.sh old refs to "trusty"; +- fix out of date select_ubuntu_repo.s refs to "trusty"; + +v12 -> v13: +- fix leftover mos9.0 / fuel-mirror references; +- fix most issues reported by `pylint opnfv_mirror_ubuntu.py`; + +v13 -> v14: +- Sync fuel_bootstrap_cli.yaml with settings.yaml.sample from 10.0: + [4] Stub pass for "Get root password for build image from settings" + [5] Sync pkg list with "Switch to Ubuntu Xenial" + [6] "Add puppet and daemonize to default packages list" + [7] "Delete kernel debug related parameter for performance" + [8] "Add to bootstrap image dependencies for network checker" +- Remove obsolete Trusty kernel images from pkg lists; +- Don't break when plugins set is empty; +- Fix missing "vim" package, which was resolved using secondary + provider packages like GUI-enabled vim flavors; + The proposed fix is quite sensitive, also cutting the final + mirror size in half (!), and allowing us to remove a previous + hacky workaround (re-running packetary only for main). +- Remove "main"-only packetary run (hacky), since the new fix for "vim" + also takes care of packetary solving dependencies from "universe". +- Add OPNFV blacklisting to workaround "upstart" package being installed, + together with "systemd" (having them both installed confuses puppet); + +NOTE: Without filtering out old package verisons using +`dpkg-scanpackages`, bootstrap build cannot solve all deps. + +FIXME: Packetary solves missing dependecies by also accepting +different packages that provide the same package name (e.g. "ifupdown" +is satisfied by "netscript" package). + +FIXME: Repo component merging is sort of slow, since packetary insists +on copying the source files to the destination dir even if they are +pointing to the same repo. + +FIXME: Packetary `create` uses a different directory scheme for the +created mirror than the upstream Ubuntu, e.g.: +[p] mirrors/ubuntu/pool/main/l/lvm2_2.02.98-6ubuntu2_amd64.deb +[U] mirrors/ubuntu/pool/main/l/lvm2/lvm2_2.02.98-6ubuntu2_amd64.deb + +This disencourages creating the "merged" mirror in the same location +as the source partial mirror. + +[1] https://wiki.openstack.org/wiki/Packetary +[2] https://review.openstack.org/#/c/392936/ +[3] https://review.openstack.org/#/c/392937/ +[4] https://review.openstack.org/#/c/277353/ +[5] https://review.openstack.org/#/c/345516/ +[6] https://review.openstack.org/#/c/344909/ +[7] https://review.openstack.org/#/c/400165/ +[8] https://review.openstack.org/#/c/415912/ + +JIRA: FUEL-218 +JIRA: FUEL-223 +JIRA: ARMBAND-169 +JIRA: ARMBAND-185 + +Change-Id: If2ee86f348b7683c83bfaf686baba4f1b8f555f0 +Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com> +--- + build/config.mk | 4 + + build/f_isoroot/Makefile | 8 +- + build/f_isoroot/f_repobuild/.gitignore | 4 + + ...s_tree-exact-match-trumps-other-providers.patch | 75 ++++++ + build/f_isoroot/f_repobuild/Makefile | 45 ++-- + build/f_isoroot/f_repobuild/config.mk | 23 ++ + .../f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml | 41 ++- + build/f_isoroot/f_repobuild/opnfv_config.yaml | 166 ++++++++++++ + build/f_isoroot/f_repobuild/opnfv_mirror_conf.py | 57 ----- + build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py | 279 +++++++++++++++++++++ + build/f_isoroot/f_repobuild/select_ubuntu_repo.sh | 41 +-- + build/install/apt-ftparchive-udeb.conf | 4 +- + build/install/install.sh | 14 +- + build/install/uninstall.sh | 16 +- + deploy/config/dea_base.yaml | 16 +- + 15 files changed, 636 insertions(+), 157 deletions(-) + create mode 100644 build/f_isoroot/f_repobuild/.gitignore + create mode 100644 build/f_isoroot/f_repobuild/0001-packages_tree-exact-match-trumps-other-providers.patch + create mode 100644 build/f_isoroot/f_repobuild/config.mk + create mode 100644 build/f_isoroot/f_repobuild/opnfv_config.yaml + delete mode 100755 build/f_isoroot/f_repobuild/opnfv_mirror_conf.py + create mode 100755 build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py + +diff --git a/build/config.mk b/build/config.mk +index 56317cb..5448c5e 100644 +--- a/build/config.mk ++++ b/build/config.mk +@@ -17,6 +17,10 @@ FUEL_MAIN_TAG = master + MOS_VERSION = 10.0 + OPENSTACK_VERSION = newton-10.0 + ++# List of space-separated Ubuntu architectures supported with current build ++# Format: same as `dpkg-architecture -qDEB_HOST_ARCH` ++# NOTE: Currently only amd64 is supported by Fuel@OPNFV. Armband adds arm64. ++export UBUNTU_ARCH ?= amd64 + + ############################################################################## + # Fuel components pinning / remote tracking; use submodules from f_repos +diff --git a/build/f_isoroot/Makefile b/build/f_isoroot/Makefile +index e189e16..b4ef1d0 100644 +--- a/build/f_isoroot/Makefile ++++ b/build/f_isoroot/Makefile +@@ -17,16 +17,12 @@ NON_10-0_REBASED_PLUGINS = f_odlpluginbuild f_onosfwpluginbuild f_ovsnfv-dpdk-pl + + # If the BUILD_FUEL_PLUGINS environment variable is set, only build the plugins + # indicated therein. +-# Temporarilu disabling f_repobuild that currently have some issues on +-# Fuel Newton. + ifdef BUILD_FUEL_PLUGINS + export BUILD_FUEL_PLUGINS +-#SUBDIRS = f_kscfg f_bootstrap f_isolinux f_repobuild $(BUILD_FUEL_PLUGINS) +-SUBDIRS = f_kscfg f_bootstrap f_isolinux $(BUILD_FUEL_PLUGINS) ++SUBDIRS = f_kscfg f_bootstrap f_isolinux f_repobuild $(BUILD_FUEL_PLUGINS) + $(warning Overriding plugin build selection to $(BUILD_FUEL_PLUGINS)) + else +-#SUBDIRS = f_kscfg f_bootstrap f_isolinux f_repobuild $(PLUGINS) +-SUBDIRS = f_kscfg f_bootstrap f_isolinux $(PLUGINS) ++SUBDIRS = f_kscfg f_bootstrap f_isolinux f_repobuild $(PLUGINS) + endif + SUBCLEAN = $(addsuffix .clean,$(SUBDIRS)) + +diff --git a/build/f_isoroot/f_repobuild/.gitignore b/build/f_isoroot/f_repobuild/.gitignore +new file mode 100644 +index 0000000..09baca8 +--- /dev/null ++++ b/build/f_isoroot/f_repobuild/.gitignore +@@ -0,0 +1,4 @@ ++packetary ++fuel-web ++nailgun ++opnfv_config +diff --git a/build/f_isoroot/f_repobuild/0001-packages_tree-exact-match-trumps-other-providers.patch b/build/f_isoroot/f_repobuild/0001-packages_tree-exact-match-trumps-other-providers.patch +new file mode 100644 +index 0000000..1df292c +--- /dev/null ++++ b/build/f_isoroot/f_repobuild/0001-packages_tree-exact-match-trumps-other-providers.patch +@@ -0,0 +1,75 @@ ++From: Alexandru Avadanii <Alexandru.Avadanii@enea.com> ++Date: Sun, 15 Jan 2017 00:40:15 +0100 ++Subject: [PATCH] packages_tree: exact match trumps other providers ++ ++Ubuntu frequently has multiple providers for the same package name, ++e.g. "vim" is provided by "vim", "vim-gnome" etc. ++ ++Below is a sample dump of packetary's list of matches for a generic ++"vim" package requirement, without a version enforced, based on ++current Xenial repos: ++ ++vim-gtk3-py2 (2:7.4.1689-3ubuntu1) ++vim-gnome (2:7.4.1689-3ubuntu1) ++vim-gtk (2:7.4.1689-3ubuntu1) ++vim-nox (2:7.4.1689-3ubuntu1) ++vim-gtk-py2 (2:7.4.1689-3ubuntu1) ++vim-gnome-py2 (2:7.4.1689-3ubuntu1) ++vim-nox-py2 (2:7.4.1689-3ubuntu1) ++vim (2:7.4.1689-3ubuntu1) ++vim-gtk3 (2:7.4.1689-3ubuntu1) ++vim-athena (2:7.4.1689-3ubuntu1) ++vim-athena-py2 (2:7.4.1689-3ubuntu1) ++vim-gtk (2:7.4.1689-3ubuntu1.2) ++vim-gnome-py2 (2:7.4.1689-3ubuntu1.2) ++vim-gtk-py2 (2:7.4.1689-3ubuntu1.2) ++vim-nox (2:7.4.1689-3ubuntu1.2) ++vim-nox-py2 (2:7.4.1689-3ubuntu1.2) ++vim-gtk3 (2:7.4.1689-3ubuntu1.2) ++vim (2:7.4.1689-3ubuntu1.2) ++vim-athena (2:7.4.1689-3ubuntu1.2) ++vim-athena-py2 (2:7.4.1689-3ubuntu1.2) ++vim-gnome (2:7.4.1689-3ubuntu1.2) ++vim-gtk3-py2 (2:7.4.1689-3ubuntu1.2) ++ ++Currently, packetary solves the "vim" relation by using the last item ++in the sorted list, in this case "vim-gtk3-py2"; instead of using the ++exact package name match "vim". ++ ++This leads to our final mirror clone missing the "vim" package, and ++inherintely failing to build bootstrap/target images using only the ++partial Ubuntu mirror in OPNFV ISO. ++ ++The proposed fix is to first check the list for an exact name match, ++which would trump any secondary providers, even if they have a ++higher package version. ++ ++Change-Id: I7279aa6526ff9133829be2e316932c9b052c7814 ++Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com> ++--- ++ packetary/objects/packages_tree.py | 5 +++++ ++ 1 file changed, 5 insertions(+) ++ ++diff --git a/packetary/objects/packages_tree.py b/packetary/objects/packages_tree.py ++index f02df59..6081fa9 100644 ++--- a/packetary/objects/packages_tree.py +++++ b/packetary/objects/packages_tree.py ++@@ -18,6 +18,7 @@ ++ ++ from collections import defaultdict ++ +++import re ++ import six ++ ++ from packetary.objects.index import Index ++@@ -57,6 +58,10 @@ class PackagesTree(object): ++ """ ++ candidates = self.find_all(name, version_range) ++ if len(candidates) > 0: +++ # exact pkg name match trumps other providers' versions +++ for candidate in reversed(candidates): +++ if re.match('{0}\s.*'.format(re.escape(name)), str(candidate)): +++ return candidate ++ # we return candidates in sorted order, so let's take the highest ++ return candidates[-1] ++ return None +diff --git a/build/f_isoroot/f_repobuild/Makefile b/build/f_isoroot/f_repobuild/Makefile +index 8beb882..76aadf9 100644 +--- a/build/f_isoroot/f_repobuild/Makefile ++++ b/build/f_isoroot/f_repobuild/Makefile +@@ -1,7 +1,8 @@ + ############################################################################## +-# Copyright (c) 2015 Ericsson AB and others. ++# Copyright (c) 2015,2016 Ericsson AB, Enea AB and others. + # stefan.k.berg@ericsson.com + # jonas.bjurel@ericsson.com ++# Alexandru.Avadanii@enea.com + # All rights reserved. This program and the accompanying materials + # are made available under the terms of the Apache License, Version 2.0 + # which accompanies this distribution, and is available at +@@ -10,41 +11,39 @@ + + SHELL := /bin/bash + TOP := $(shell pwd) +-TMP_ROOT_DIR := $(shell echo "$(MIRROR_UBUNTU_ROOT)" | cut -d "/" -f2) + + include ../../config.mk ++include config.mk + + export MOS_VERSION +-export OPENSTACK_VERSION ++export MIRROR_UBUNTU_OPNFV_PATH:=$(TOP)/nailgun/mirrors/ubuntu + + .PHONY: all + all: nailgun + + nailgun: +- sudo apt-get install -y git libxml2-dev libxslt-dev python-dev python-pip libz-dev libyaml-dev createrepo python-yaml +- rm -Rf nailgun +- sudo mkdir -p /var/www/nailgun +- ln -sf ${F_SUBMOD_DIR}/fuel-mirror fuel-mirror +- sudo pip install -U -r ./fuel-mirror/requirements.txt +- sudo pip install ./fuel-mirror +- sudo pip install ./fuel-mirror/contrib/fuel_mirror +- ./opnfv_mirror_conf.py +- sudo fuel-mirror --debug --config ./opnfv-config.yaml create --group ubuntu --pattern=ubuntu +- sudo chmod -R 755 /var/www/nailgun +- cp -Rp /var/www/nailgun . +- # In the end we want to have ubuntu repository in mirrors/ubuntu directory +- -if [ "$(MIRROR_UBUNTU_ROOT)" != "/ubuntu/" ]; then \ +- mkdir -p nailgun/mirrors/ubuntu;\ +- mv nailgun/mirrors$(MIRROR_UBUNTU_ROOT)* nailgun/mirrors/ubuntu;\ +- [ "$(MIRROR_UBUNTU_ROOT)" != "/" ] && rm -rf nailgun/mirrors/$(TMP_ROOT_DIR);\ ++ sudo apt-get install -y createrepo git libxml2-dev libxslt1-dev \ ++ python-dev zlib1g-dev ++ rm -Rf nailgun packetary opnfv_config && mkdir opnfv_config ++ # We will analyze fuel-web's fixture files for package lists ++ ln -sf ${F_SUBMOD_DIR}/fuel-web fuel-web ++ git clone --quiet $(PACKETARY_REPO) ++ if [ -n $(PACKETARY_COMMIT) ]; then \ ++ git -C packetary checkout $(PACKETARY_COMMIT); \ + fi ++ # FIXME(alav): Temporary, until we decide this is upstreamable ++ git -C packetary am $(TOP)/*.patch ++ sudo pip install -U -r ./packetary/requirements.txt ++ sudo pip install -U ./packetary ++ # Handle config and mirror build in one place ++ ./opnfv_mirror_ubuntu.py + # Store artifact in cache straight away if caching is enabled + # (no .cacheid will be present unless this is a cached build) + test -f .cacheid && $(MAKE) -f Makefile put-cache || exit 0 + + .PHONY: clean + clean: +- @rm -rf ../release/opnfv/nailgun nailgun fuel-mirror opnfv-config.yaml ubuntu.yaml ++ @rm -rf ../release/opnfv/nailgun nailgun packetary fuel-web opnfv_config + + .PHONY: release + release:nailgun +@@ -59,10 +58,12 @@ release:nailgun + + # Create a unique hash to be used for getting and putting cache, based on: + # - Year and week (causing the cache to be rebuilt weekly) +-# - The contents of this Makefile ++# - The contents of this Makefile + all sh,mk,py,yaml files in CWD ++# - repo packages fingerprint ++# - repo arch list + .cacheid: + date +"Repocache %G%V" > .cachedata +- sha1sum Makefile >> .cachedata ++ sha1sum Makefile *.{sh,mk,py,yaml} >> .cachedata + $(CACHETOOL) packages >> .cachedata + echo -n $(UBUNTU_ARCH) | sha1sum | awk {'print $$1'} >> .cachedata + cat .cachedata | $(CACHETOOL) getid > .cacheid +diff --git a/build/f_isoroot/f_repobuild/config.mk b/build/f_isoroot/f_repobuild/config.mk +new file mode 100644 +index 0000000..9852093 +--- /dev/null ++++ b/build/f_isoroot/f_repobuild/config.mk +@@ -0,0 +1,23 @@ ++############################################################################## ++# Copyright (c) 2016 Ericsson AB, Enea AB and others. ++# stefan.k.berg@ericsson.com ++# jonas.bjurel@ericsson.com ++# Alexandru.Avadanii@enea.com ++# All rights reserved. This program and the accompanying materials ++# are made available under the terms of the Apache License, Version 2.0 ++# which accompanies this distribution, and is available at ++# http://www.apache.org/licenses/LICENSE-2.0 ++############################################################################## ++ ++# Use a recent master commit, since tags/branches are not yet mature ++export PACKETARY_REPO?=https://github.com/openstack/packetary ++export PACKETARY_COMMIT?=3021c001561b4baef352bf0b881d064ac687cc20 ++ ++# arm64 Ubuntu mirror is separated from archive.ubuntu.com ++export MIRROR_UBUNTU_URL_arm64=http://ports.ubuntu.com/ubuntu-ports/ ++export MIRROR_UBUNTU_ROOT_arm64=ubuntu-ports ++ ++# Merge all local mirror repo components/section into single "main" ++# NOTE: When changing this, make sure to also update all consumer config, like: ++# - fuel_bootstrap_cli.yaml ++export MIRROR_UBUNTU_MERGE=true +diff --git a/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml b/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml +index fcf4257..81ca6eb 100644 +--- a/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml ++++ b/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml +@@ -8,19 +8,18 @@ + ############################################################################## + --- + root_ssh_authorized_file: /root/.ssh/id_rsa.pub +- extend_kopts: "biosdevname=0 net.ifnames=1 debug ignore_loglevel log_buf_len=10M print_fatal_signals=1 LOGLEVEL=8" +- ubuntu_release: trusty ++ extend_kopts: "biosdevname=0 net.ifnames=1" ++ ubuntu_release: xenial + extra_dirs: +- - /usr/share/fuel_bootstrap_cli/files/trusty ++ - /usr/share/fuel_bootstrap_cli/files/xenial + output_dir: /tmp/ +- kernel_flavor: linux-image-generic-lts-trusty ++ kernel_flavor: linux-image-generic-lts-xenial + packages: ++ - daemonize + - fuel-agent + - hwloc +- - hpsa-dkms + - i40e-dkms + - linux-firmware +- - linux-firmware-nonfree + - linux-headers-generic + - live-boot + - live-boot-initramfs-tools +@@ -33,13 +32,23 @@ + - nailgun-mcagents + - network-checker + - ntp ++ - ntpdate + - openssh-client + - openssh-server ++ - puppet + - squashfs-tools + - ubuntu-minimal + - vim + - wget + - xz-utils ++ # NOTE(el): Packages required for new generation ++ # network checker to be run without an access ++ # to repositories. ++ - sysfsutils ++ - bridge-utils ++ - ifenslave ++ - irqbalance ++ - iputils-arping + bootstrap_images_dir: /var/www/nailgun/bootstraps + active_bootstrap_symlink: /var/www/nailgun/bootstraps/active_bootstrap + flavor: ubuntu +@@ -47,29 +56,19 @@ + https_proxy: "" + repos: + - name: ubuntu +- section: "main universe multiverse" ++ section: "main" + uri: "http://127.0.0.1:8080/mirrors/ubuntu" + priority: +- suite: trusty +- type: deb +- - name: ubuntu-updates +- section: "main universe multiverse" +- uri: "http://127.0.0.1:8080/mirrors/ubuntu" +- priority: +- suite: trusty-updates +- type: deb +- - name: ubuntu-security +- section: "main universe multiverse" +- uri: "http://127.0.0.1:8080/mirrors/ubuntu" +- priority: +- suite: trusty-security ++ suite: xenial + type: deb + - name: mos + section: "main restricted" + uri: "http://127.0.0.1:8080/ubuntu/x86_64" + priority: 1050 +- suite: mos9.0 ++ suite: mos10.0 + type: deb + skip_default_img_build: false + direct_repo_addresses: + - "127.0.0.1" ++ # User can provide default hashed root password for bootstrap image ++ # hashed_root_password: "$6$IInX3Cqo$5xytL1VZbZTusOewFnG6couuF0Ia61yS3rbC6P5YbZP2TYclwHqMq9e3Tg8rvQxhxSlBXP1DZhdUamxdOBXK0." +diff --git a/build/f_isoroot/f_repobuild/opnfv_config.yaml b/build/f_isoroot/f_repobuild/opnfv_config.yaml +new file mode 100644 +index 0000000..b8d3441 +--- /dev/null ++++ b/build/f_isoroot/f_repobuild/opnfv_config.yaml +@@ -0,0 +1,166 @@ ++############################################################################## ++# Copyright (c) 2016 Enea AB and others. ++# Alexandru.Avadanii@enea.com ++# All rights reserved. This program and the accompanying materials ++# are made available under the terms of the Apache License, Version 2.0 ++# which accompanies this distribution, and is available at ++# http://www.apache.org/licenses/LICENSE-2.0 ++############################################################################## ++ ++# REPO definitions ++ ++# Based on old fuel-mirror data [1], adapted for OPNFV and packetary. ++# NOTE: 'uri' field will be added by opnfv_mirror_ubuntu.py from ENV. ++# [1] https://github.com/openstack/fuel-mirror/blob/stable/mitaka/ ++# contrib/fuel_mirror/data/ubuntu.yaml ++ ++# Main is a required parameter which defines what repository will be used ++# for images creation and that mirror should contain all packages for minimal ++# system creation. ++groups: ++ ubuntu: ++ - name: "ubuntu" ++ main: true ++ suite: "xenial" ++ section: ++ - "main" ++ - "multiverse" ++ - "restricted" ++ - "universe" ++ type: "deb" ++ priority: null ++ ++ - name: "ubuntu-updates" ++ suite: "xenial-updates" ++ section: ++ - "main" ++ - "multiverse" ++ - "restricted" ++ - "universe" ++ type: "deb" ++ priority: null ++ ++ - name: "ubuntu-security" ++ suite: "xenial-security" ++ section: ++ - "main" ++ - "multiverse" ++ - "restricted" ++ - "universe" ++ type: "deb" ++ priority: null ++ ++ mos: ++ - name: "mos" ++ suite: "mos$mos_version" ++ section: ++ - "main" ++ - "restricted" ++ type: "deb" ++ priority: 1000 ++ ++ - name: "mos-updates" ++ suite: "mos$mos_version-updates" ++ section: ++ - "main" ++ - "restricted" ++ type: "deb" ++ priority: 1000 ++ ++ - name: "mos-security" ++ suite: "mos$mos_version-security" ++ section: ++ - "main" ++ - "restricted" ++ type: "deb" ++ priority: 1000 ++ ++ - name: "mos-holdback" ++ suite: "mos$mos_version-holdback" ++ section: ++ - "main" ++ - "restricted" ++ type: "deb" ++ priority: 1000 ++ ++# PACKAGES ++ ++# This section lists packages that should be present in the local Ubuntu mirror, ++# but are not direct dependencies of any other packages from MOS or Ubuntu. ++# e.g.: additional kernels, bootloaders etc. ++ ++packages: ++ - name: "vgabios" ++# Packages are required to build bootstrap images for a system. ++# The mirror should contiain such packages in addition to local mirror. ++ - name: "acpi-support" ++ - name: "anacron" ++ - name: "aptitude" ++ - name: "atop" ++ - name: "acct" ++ - name: "bash-completion" ++ - name: "bc" ++ - name: "build-essential" ++ - name: "cloud-init" ++ - name: "conntrackd" ++ - name: "cpu-checker" ++ - name: "cpufrequtils" ++ - name: "debconf-utils" ++ - name: "devscripts" ++ - name: "fping" ++ - name: "git" ++ - name: "grub-pc" ++ - name: "htop" ++ - name: "hwloc" ++ - name: "ifenslave" ++ - name: "iperf" ++ - name: "iptables-persistent" ++ - name: "irqbalance" ++ - name: "language-pack-en" ++ - name: "libapache2-mod-fastcgi" ++ - name: "libnss3-tools" ++ - name: "linux-headers-generic-lts-xenial" ++ - name: "linux-image-generic-lts-xenial" ++ - name: "live-boot" ++ - name: "livecd-rootfs" ++ - name: "mc" ++ - name: "memcached" ++ - name: "monit" ++ - name: "msmtp-mta" ++ - name: "multipath-tools" ++ - name: "multipath-tools-boot" ++ - name: "nginx" ++ - name: "ntp" ++ - name: "openssh-server" ++ - name: "percona-toolkit" ++ - name: "percona-xtrabackup" ++ - name: "pm-utils" ++ - name: "puppet" ++ - name: "python-lesscpy" ++ - name: "python-pip" ++ - name: "rsyslog-gnutls" ++ - name: "rsyslog-relp" ++ - name: "screen" ++ - name: "squashfs-tools" ++ - name: "swift-plugin-s3" ++ - name: "sysfsutils" ++ - name: "sysstat" ++ - name: "telnet" ++ - name: "tmux" ++ - name: "traceroute" ++ - name: "ubuntu-standard" ++ - name: "vim" ++ - name: "virt-what" ++ - name: "xinetd" ++ - name: "xmlstarlet" ++ - name: "tftpd-hpa" ++ - name: "syslinux" ++ ++# OPNFV BLACKLIST ++ ++# Packetary's dependency solving mechanism brings in more than one alternative ++# For example, "Depends: upstart | systemd-sysv" brings both packages, which ++# leads to debootstrap using "upstart" (1st option available in local repo). ++ ++opnfv_blacklist: ++ - name: "upstart" +diff --git a/build/f_isoroot/f_repobuild/opnfv_mirror_conf.py b/build/f_isoroot/f_repobuild/opnfv_mirror_conf.py +deleted file mode 100755 +index 1c7eb59..0000000 +--- a/build/f_isoroot/f_repobuild/opnfv_mirror_conf.py ++++ /dev/null +@@ -1,57 +0,0 @@ +-#!/usr/bin/env python +-############################################################################## +-# Copyright (c) 2015 Ericsson AB and others. +-# mskalski@mirantis.com +-# All rights reserved. This program and the accompanying materials +-# are made available under the terms of the Apache License, Version 2.0 +-# which accompanies this distribution, and is available at +-# http://www.apache.org/licenses/LICENSE-2.0 +-############################################################################## +-import os +-import yaml +- +-current_snapshot = os.environ["LATEST_TARGET_UBUNTU"] +-mos_version = os.environ['MOS_VERSION'] +-openstack_version = os.environ['OPENSTACK_VERSION'] +-mos_ubuntu = os.environ['MIRROR_MOS_UBUNTU'] +-mos_ubuntu_root = os.environ['MIRROR_MOS_UBUNTU_ROOT'] +-mirror_ubuntu = os.environ['MIRROR_UBUNTU_URL'] +-if os.environ.get('BUILD_FUEL_PLUGINS'): +- plugins = os.environ['BUILD_FUEL_PLUGINS'] +-else: +- plugins = os.environ['PLUGINS'] +- +- +-configuration_file = open('fuel-mirror/contrib/fuel_mirror/etc/config.yaml').read() +-conf = yaml.load(configuration_file) +-conf['pattern_dir'] = '.' +-conf['openstack_version'] = openstack_version +-conf['mos_version'] = mos_version +- +-with open('opnfv-config.yaml', 'w') as outfile: +- outfile.write( yaml.dump(conf, default_flow_style=False) ) +- +-pattern_file = open('fuel-mirror/contrib/fuel_mirror/data/ubuntu.yaml').read() +-pattern = yaml.load(pattern_file) +-pattern['mos_baseurl'] = "http://{}{}".format(mos_ubuntu, mos_ubuntu_root) +-pattern['ubuntu_baseurl'] = mirror_ubuntu +-for group in pattern['groups']['mos']: +- group['uri'] = pattern['mos_baseurl'] +-for group in pattern['groups']['ubuntu']: +- group['uri'] = pattern['ubuntu_baseurl'] +- +-for plugin in plugins.split(): +- path = "../{}/packages.yaml".format(plugin) +- if os.path.isfile(path): +- f = open(path).read() +- plugin_yaml = yaml.load(f) +- plugin_set = set(plugin_yaml['packages']) +- main_set = set(pattern['packages']) +- new_packages = plugin_set - main_set +- print "Plugin {} require new packages: {}".format(plugin, ', '.join(new_packages)) +- pattern['packages'] = pattern['packages'] + list(new_packages) +- +-pattern['requirements']['ubuntu'] = pattern['packages'] +- +-with open('ubuntu.yaml', 'w') as outfile: +- outfile.write( yaml.safe_dump(pattern, default_flow_style=False) ) +diff --git a/build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py b/build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py +new file mode 100755 +index 0000000..1a603ee +--- /dev/null ++++ b/build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py +@@ -0,0 +1,279 @@ ++#!/usr/bin/env python ++############################################################################## ++# Copyright (c) 2015,2016 Ericsson AB, Mirantis Inc., Enea AB and others. ++# mskalski@mirantis.com ++# Alexandru.Avadanii@enea.com ++# All rights reserved. This program and the accompanying materials ++# are made available under the terms of the Apache License, Version 2.0 ++# which accompanies this distribution, and is available at ++# http://www.apache.org/licenses/LICENSE-2.0 ++############################################################################## ++ ++"""Build multiarch partial local Ubuntu mirror using packetary""" ++ ++############################################################################## ++# Design quirks / workarounds: ++# 1. Fuel-agent uses `debootstrap` to build bootstrap and target chroots from ++# the local mirror; which only uses the "main" component from the first ++# repository, i.e. does not include "updates"/"security". ++# In order to fullfill all debootstrap dependencies in "main" repo, we will ++# do an extra packetary run using a reduced scope: ++# - only "main" component of the first mirror; ++# - reduced package dependency list (without MOS/OPNFV plugin deps). ++# 2. If repo structure is not mandatory to be in sync with official mirrors, ++# we can mitigate the issue by "merging" all repo-components into a single ++# "main". ++############################################################################## ++# Mirror build steps (for EACH architecture in UBUNTU_ARCH): ++# 1. Collect bootstrap package deps from <fuel_bootstrap_cli.yaml>; ++# 2. Collect all fixture release packages from fuel-web's <openstack.yaml>; ++# 3. Parse new "opnfv_config.yaml" list of packages (from old fuel-mirror); ++# 4. Inherit enviroment variable(s) for mirror URLs, paths etc. ++# - Allow arch-specific overrides for each env var; ++# 5. Mirror config is defined based on common config + OPNFV overrides; ++# - Convert old configuration format to packetary style where needed; ++# 6. Package lists are defined based on common config + OPNFV deps; ++# - Keep track of "main" packages separately, required by debootstrap; ++# 7. Clone/update all mirror components; ++# 8. IF mirror merging is disabled: ++# - Clone/update "main" mirror component (fix missing debootstrap deps); ++# 9. IF mirror merging is enabled: ++# - Use `dpkg-scanpackages` to filter out old versions of duplicate pkgs; ++# - Run `packetary create` on the set of downloaded packages, merging ++# them on the fly into a single-component mirror; ++############################################################################## ++ ++from copy import deepcopy ++import os ++import shutil ++import sys ++import yaml ++from contextlib import contextmanager ++from cStringIO import StringIO ++from packetary.cli.app import main ++ ++@contextmanager ++def capture_stdout(output): ++ """Context manager for capturing stdout""" ++ stdout = sys.stdout ++ sys.stdout = output ++ yield ++ sys.stdout = stdout ++ ++# FIXME: Find a better approach for eliminating duplicate logs than this ++def force_logger_reload(): ++ """Force logger reload (ugly hack to prevent log duplication)""" ++ for mod in sys.modules.keys(): ++ if mod.startswith('logging'): ++ try: ++ reload(sys.modules[mod]) ++ except: ++ pass ++ ++def get_unres_pkgs(architecture, cfg_mirror): ++ """Determine missing package dependecies for a mirror defition""" ++ unresolved_pkgs = list() ++ packetary_output = StringIO() ++ with capture_stdout(packetary_output): ++ main('unresolved -a {0} -r {1} -c name version --sep ;' ++ .format(_ARCH[architecture], cfg_mirror).split(' ')) ++ for dep_pkg in packetary_output.getvalue().splitlines(): ++ if dep_pkg.startswith('#'): ++ continue ++ dep = dep_pkg.split(';') ++ unresolved_pkgs += [{'name': dep[0], 'version': dep[1]}] ++ force_logger_reload() ++ return unresolved_pkgs ++ ++def from_legacy_pkglist(legacy_pkglist): ++ """Package list conversion from `old fuel-mirror` to `packetary` style""" ++ pkglist = list() ++ for pkg in legacy_pkglist: ++ pkglist += [{'name': pkg}] ++ return pkglist ++ ++def to_legacy_pkglist(pkglist): ++ """Package list conversion from `packetary` style to `old fuel-mirror`""" ++ legacy_pkglist = list() ++ for pkg in pkglist: ++ legacy_pkglist.append(pkg['name']) ++ return legacy_pkglist ++ ++def legacy_diff(base_pkglist, new_pkglist, requester, architecture): ++ """Package list diff (old format)""" ++ diff_set = set(new_pkglist) ++ if base_pkglist: ++ diff_set -= set(base_pkglist) ++ if diff_set: ++ print(' * {0} requires new packages for architecture [{1}]: {2}' ++ .format(requester, architecture, ', '.join(diff_set))) ++ return list(diff_set) ++ ++def do_local_repo(architecture, cfg_repo, cfg_packages_paths): ++ """Create single-component local repo (one architecture per call)""" ++ # Packetary does not use a global config file, so pass old settings here. ++ main('create -t deb -a {0} --repository {1} --package-files {2}' ++ ' --ignore-errors-num 2 --retries-num 3 --threads-num 10' ++ .format(_ARCH[architecture], cfg_repo, cfg_packages_paths).split(' ')) ++ force_logger_reload() ++ ++def do_partial_mirror(architecture, cfg_mirror, cfg_packages): ++ """Clone partial local mirror (one architecture per call)""" ++ # Note: '-d .' is ignored, as each mirror defines its own path. ++ main('clone -t deb -a {0} -r {1} -R {2} -d .' ++ ' --ignore-errors-num 2 --retries-num 3 --threads-num 10' ++ .format(_ARCH[architecture], cfg_mirror, cfg_packages).split(' ')) ++ force_logger_reload() ++ ++def write_cfg_file(cfg_mirror, data): ++ """Write configuration (yaml) file (package list / mirror defition)""" ++ with open(cfg_mirror, 'w') as outfile: ++ outfile.write(yaml.safe_dump(data, default_flow_style=False)) ++ ++def get_env(env_var, architecture=None): ++ """Evaluate architecture-specific overrides of env vars""" ++ if architecture: ++ env_var_arch = '{0}_{1}'.format(env_var, architecture) ++ if os.environ.get(env_var_arch): ++ return os.environ[env_var_arch] ++ if os.environ.get(env_var): ++ return os.environ[env_var] ++ return None ++ ++# Architecture name mapping (dpkg:packetary) for packetary CLI invocation ++_ARCH = { ++ "i386": "i386", ++ "amd64": "x86_64", ++ "arm64": "aarch64", ++} ++ ++# Arch-indepedent configuration (old fuel-mirror + OPNFV extra packages) ++CFG_D = 'opnfv_config' ++CFG_OPNFV = 'opnfv_config.yaml' ++MOS_VERSION = get_env('MOS_VERSION') ++UBUNTU_ARCH = get_env('UBUNTU_ARCH') ++MIRROR_UBUNTU_PATH = get_env('MIRROR_UBUNTU_OPNFV_PATH') ++MIRROR_UBUNTU_TMP_PATH = '{0}.tmp'.format(MIRROR_UBUNTU_PATH) ++MIRROR_UBUNTU_MERGE = get_env('MIRROR_UBUNTU_MERGE') ++CFG_MM_UBUNTU = '{0}/ubuntu_mirror_local.yaml'.format(CFG_D) ++FUEL_BOOTSTRAP_CLI_FILE = open('fuel_bootstrap_cli.yaml').read() ++FUEL_BOOTSTRAP_CLI = yaml.load(FUEL_BOOTSTRAP_CLI_FILE) ++FIXTURE_FILE = open('fuel-web/nailgun/nailgun/fixtures/openstack.yaml').read() ++FIXTURE = yaml.load(FIXTURE_FILE) ++OPNFV_CFG_YAML = open(CFG_OPNFV).read() ++OPNFV_CFG = yaml.load(OPNFV_CFG_YAML) ++ ++# Create local partial mirror using packetary, one arch at a time ++for arch in UBUNTU_ARCH.split(' '): ++ # Mirror / Package env vars, arch-overrideable ++ mos_ubuntu = get_env('MIRROR_MOS_UBUNTU', arch) ++ mos_ubuntu_root = get_env('MIRROR_MOS_UBUNTU_ROOT', arch) ++ mirror_ubuntu = get_env('MIRROR_UBUNTU_URL', arch) ++ plugins = get_env('BUILD_FUEL_PLUGINS', arch) ++ if plugins is None: ++ plugins = get_env('PLUGINS', arch) ++ ++ # Mirror / Package list configuration files (arch-specific) ++ cfg_m_mos = '{0}/mos_{1}_mirror.yaml'.format(CFG_D, arch) ++ cfg_m_ubuntu = '{0}/ubuntu_{1}_mirror.yaml'.format(CFG_D, arch) ++ cfg_p_ubuntu = '{0}/ubuntu_{1}_packages.yaml'.format(CFG_D, arch) ++ cfg_m_ubuntu_main = '{0}/ubuntu_{1}_mirror_main.yaml'.format(CFG_D, arch) ++ cfg_p_ubuntu_main = '{0}/ubuntu_{1}_packages_main.yaml'.format(CFG_D, arch) ++ ++ # Mirror config fork before customizing (arch-specific) ++ arch_mos = 'mos_{0}'.format(arch) ++ arch_ubuntu = 'ubuntu_{0}'.format(arch) ++ arch_packages = 'packages_{0}'.format(arch) ++ OPNFV_CFG['groups'][arch_mos] = deepcopy(OPNFV_CFG['groups']['mos']) ++ OPNFV_CFG['groups'][arch_ubuntu] = deepcopy(OPNFV_CFG['groups']['ubuntu']) ++ OPNFV_CFG[arch_packages] = OPNFV_CFG['packages'] ++ ++ # Mirror config update & conversion to packetary input ++ group_main_ubuntu = dict() ++ for group in OPNFV_CFG['groups'][arch_mos]: ++ group['uri'] = "http://{}{}".format(mos_ubuntu, mos_ubuntu_root) ++ group['suite'] = group['suite'].replace('$mos_version', MOS_VERSION) ++ for group in OPNFV_CFG['groups'][arch_ubuntu]: ++ group['uri'] = mirror_ubuntu ++ # FIXME: At `create`, packetary insists on copying all pkgs to dest dir, ++ # so configure it for another dir, which will replace the orig. ++ group['path'] = MIRROR_UBUNTU_TMP_PATH ++ if not group_main_ubuntu and 'main' in group: ++ group_main_ubuntu = [deepcopy(group)] ++ group_main_ubuntu[0]['section'] = ['main'] ++ ++ # Mirror config dump: MOS (for dep resolution), Ubuntu, Ubuntu[main] ++ write_cfg_file(cfg_m_mos, OPNFV_CFG['groups'][arch_mos]) ++ write_cfg_file(cfg_m_ubuntu, OPNFV_CFG['groups'][arch_ubuntu]) ++ if MIRROR_UBUNTU_MERGE is None: ++ write_cfg_file(cfg_m_ubuntu_main, group_main_ubuntu) ++ else: ++ # FIXME: For multiarch, only one dump would be enough ++ group_main_ubuntu[0]['origin'] = 'Ubuntu' ++ group_main_ubuntu[0]['path'] = MIRROR_UBUNTU_PATH ++ group_main_ubuntu[0]['uri'] = MIRROR_UBUNTU_PATH ++ write_cfg_file(CFG_MM_UBUNTU, group_main_ubuntu[0]) ++ ++ # Collect package dependencies from: ++ ## 1. fuel_bootstrap_cli.yaml (bootstrap image additional packages) ++ legacy_unresolved = legacy_diff(None, FUEL_BOOTSTRAP_CLI['packages'] + [ ++ FUEL_BOOTSTRAP_CLI['kernel_flavor'], ++ FUEL_BOOTSTRAP_CLI['kernel_flavor'].replace('image', 'headers')], ++ 'Bootstrap', arch) ++ ## 2. openstack.yaml FIXTURE definition (default target image packages) ++ for release in FIXTURE: ++ editable = release['fields']['attributes_metadata']['editable'] ++ if 'provision' in editable and 'packages' in editable['provision']: ++ release_pkgs = editable['provision']['packages']['value'].split() ++ legacy_unresolved += legacy_diff(legacy_unresolved, release_pkgs, ++ 'Release {0}'.format(release['fields']['name']), arch) ++ ## 3. OPNFV additional packages (includes old fuel-mirror ubuntu.yaml pkgs) ++ unresolved = dict() ++ unresolved['mandatory'] = 'exact' ++ unresolved['packages'] = from_legacy_pkglist(legacy_unresolved) ++ if 'packages' in OPNFV_CFG: ++ legacy_diff(legacy_unresolved, to_legacy_pkglist(OPNFV_CFG['packages']), ++ 'OPNFV config', arch) ++ unresolved['packages'] += OPNFV_CFG['packages'] ++ ++ # OPNFV plugins dependency resolution ++ if plugins: ++ for plugin in plugins.split(): ++ path = "../{}/packages.yaml".format(plugin) ++ if os.path.isfile(path): ++ f = open(path).read() ++ plugin_yaml = yaml.load(f) ++ new_pkgs = legacy_diff( ++ to_legacy_pkglist(unresolved['packages']), ++ plugin_yaml['packages'], 'Plugin {0}'.format(plugin), arch) ++ unresolved['packages'] += from_legacy_pkglist(new_pkgs) ++ ++ # Package list (reduced, i.e. no MOS deps, but with OPNFV plugin deps) ++ if MIRROR_UBUNTU_MERGE is None: ++ write_cfg_file(cfg_p_ubuntu_main, unresolved) ++ ++ # Mirror package list (full, including MOS/OPNFV plugin deps) ++ unresolved['packages'] += get_unres_pkgs(arch, cfg_m_mos) ++ write_cfg_file(cfg_p_ubuntu, unresolved) ++ do_partial_mirror(arch, cfg_m_ubuntu, cfg_p_ubuntu) ++ if MIRROR_UBUNTU_MERGE is None: ++ # Ubuntu[main] must be evaluated after Ubuntu ++ do_partial_mirror(arch, cfg_m_ubuntu_main, cfg_p_ubuntu_main) ++ ++if MIRROR_UBUNTU_MERGE is None: ++ shutil.move(MIRROR_UBUNTU_TMP_PATH, MIRROR_UBUNTU_PATH) ++else: ++ # Construct single-component mirror from all components ++ for arch in UBUNTU_ARCH.split(' '): ++ cfg_pp_ubuntu = '{0}/ubuntu_{1}_packages_paths.yaml'.format(CFG_D, arch) ++ # OPNFV blacklist ++ opnfv_blacklist = to_legacy_pkglist(OPNFV_CFG['opnfv_blacklist']) ++ # FIXME: We need scanpackages to omit older DEBs ++ # Inspired from http://askubuntu.com/questions/198474/ ++ os.system('dpkg-scanpackages -a {0} {1} 2>/dev/null | ' ++ 'grep -e "^Filename:" | sed "s|Filename: |- file://|g" | ' ++ 'grep -v -E "\/({2})_" > {3}' ++ .format(arch, MIRROR_UBUNTU_TMP_PATH, ++ '|'.join(opnfv_blacklist), cfg_pp_ubuntu)) ++ do_local_repo(arch, CFG_MM_UBUNTU, cfg_pp_ubuntu) ++ shutil.rmtree(MIRROR_UBUNTU_TMP_PATH) +diff --git a/build/f_isoroot/f_repobuild/select_ubuntu_repo.sh b/build/f_isoroot/f_repobuild/select_ubuntu_repo.sh +index 5618eb7..c8c86db 100755 +--- a/build/f_isoroot/f_repobuild/select_ubuntu_repo.sh ++++ b/build/f_isoroot/f_repobuild/select_ubuntu_repo.sh +@@ -1,5 +1,6 @@ + #!/bin/bash + ++UBUNTU_DISTRO="xenial" + BLACKLIST="http://mirrors.se.eu.kernel.org/ubuntu/" + #BLACKLIST+=" http://foo.bar" + +@@ -28,26 +29,26 @@ blacklisted () { + check_mirror () { + mirror=$1 + status=0 +- for packdir in dists/trusty-updates/main/binary-amd64 \ +- dists/trusty-updates/restricted/binary-amd64 \ +- dists/trusty-updates/universe/binary-amd64 \ +- dists/trusty-updates/multiverse/binary-amd64 \ +- dists/trusty-security/main/binary-amd64 \ +- dists/trusty-security/restricted/binary-amd64 \ +- dists/trusty-security/universe/binary-amd64 \ +- dists/trusty-security/multiverse/binary-amd64 \ +- dists/trusty-proposed/main/binary-amd64 \ +- dists/trusty-proposed/restricted/binary-amd64 \ +- dists/trusty-proposed/universe/binary-amd64 \ +- dists/trusty-proposed/multiverse/binary-amd64 \ +- dists/trusty/main/binary-amd64 \ +- dists/trusty/restricted/binary-amd64 \ +- dists/trusty/universe/binary-amd64 \ +- dists/trusty/multiverse/binary-amd64 \ +- dists/trusty-backports/main/binary-amd64 \ +- dists/trusty-backports/restricted/binary-amd64 \ +- dists/trusty-backports/universe/binary-amd64 \ +- dists/trusty-backports/multiverse/binary-amd64 ++ for packdir in dists/${UBUNTU_DISTRO}-updates/main/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}-updates/restricted/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}-updates/universe/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}-updates/multiverse/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}-security/main/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}-security/restricted/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}-security/universe/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}-security/multiverse/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}-proposed/main/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}-proposed/restricted/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}-proposed/universe/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}-proposed/multiverse/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}/main/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}/restricted/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}/universe/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}/multiverse/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}-backports/main/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}-backports/restricted/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}-backports/universe/binary-amd64 \ ++ dists/${UBUNTU_DISTRO}-backports/multiverse/binary-amd64 + do + for packfile in Release Packages.gz + do +diff --git a/build/install/apt-ftparchive-udeb.conf b/build/install/apt-ftparchive-udeb.conf +index 3b5b239..0cabe94 100644 +--- a/build/install/apt-ftparchive-udeb.conf ++++ b/build/install/apt-ftparchive-udeb.conf +@@ -17,8 +17,8 @@ TreeDefault { + }; + + BinDirectory "pool/debian-installer" { +- Packages "dists/trusty/main/debian-installer/binary-amd64/Packages"; +- BinOverride "./indices/override.trusty.main.debian-installer"; ++ Packages "dists/xenial/main/debian-installer/binary-amd64/Packages"; ++ BinOverride "./indices/override.xenial.main.debian-installer"; + }; + + Default { +diff --git a/build/install/install.sh b/build/install/install.sh +index 198f26c..b2f6ccc 100755 +--- a/build/install/install.sh ++++ b/build/install/install.sh +@@ -122,17 +122,17 @@ prep_make_live() { + ssh-copy-id root@$FUELHOST + sshfs root@1${FUELHOST}:/ $TMP_HOSTMOUNT + +- if [ -f $REPO/dists/trusty/main/binary-amd64/Packages.backup ]; then ++ if [ -f $REPO/dists/xenial/main/binary-amd64/Packages.backup ]; then + echo "Error - found backup file for Packages!" + exit 1 + fi + +- if [ -f $REPO/dists/trusty/main/binary-amd64/Packages.gz.backup ]; then ++ if [ -f $REPO/dists/xenial/main/binary-amd64/Packages.gz.backup ]; then + echo "Error - found backup file for Packages.gz!" + exit 1 + fi + +- if [ -f $REPO/dists/trusty/Release.backup ]; then ++ if [ -f $REPO/dists/xenial/Release.backup ]; then + echo "Error - found backup file for Release!" + exit 1 + fi +@@ -142,9 +142,9 @@ prep_make_live() { + exit 1 + fi + +- cp $REPO/dists/trusty/main/binary-amd64/Packages $REPO/dists/trusty/main/binary-amd64/Packages.backup +- cp $REPO/dists/trusty/main/binary-amd64/Packages.gz $REPO/dists/trusty/main/binary-amd64/Packages.gz.backup +- cp $REPO/dists/trusty/Release $REPO/dists/trusty/Release.backup ++ cp $REPO/dists/xenial/main/binary-amd64/Packages $REPO/dists/xenial/main/binary-amd64/Packages.backup ++ cp $REPO/dists/xenial/main/binary-amd64/Packages.gz $REPO/dists/xenial/main/binary-amd64/Packages.gz.backup ++ cp $REPO/dists/xenial/Release $REPO/dists/xenial/Release.backup + cp -Rvp $DEST/etc/puppet $DEST/etc/puppet.backup + } + +@@ -443,7 +443,7 @@ copy_packages() { + echo Not running apt-ftparchive generate "${APT_UDEB_CONF}" + + # Fuel also needs this index file +- # cat dists/trusty/main/binary-amd64/Packages | \ ++ # cat dists/xenial/main/binary-amd64/Packages | \ + # awk '/^Package:/{pkg=$2} + # /^Version:/{print pkg ": \"" $2 "\""}' > ubuntu-versions.yaml + # cp ubuntu-versions.yaml $DEST +diff --git a/build/install/uninstall.sh b/build/install/uninstall.sh +index a9e74bc..5252c30 100755 +--- a/build/install/uninstall.sh ++++ b/build/install/uninstall.sh +@@ -31,17 +31,17 @@ DEST=$MOUNT + REPO=$DEST/var/www/nailgun/ubuntu/fuelweb/x86_64 + + cd $REPO +-if [ ! -f $REPO/dists/trusty/main/binary-amd64/Packages.backup ]; then ++if [ ! -f $REPO/dists/xenial/main/binary-amd64/Packages.backup ]; then + echo "Error - didn't find backup file for Packages!" + exit 1 + fi + +-if [ ! -f $REPO/dists/trusty/main/binary-amd64/Packages.gz.backup ]; then ++if [ ! -f $REPO/dists/xenial/main/binary-amd64/Packages.gz.backup ]; then + echo "Error - didn't find backup file for Packages.gz!" + exit 1 + fi + +-if [ ! -f $REPO/dists/trusty/Release.backup ]; then ++if [ ! -f $REPO/dists/xenial/Release.backup ]; then + echo "Error - didn't find backup file for Release!" + exit 1 + fi +@@ -71,9 +71,9 @@ cd $REPO + + echo "Restoring backups of datafiles" + +-rm -f $REPO/dists/trusty/main/binary-amd64/Packages $REPO/dists/trusty/main/binary-amd64/Packages.gz +-rm -f $REPO/dists/trusty/Release $DEST/etc/puppet/manifests/site.pp +-mv $REPO/dists/trusty/main/binary-amd64/Packages.backup $REPO/dists/trusty/main/binary-amd64/Packages +-mv $REPO/dists/trusty/main/binary-amd64/Packages.gz.backup $REPO/dists/trusty/main/binary-amd64/Packages.gz +-mv $REPO/dists/trusty/Release.backup $REPO/dists/trusty/Release ++rm -f $REPO/dists/xenial/main/binary-amd64/Packages $REPO/dists/xenial/main/binary-amd64/Packages.gz ++rm -f $REPO/dists/xenial/Release $DEST/etc/puppet/manifests/site.pp ++mv $REPO/dists/xenial/main/binary-amd64/Packages.backup $REPO/dists/xenial/main/binary-amd64/Packages ++mv $REPO/dists/xenial/main/binary-amd64/Packages.gz.backup $REPO/dists/xenial/main/binary-amd64/Packages.gz ++mv $REPO/dists/xenial/Release.backup $REPO/dists/xenial/Release + mv $DEST/etc/puppet/manifests/site.pp.backup $DEST/etc/puppet/manifests/site.pp +diff --git a/deploy/config/dea_base.yaml b/deploy/config/dea_base.yaml +index ce81994..2520cf5 100644 +--- a/deploy/config/dea_base.yaml ++++ b/deploy/config/dea_base.yaml +@@ -722,22 +722,10 @@ settings: + value: + - name: ubuntu + priority: null +- section: main universe multiverse ++ section: main + suite: xenial + type: deb +- uri: http://archive.ubuntu.com/ubuntu/ +- - name: ubuntu-updates +- priority: null +- section: main universe multiverse +- suite: xenial-updates +- type: deb +- uri: http://archive.ubuntu.com/ubuntu/ +- - name: ubuntu-security +- priority: null +- section: main universe multiverse +- suite: xenial-security +- type: deb +- uri: http://archive.ubuntu.com/ubuntu/ ++ uri: http://10.20.0.2:8080/mirrors/ubuntu/ + - name: mos + priority: 1050 + section: main restricted diff --git a/patches/opnfv-fuel/multiarch-mirrors/0001-build-install-Add-arm64-deb-repositories-setup.patch b/patches/opnfv-fuel/multiarch-mirrors/0001-build-install-Add-arm64-deb-repositories-setup.patch index 35129e01..5513309e 100644 --- a/patches/opnfv-fuel/multiarch-mirrors/0001-build-install-Add-arm64-deb-repositories-setup.patch +++ b/patches/opnfv-fuel/multiarch-mirrors/0001-build-install-Add-arm64-deb-repositories-setup.patch @@ -11,7 +11,7 @@ approach. FIXME: Use UBUNTU_ARCH env var instead of hardcoding the arch list. While fixing that, also take care of missed s/trusty/mos9.0/g -in {un,}install.sh. +and/or s/xenial/mos10.0/g in {un,}install.sh. Closes-bug: ARMBAND-163 @@ -58,13 +58,13 @@ index edcb1f2..81ead2b 100644 }; -TreeDefault { -+Tree "dists/mos9.0" { ++Tree "dists/mos10.0" { + Architectures "amd64 arm64"; Directory "pool"; -}; - -BinDirectory "pool/main" { -- Packages "dists/mos9.0/main/binary-amd64/Packages"; +- Packages "dists/mos10.0/main/binary-amd64/Packages"; + Sections "main"; }; @@ -73,14 +73,14 @@ diff --git a/build/install/apt-ftparchive-release.conf b/build/install/apt-ftpar index e40ff68..ceaf71a 100644 --- a/build/install/apt-ftparchive-release.conf +++ b/build/install/apt-ftparchive-release.conf -@@ -13,6 +13,7 @@ APT::FTPArchive::Release::Label "mos9.0"; - APT::FTPArchive::Release::Suite "mos9.0"; - APT::FTPArchive::Release::Version "14.04"; - APT::FTPArchive::Release::Codename "mos9.0"; +@@ -13,6 +13,7 @@ APT::FTPArchive::Release::Label "mos10.0"; + APT::FTPArchive::Release::Suite "mos10.0"; + APT::FTPArchive::Release::Version "16.04"; + APT::FTPArchive::Release::Codename "mos10.0"; -APT::FTPArchive::Release::Architectures "amd64"; +APT::FTPArchive::Release::Architectures "amd64 arm64"; APT::FTPArchive::Release::Components "main"; - APT::FTPArchive::Release::Description "Ubuntu Trusty Tahr 14.04 LTS"; + APT::FTPArchive::Release::Description "Ubuntu 16.04.1 LTS"; +APT::FTPArchive::Contents "no"; diff --git a/build/install/apt-ftparchive-udeb.conf b/build/install/apt-ftparchive-udeb.conf index 3b5b239..8852db7 100644 @@ -94,14 +94,14 @@ index 3b5b239..8852db7 100644 }; -TreeDefault { -+Tree "dists/mos9.0" { ++Tree "dists/mos10.0" { + Architectures "amd64 arm64"; Directory "pool"; -}; - -BinDirectory "pool/debian-installer" { -- Packages "dists/trusty/main/debian-installer/binary-amd64/Packages"; -- BinOverride "./indices/override.trusty.main.debian-installer"; +- Packages "dists/xenial/main/debian-installer/binary-amd64/Packages"; +- BinOverride "./indices/override.xenial.main.debian-installer"; + Sections "main"; }; @@ -114,26 +114,26 @@ index d1a49f3..0561207 100755 ssh-copy-id root@$FUELHOST sshfs root@1${FUELHOST}:/ $TMP_HOSTMOUNT -- if [ -f $REPO/dists/trusty/main/binary-amd64/Packages.backup ]; then +- if [ -f $REPO/dists/xenial/main/binary-amd64/Packages.backup ]; then + for arch in arm64 amd64; do -+ if [ -f $REPO/dists/trusty/main/binary-${arch}/Packages.backup ]; then ++ if [ -f $REPO/dists/xenial/main/binary-${arch}/Packages.backup ]; then + echo "Error - found backup file for Packages for ${arch}!" + exit 1 + fi + -+ if [ -f $REPO/dists/trusty/main/binary-${arch}/Packages.gz.backup ]; then ++ if [ -f $REPO/dists/xenial/main/binary-${arch}/Packages.gz.backup ]; then + echo "Error - found backup file for Packages.gz for ${arch}!" + exit 1 + fi + done + -+ if [ -f $REPO/dists/trusty/main/binary-arm64/Packages.backup ]; then ++ if [ -f $REPO/dists/xenial/main/binary-arm64/Packages.backup ]; then echo "Error - found backup file for Packages!" exit 1 fi -- if [ -f $REPO/dists/trusty/main/binary-amd64/Packages.gz.backup ]; then -+ if [ -f $REPO/dists/trusty/main/binary-arm64/Packages.gz.backup ]; then +- if [ -f $REPO/dists/xenial/main/binary-amd64/Packages.gz.backup ]; then ++ if [ -f $REPO/dists/xenial/main/binary-arm64/Packages.gz.backup ]; then echo "Error - found backup file for Packages.gz!" exit 1 fi @@ -141,13 +141,13 @@ index d1a49f3..0561207 100755 exit 1 fi -- cp $REPO/dists/trusty/main/binary-amd64/Packages $REPO/dists/trusty/main/binary-amd64/Packages.backup -- cp $REPO/dists/trusty/main/binary-amd64/Packages.gz $REPO/dists/trusty/main/binary-amd64/Packages.gz.backup +- cp $REPO/dists/xenial/main/binary-amd64/Packages $REPO/dists/xenial/main/binary-amd64/Packages.backup +- cp $REPO/dists/xenial/main/binary-amd64/Packages.gz $REPO/dists/xenial/main/binary-amd64/Packages.gz.backup + for arch in arm64 amd64; do -+ cp $REPO/dists/trusty/main/binary-${arch}/Packages $REPO/dists/trusty/main/binary-${arch}/Packages.backup -+ cp $REPO/dists/trusty/main/binary-${arch}/Packages.gz $REPO/dists/trusty/main/binary-${arch}/Packages.gz.backup ++ cp $REPO/dists/xenial/main/binary-${arch}/Packages $REPO/dists/xenial/main/binary-${arch}/Packages.backup ++ cp $REPO/dists/xenial/main/binary-${arch}/Packages.gz $REPO/dists/xenial/main/binary-${arch}/Packages.gz.backup + done - cp $REPO/dists/trusty/Release $REPO/dists/trusty/Release.backup + cp $REPO/dists/xenial/Release $REPO/dists/xenial/Release.backup cp -Rvp $DEST/etc/puppet $DEST/etc/puppet.backup } diff --git a/build/install/uninstall.sh b/build/install/uninstall.sh @@ -158,25 +158,25 @@ index a9e74bc..e90c632 100755 REPO=$DEST/var/www/nailgun/ubuntu/fuelweb/x86_64 cd $REPO --if [ ! -f $REPO/dists/trusty/main/binary-amd64/Packages.backup ]; then +-if [ ! -f $REPO/dists/xenial/main/binary-amd64/Packages.backup ]; then - echo "Error - didn't find backup file for Packages!" - exit 1 -fi +for arch in arm64 amd64; do -+ if [ ! -f $REPO/dists/trusty/main/binary-${arch}/Packages.backup ]; then ++ if [ ! -f $REPO/dists/xenial/main/binary-${arch}/Packages.backup ]; then + echo "Error - didn't find backup file for Packages for ${arch}!" + exit 1 + fi --if [ ! -f $REPO/dists/trusty/main/binary-amd64/Packages.gz.backup ]; then +-if [ ! -f $REPO/dists/xenial/main/binary-amd64/Packages.gz.backup ]; then - echo "Error - didn't find backup file for Packages.gz!" - exit 1 -fi -+ if [ ! -f $REPO/dists/trusty/main/binary-${arch}/Packages.gz.backup ]; then ++ if [ ! -f $REPO/dists/xenial/main/binary-${arch}/Packages.gz.backup ]; then + echo "Error - didn't find backup file for Packages.gz for ${arch}!" + exit 1 + fi +done - if [ ! -f $REPO/dists/trusty/Release.backup ]; then + if [ ! -f $REPO/dists/xenial/Release.backup ]; then echo "Error - didn't find backup file for Release!" |