From: Alexandru Avadanii Date: Wed, 6 Jul 2016 17:03:49 +0200 Subject: [PATCH] f_repobuild: Use packetary to build partial mirror FIXME: DO NOT MERGE until [2, 3] are merged upstream & included here! TODO: drop ALL fuel-mirror dependencies, for Fuel Newton compatibility. Packetary [1] will replace fuel-mirror in upstream Fuel. fuel-mirror is using packetary under the hood already, so the transition should not be too hard. Adapt OPNFV Ubuntu partial mirror build: - obsolete opnfv-config.yaml (packetary uses CLI args for this); - split old ubuntu.yaml into: * mos__mirror.yaml - MOS mirror cfg, only for deps analysis; * ubuntu__mirror.yaml - Ubuntu mirror cfg, for pkgs download; * ubuntu__packages.yaml - Ubuntu packages to download; While the consumers of our mirror still use `debootstrap`, which only supports the "main" repo component, make sure the "main" package set is complete, in the scope of: * ubuntu__mirror_main.yaml - Ubuntu [main only] mirror; * ubuntu__packages_main.yaml - Ubuntu packages for [main]; New workflow (executed for EACH architecture): 1. Parse "fuel-mirror/contrib/fuel_mirror/data/ubuntu.yaml" and write correponding sections to new yaml files described above; NOTE: Since fuel-mirror is going to be made obsolete, there are a number of new ways we can gather the data currently passed via fuel-mirror's "ubuntu.yaml" (e.g. use "mos-repos" from Fuel ISO to determine MOS dependencies). 2. Add OPNFV extra package dependencies (NEW) Currently, this is empty / not used, but might be used to add extra packages to the local mirror, like additional kernels, bootloaders etc. NOTE: Dependencies up to and including this group should also be satisfiable from Ubuntu "main" repo component only. 3. Append plugin dependencies to the package list; 4. Run `packetary unresolved` for MOS repos, gathering MOS deps that should be fetched from Ubuntu. The new packages will be appended to the predefined list from old "ubuntu.yaml"; 5. Run `packetary clone` to download all required pkgs for the partial Ubuntu mirror. 6. IF repo component merging is disabled, run `packetary clone` only on Ubuntu[main] repo components to download any (still) missing dependencies for `debootstrap`, which only looks at this component, and not the whole mirror. 7. IF repo component merging is enabled, run first `dpkg-scanpackages` to filter out old duplicate versions of packages, then run `packetary create` to merge all repo comps into a single "main". v1 -> v2: - multiarch support (activated via UBUNTU_ARCH envvar); - fixed debootstrap missing deps in "main" component of Ubuntu mirror; - factored out some hardcodes into variables/functions; - add .gitignore file; - move generated config files to "opnfv_config" subdir; - added arm64 Ubuntu repo URL (ports.ubuntu.com is separated from the main Ubuntu mirror, archive.ubuntu.com, which only holds x86 pkgs); v3 -> v4: - introduce env var "MIRROR_UBUNTU_MERGE" to control local mirror repo-component merging into a single "main"; - enable mirror repo-component merging by default, edit fuel_bootstrap_cli.yaml accordingly; v4 -> v6: - edit dea_base to use only main mirror; - fix duplicated logs; - remove `-q` flag for `create`; NOTE: Without filtering out old package verisons using `dpkg-scanpackages`, bootstrap build cannot solve all deps. FIXME: Packetary solves missing dependecies by also accepting different packages that provide the same package name (e.g. "ifupdown" is satisfied by "netscript" package). FIXME: Repo component merging is sort of slow, since packetary insists on copying the source files to the destination dir even if they are pointing to the same repo. FIXME: Packetary `create` uses a different directory scheme for the created mirror than the upstream Ubuntu, e.g.: [p] mirrors/ubuntu/pool/main/l/lvm2_2.02.98-6ubuntu2_amd64.deb [U] mirrors/ubuntu/pool/main/l/lvm2/lvm2_2.02.98-6ubuntu2_amd64.deb This disencourages creating the "merged" mirror in the same location as the source partial mirror. [1] https://wiki.openstack.org/wiki/Packetary [2] https://review.openstack.org/#/c/392936/ [3] https://review.openstack.org/#/c/392937/ JIRA: FUEL-218 JIRA: FUEL-223 JIRA: ARMBAND-169 Change-Id: If2ee86f348b7683c83bfaf686baba4f1b8f555f0 Signed-off-by: Alexandru Avadanii --- build/config.mk | 5 + build/f_isoroot/f_repobuild/.gitignore | 4 + build/f_isoroot/f_repobuild/Makefile | 35 ++- build/f_isoroot/f_repobuild/config.mk | 21 +- .../f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml | 14 +- build/f_isoroot/f_repobuild/opnfv_mirror_conf.py | 57 ----- build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py | 247 +++++++++++++++++++++ build/f_isoroot/f_repobuild/opnfv_packages.yaml | 14 ++ deploy/config/dea_base.yaml | 14 +- 9 files changed, 307 insertions(+), 104 deletions(-) create mode 100644 build/f_isoroot/f_repobuild/.gitignore delete mode 100755 build/f_isoroot/f_repobuild/opnfv_mirror_conf.py create mode 100755 build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py create mode 100644 build/f_isoroot/f_repobuild/opnfv_packages.yaml diff --git a/build/config.mk b/build/config.mk index f78043e..96dd138 100644 --- a/build/config.mk +++ b/build/config.mk @@ -13,6 +13,11 @@ FUEL_MAIN_TAG := 9.0.1 MOS_VERSION = 9.0 OPENSTACK_VERSION = mitaka-9.0 +# List of space-separated Ubuntu architectures supported with current build +# Format: same as `dpkg-architecture -qDEB_HOST_ARCH1` +# NOTE: Currently only amd64 is supported by Fuel@OPNFV. Armband adds arm64. +export UBUNTU_ARCH ?= amd64 + # Pinning down exact Fuel repo versions for Fuel 9.0.1 export FUELLIB_COMMIT?=e283b62750d9e26355981b3ad3be7c880944ae0f export NAILGUN_COMMIT?=e2b85bafb68c348f25cb7cceda81edc668ba2e64 diff --git a/build/f_isoroot/f_repobuild/.gitignore b/build/f_isoroot/f_repobuild/.gitignore new file mode 100644 index 0000000..cfedefb --- /dev/null +++ b/build/f_isoroot/f_repobuild/.gitignore @@ -0,0 +1,4 @@ +packetary +fuel-mirror +nailgun +opnfv_config diff --git a/build/f_isoroot/f_repobuild/Makefile b/build/f_isoroot/f_repobuild/Makefile index 7c6b163..220eedd 100644 --- a/build/f_isoroot/f_repobuild/Makefile +++ b/build/f_isoroot/f_repobuild/Makefile @@ -1,7 +1,8 @@ ############################################################################## -# Copyright (c) 2015 Ericsson AB and others. +# Copyright (c) 2015,2016 Ericsson AB, Enea AB and others. # stefan.k.berg@ericsson.com # jonas.bjurel@ericsson.com +# Alexandru.Avadanii@enea.com # All rights reserved. This program and the accompanying materials # are made available under the terms of the Apache License, Version 2.0 # which accompanies this distribution, and is available at @@ -10,43 +11,37 @@ SHELL := /bin/bash TOP := $(shell pwd) -TMP_ROOT_DIR := $(shell echo "$(MIRROR_UBUNTU_ROOT)" | cut -d "/" -f2) include ../../config.mk include config.mk export MOS_VERSION -export OPENSTACK_VERSION +export MIRROR_UBUNTU_OPNFV_PATH:=$(TOP)/nailgun/mirrors/ubuntu .PHONY: all all: nailgun nailgun: - sudo apt-get install -y git libxml2-dev libxslt-dev python-dev python-pip libz-dev libyaml-dev createrepo python-yaml - rm -Rf nailgun - sudo mkdir -p /var/www/nailgun - git clone $(FUEL_MIRROR_REPO) + sudo apt-get install -y createrepo git libxml2-dev libxslt1-dev \ + python-dev zlib1g-dev + rm -Rf nailgun packetary opnfv_config && mkdir opnfv_config + git clone --quiet $(FUEL_MIRROR_REPO) cd fuel-mirror && git checkout -q $(FUEL_MIRROR_COMMIT) - sudo pip install -U -r ./fuel-mirror/requirements.txt - sudo pip install ./fuel-mirror - sudo pip install ./fuel-mirror/contrib/fuel_mirror - ./opnfv_mirror_conf.py - sudo fuel-mirror --debug --config ./opnfv-config.yaml create --group ubuntu --pattern=ubuntu - sudo chmod -R 755 /var/www/nailgun - cp -Rp /var/www/nailgun . - # On the end we want to have ubuntu repository in mirrors/ubuntu directory - -if [ "$(MIRROR_UBUNTU_ROOT)" != "/ubuntu/" ]; then \ - mkdir -p nailgun/mirrors/ubuntu;\ - mv nailgun/mirrors$(MIRROR_UBUNTU_ROOT)* nailgun/mirrors/ubuntu;\ - [ "$(MIRROR_UBUNTU_ROOT)" != "/" ] && rm -rf nailgun/mirrors/$(TMP_ROOT_DIR);\ + git clone --quiet $(PACKETARY_REPO) + if [ -n $(PACKETARY_COMMIT) ]; then \ + git -C packetary checkout $(PACKETARY_COMMIT); \ fi + sudo pip install -U -r ./packetary/requirements.txt + sudo pip install ./packetary + # Handle config and mirror build in one place + ./opnfv_mirror_ubuntu.py # Store artifact in cache straight away if caching is enabled # (no .cacheid will be present unless this is a cached build) test -f .cacheid && $(MAKE) -f Makefile put-cache || exit 0 .PHONY: clean clean: - @rm -rf ../release/opnfv/nailgun nailgun fuel-mirror opnfv-config.yaml ubuntu.yaml + @rm -rf ../release/opnfv/nailgun nailgun packetary fuel-mirror opnfv_config .PHONY: release release:nailgun diff --git a/build/f_isoroot/f_repobuild/config.mk b/build/f_isoroot/f_repobuild/config.mk index d817328..33d0e70 100644 --- a/build/f_isoroot/f_repobuild/config.mk +++ b/build/f_isoroot/f_repobuild/config.mk @@ -1,6 +1,9 @@ ############################################################################## -# Copyright (c) 2016 Ericsson AB and others. +# Copyright (c) 2016 Ericsson AB, Enea AB and others. # mskalski@mirantis.com +# stefan.k.berg@ericsson.com +# jonas.bjurel@ericsson.com +# Alexandru.Avadanii@enea.com # All rights reserved. This program and the accompanying materials # are made available under the terms of the Apache License, Version 2.0 # which accompanies this distribution, and is available at @@ -11,3 +14,19 @@ FUEL_MIRROR_REPO?=https://github.com/openstack/fuel-mirror # Point to the commit where 302 redirects are handled # https://bugs.launchpad.net/fuel/mitaka/+bug/1593674 FUEL_MIRROR_COMMIT?=192a3d9f8f993afb12c5108dd9339c6688c23e11 + +# Use a recent master commit, since tags/branches are not yet mature +# FIXME(armband): Update upstream commit ref once [1, 2] are merged +# [1] https://review.openstack.org/#/c/392937/ +# [2] https://review.openstack.org/#/c/392936/ +export PACKETARY_REPO?=https://github.com/openstack/packetary +export PACKETARY_COMMIT?=c46465c3255a9f5e59a05b8701e06054df39f32f + +# arm64 Ubuntu mirror is separated from archive.ubuntu.com +export MIRROR_UBUNTU_URL_arm64=http://ports.ubuntu.com/ubuntu-ports/ +export MIRROR_UBUNTU_ROOT_arm64=ubuntu-ports + +# Merge all local mirror repo components/section into single "main" +# NOTE: When changing this, make sure to also update all consumer config, like: +# - fuel_bootstrap_cli.yaml +export MIRROR_UBUNTU_MERGE=true diff --git a/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml b/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml index 19dea1c..52786fb 100644 --- a/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml +++ b/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml @@ -48,23 +48,11 @@ https_proxy: "" repos: - name: ubuntu - section: "main universe multiverse" + section: "main" uri: "http://127.0.0.1:8080/mirrors/ubuntu" priority: suite: trusty type: deb - - name: ubuntu-updates - section: "main universe multiverse" - uri: "http://127.0.0.1:8080/mirrors/ubuntu" - priority: - suite: trusty-updates - type: deb - - name: ubuntu-security - section: "main universe multiverse" - uri: "http://127.0.0.1:8080/mirrors/ubuntu" - priority: - suite: trusty-security - type: deb - name: mos section: "main restricted" uri: "http://127.0.0.1:8080/ubuntu/x86_64" diff --git a/build/f_isoroot/f_repobuild/opnfv_mirror_conf.py b/build/f_isoroot/f_repobuild/opnfv_mirror_conf.py deleted file mode 100755 index 1c7eb59..0000000 --- a/build/f_isoroot/f_repobuild/opnfv_mirror_conf.py +++ /dev/null @@ -1,57 +0,0 @@ -#!/usr/bin/env python -############################################################################## -# Copyright (c) 2015 Ericsson AB and others. -# mskalski@mirantis.com -# All rights reserved. This program and the accompanying materials -# are made available under the terms of the Apache License, Version 2.0 -# which accompanies this distribution, and is available at -# http://www.apache.org/licenses/LICENSE-2.0 -############################################################################## -import os -import yaml - -current_snapshot = os.environ["LATEST_TARGET_UBUNTU"] -mos_version = os.environ['MOS_VERSION'] -openstack_version = os.environ['OPENSTACK_VERSION'] -mos_ubuntu = os.environ['MIRROR_MOS_UBUNTU'] -mos_ubuntu_root = os.environ['MIRROR_MOS_UBUNTU_ROOT'] -mirror_ubuntu = os.environ['MIRROR_UBUNTU_URL'] -if os.environ.get('BUILD_FUEL_PLUGINS'): - plugins = os.environ['BUILD_FUEL_PLUGINS'] -else: - plugins = os.environ['PLUGINS'] - - -configuration_file = open('fuel-mirror/contrib/fuel_mirror/etc/config.yaml').read() -conf = yaml.load(configuration_file) -conf['pattern_dir'] = '.' -conf['openstack_version'] = openstack_version -conf['mos_version'] = mos_version - -with open('opnfv-config.yaml', 'w') as outfile: - outfile.write( yaml.dump(conf, default_flow_style=False) ) - -pattern_file = open('fuel-mirror/contrib/fuel_mirror/data/ubuntu.yaml').read() -pattern = yaml.load(pattern_file) -pattern['mos_baseurl'] = "http://{}{}".format(mos_ubuntu, mos_ubuntu_root) -pattern['ubuntu_baseurl'] = mirror_ubuntu -for group in pattern['groups']['mos']: - group['uri'] = pattern['mos_baseurl'] -for group in pattern['groups']['ubuntu']: - group['uri'] = pattern['ubuntu_baseurl'] - -for plugin in plugins.split(): - path = "../{}/packages.yaml".format(plugin) - if os.path.isfile(path): - f = open(path).read() - plugin_yaml = yaml.load(f) - plugin_set = set(plugin_yaml['packages']) - main_set = set(pattern['packages']) - new_packages = plugin_set - main_set - print "Plugin {} require new packages: {}".format(plugin, ', '.join(new_packages)) - pattern['packages'] = pattern['packages'] + list(new_packages) - -pattern['requirements']['ubuntu'] = pattern['packages'] - -with open('ubuntu.yaml', 'w') as outfile: - outfile.write( yaml.safe_dump(pattern, default_flow_style=False) ) diff --git a/build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py b/build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py new file mode 100755 index 0000000..901ed4b --- /dev/null +++ b/build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py @@ -0,0 +1,247 @@ +#!/usr/bin/env python +############################################################################## +# Copyright (c) 2015,2016 Ericsson AB, Mirantis Inc., Enea AB and others. +# mskalski@mirantis.com +# Alexandru.Avadanii@enea.com +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Apache License, Version 2.0 +# which accompanies this distribution, and is available at +# http://www.apache.org/licenses/LICENSE-2.0 +############################################################################## + +############################################################################## +# Build multiarch partial local Ubuntu mirror +############################################################################## +# Design quirks / workarounds: +# 1. Fuel-agent uses `debootstrap` to build bootstrap and target chroots from +# the local mirror; which only uses the "main" component from the first +# repository, i.e. does not include "updates"/"security". +# In order to fullfill all debootstrap dependencies in "main" repo, we will +# do an extra packetary run using a reduced scope: +# - only "main" component of the first mirror; +# - reduced package dependency list (without MOS/OPNFV plugin deps). +# 2. If repo structure is not mandatory to be in sync with official mirrors, +# we can mitigate the issue by "merging" all repo-components into a single +# "main". +############################################################################## +# Mirror build steps: +# 1. Parse old package list ("ubuntu.yaml"); +# 2. Parse new "opnfv_packages.yaml" list of additional packages; +# 3. Inherit enviroment variable(s) for mirror URLs, paths etc. +# - Allow arch-specific overrides for each env var; +# 4. For each architecture in UBUNTU_ARCH: +# 4.1. Mirror config is defined based on common config + OPNFV overrides; +# - Convert old configuration format to packetary style where needed; +# 4.2. Package lists are defined based on common config + OPNFV deps; +# - Keep track of "main" packages separately, required by debootstrap; +# 4.3. Clone/update all mirror components; +# 4.4. IF mirror merging is disabled OR workaround for ifupdown (see below): +# - Clone/update "main" mirror component (fix missing debootstrap deps); +# 5. IF mirror merging is enabled: +# - Use `dpkg-scanpackages` to filter out old versions of duplicate pkgs; +# - Run `packetary create` on the set of downloaded packages, merging +# them on the fly into a single-component mirror; +############################################################################## + +import copy +import os +import shutil +import sys +import yaml +from contextlib import contextmanager +from cStringIO import StringIO +from packetary.cli.app import main + +@contextmanager +def captureStdOut(output): + stdout = sys.stdout + sys.stdout = output + yield + sys.stdout = stdout + +# FIXME: Find a better approach for eliminating duplicate logs than this +def force_logger_reload(): + for mod in sys.modules.keys(): + if mod.startswith('logging'): + try: + reload(sys.modules[mod]) + except: + pass + +# Determine missing package dependecies for a mirror defition +def get_unres_pkgs(arch, cfg_mirror): + unresolved = dict() + unresolved['packages'] = list() + packetary_output = StringIO() + with captureStdOut(packetary_output): + main('unresolved -a {0} -r {1} -c name version --sep ;' + .format(_ARCHITECTURES[arch], cfg_mirror).split(' ')) + for dep_pkg in packetary_output.getvalue().splitlines(): + if dep_pkg.startswith('#'): + continue + dep = dep_pkg.split(';') + unresolved['packages'] += [ {'name': dep[0], 'version': dep[1]} ] + force_logger_reload() + return unresolved + +# Create single-component local repo (one arch per call) +def do_local_repo(arch, cfg_repo, cfg_packages_paths): + # Packetary does not use a global config file, so pass old settings here. + main('create -t deb -a {0} --repository {1} --package-files {2}' + ' --ignore-errors-num 2 --retries-num 3 --threads-num 10' + .format(_ARCHITECTURES[arch], cfg_repo, cfg_packages_paths).split(' ')) + force_logger_reload() + +# Clone partial local mirror (one arch per call) +def do_partial_mirror(arch, cfg_mirror, cfg_packages): + # Note: '-d .' is ignored, as each mirror defines its own path. + main('clone -t deb -a {0} -r {1} -R {2} -d .' + ' --ignore-errors-num 2 --retries-num 3 --threads-num 10' + .format(_ARCHITECTURES[arch], cfg_mirror, cfg_packages).split(' ')) + force_logger_reload() + +# Write configuration (yaml) file (package list / mirror defition) +def write_cfg_file(cfg_mirror, data): + with open(cfg_mirror, 'w') as outfile: + outfile.write( yaml.safe_dump(data, default_flow_style=False) ) + +# Allow arch-specific overrides of env vars +def get_env(env_var, arch=None): + if arch: + env_var_arch = '{0}_{1}'.format(env_var, arch) + if os.environ.get(env_var_arch): + return os.environ[env_var_arch] + if os.environ.get(env_var): + return os.environ[env_var] + return None + +# Architecture name mapping (dpkg:packetary) for packetary CLI invocation +_ARCHITECTURES = { + "i386": "i386", + "amd64": "x86_64", + "arm64": "aarch64", +} + +# Arch-indepedent configuration (old fuel-mirror + OPNFV extra packages) +cfg_dir = 'opnfv_config' +cfg_p_opnfv = 'opnfv_packages.yaml' +mos_version = get_env('MOS_VERSION') +ubuntu_arch = get_env('UBUNTU_ARCH') +mirror_ubuntu_path = get_env('MIRROR_UBUNTU_OPNFV_PATH') +mirror_ubuntu_tmp_path = '{0}.tmp'.format(mirror_ubuntu_path) +mirror_ubuntu_merge = get_env('MIRROR_UBUNTU_MERGE') +cfg_mm_ubuntu = '{0}/ubuntu_mirror_local.yaml'.format(cfg_dir) +pattern_file = open('fuel-mirror/contrib/fuel_mirror/data/ubuntu.yaml').read() +pattern = yaml.load(pattern_file) +opnfv_pkgs_yaml = open(cfg_p_opnfv).read() +opnfv_pkgs = yaml.load(opnfv_pkgs_yaml) + +# FIXME: Packetary solves missing dependecies by also accepting +# different packages that provide the same name (e.g. "ifupdown" dependency +# is satisfied by "netscript" package from "universe" repo-component). +# Work around this by resolving all deps in "main" repo-component, +# then scan and keep only latest debs for the whole repo. +mirror_ubuntu_resolve_main_deps = True + +# Create local partial mirror using packetary, one arch at a time +for arch in ubuntu_arch.split(' '): + # Mirror / Package env vars, arch-overrideable + mos_ubuntu = get_env('MIRROR_MOS_UBUNTU', arch) + mos_ubuntu_root = get_env('MIRROR_MOS_UBUNTU_ROOT', arch) + mirror_ubuntu = get_env('MIRROR_UBUNTU_URL', arch) + plugins = get_env('BUILD_FUEL_PLUGINS', arch) + if plugins is None: + plugins = get_env('PLUGINS', arch) + + # Mirror / Package list configuration files (arch-specific) + cfg_m_mos = '{0}/mos_{1}_mirror.yaml'.format(cfg_dir, arch) + cfg_m_ubuntu = '{0}/ubuntu_{1}_mirror.yaml'.format(cfg_dir, arch) + cfg_p_ubuntu = '{0}/ubuntu_{1}_packages.yaml'.format(cfg_dir, arch) + cfg_m_ubuntu_main = '{0}/ubuntu_{1}_mirror_main.yaml'.format(cfg_dir, arch) + cfg_p_ubuntu_main = '{0}/ubuntu_{1}_packages_main.yaml'.format(cfg_dir, arch) + + # Mirror config fork before customizing (arch-specific) + arch_group_mos = 'mos_{0}'.format(arch) + arch_group_ubuntu = 'ubuntu_{0}'.format(arch) + arch_packages = 'packages_{0}'.format(arch) + pattern['groups'][arch_group_mos] = copy.deepcopy(pattern['groups']['mos']) + pattern['groups'][arch_group_ubuntu] = copy.deepcopy(pattern['groups']['ubuntu']) + pattern[arch_packages] = pattern['packages'] + + # Mirror config update & conversion to packetary input + group_main_ubuntu = dict() + for group in pattern['groups'][arch_group_mos]: + group['uri'] = "http://{}{}".format(mos_ubuntu, mos_ubuntu_root) + group['suite'] = group['suite'].replace('$mos_version', mos_version) + group['section'] = group['section'].split() + for group in pattern['groups'][arch_group_ubuntu]: + group['uri'] = mirror_ubuntu + # FIXME: At `create`, packetary insists on copying all pkgs to dest dir, + # so configure it for another dir, which will replace the orig at the end. + group['path'] = mirror_ubuntu_tmp_path + group['section'] = group['section'].split() + if not group_main_ubuntu and 'main' in group: + group_main_ubuntu = [ copy.deepcopy(group) ] + group_main_ubuntu[0]['section'] = [ 'main' ] + + # Mirror config dump: MOS (for dep resolution), Ubuntu, Ubuntu[main] + write_cfg_file(cfg_m_mos, pattern['groups'][arch_group_mos]) + write_cfg_file(cfg_m_ubuntu, pattern['groups'][arch_group_ubuntu]) + if mirror_ubuntu_resolve_main_deps or mirror_ubuntu_merge is None: + write_cfg_file(cfg_m_ubuntu_main, group_main_ubuntu) + if mirror_ubuntu_merge is not None: + # FIXME: For multiarch, only one dump would be enough + group_main_ubuntu[0]['origin'] = 'Ubuntu' + group_main_ubuntu[0]['path'] = mirror_ubuntu_path + group_main_ubuntu[0]['uri'] = mirror_ubuntu_path + write_cfg_file(cfg_mm_ubuntu, group_main_ubuntu[0]) + + # Package list conversion from `old fuel-mirror` to `packetary` style + OPNFV + unresolved_pkgs = dict() + unresolved_pkgs['packages'] = list() + unresolved_pkgs['mandatory'] = 'exact' + if opnfv_pkgs['packages'] is not None: + unresolved_pkgs['packages'] = opnfv_pkgs['packages'] + for pkg in pattern['packages']: + unresolved_pkgs['packages'] += [ {'name': pkg} ] + + # OPNFV plugins dependency resolution + for plugin in plugins.split(): + path = "../{}/packages.yaml".format(plugin) + if os.path.isfile(path): + f = open(path).read() + plugin_yaml = yaml.load(f) + plugin_set = set(plugin_yaml['packages']) + main_set = set(pattern['packages']) + new_packages = plugin_set - main_set + print('Plugin {0} requires new packages for arch [{1}]: {2}' + .format(plugin, arch, ', '.join(new_packages))) + for pkg in new_packages: + unresolved_pkgs['packages'] += [ {'name': pkg} ] + + # Package list (reduced, i.e. no MOS/OPNFV plugin deps) + # FIXME: This helps work around packetary solving main deps from universe + if mirror_ubuntu_resolve_main_deps or mirror_ubuntu_merge is None: + write_cfg_file(cfg_p_ubuntu_main, unresolved_pkgs) + + # Mirror package list (full, including MOS/OPNFV plugin deps) + unresolved_pkgs['packages'] += get_unres_pkgs(arch, cfg_m_mos)['packages'] + write_cfg_file(cfg_p_ubuntu, unresolved_pkgs) + do_partial_mirror(arch, cfg_m_ubuntu, cfg_p_ubuntu) + if mirror_ubuntu_resolve_main_deps or mirror_ubuntu_merge is None: + # Ubuntu[main] must be evaluated after Ubuntu + do_partial_mirror(arch, cfg_m_ubuntu_main, cfg_p_ubuntu_main) + +if mirror_ubuntu_merge is None: + shutil.move(mirror_ubuntu_tmp_path, mirror_ubuntu_path) +else: + # Construct single-component mirror from all components + for arch in ubuntu_arch.split(' '): + cfg_pp_ubuntu = '{0}/ubuntu_{1}_packages_paths.yaml'.format(cfg_dir, arch) + # FIXME: We need scanpackages to omit older DEBs + # Inspired from http://askubuntu.com/questions/198474/ + os.system('dpkg-scanpackages -a {0} {1} 2>/dev/null | ' + 'grep -e "^Filename:" | sed "s|Filename: |- file://|g" > {2}' + .format(arch, mirror_ubuntu_tmp_path, cfg_pp_ubuntu)) + do_local_repo(arch, cfg_mm_ubuntu, cfg_pp_ubuntu) + shutil.rmtree(mirror_ubuntu_tmp_path) diff --git a/build/f_isoroot/f_repobuild/opnfv_packages.yaml b/build/f_isoroot/f_repobuild/opnfv_packages.yaml new file mode 100644 index 0000000..3f5c59f --- /dev/null +++ b/build/f_isoroot/f_repobuild/opnfv_packages.yaml @@ -0,0 +1,15 @@ +############################################################################## +# Copyright (c) 2016 Enea AB and others. +# Alexandru.Avadanii@enea.com +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Apache License, Version 2.0 +# which accompanies this distribution, and is available at +# http://www.apache.org/licenses/LICENSE-2.0 +############################################################################## + +# This file lists packages that should be present in the local Ubuntu mirror, +# but are not direct dependencies of any other packages from MOS or Ubuntu. +# e.g.: additional kernels, bootloaders etc. + +packages: + - name: vgabios diff --git a/deploy/config/dea_base.yaml b/deploy/config/dea_base.yaml index f5a03e2..370a75a 100644 --- a/deploy/config/dea_base.yaml +++ b/deploy/config/dea_base.yaml @@ -633,22 +633,10 @@ settings: value: - name: ubuntu priority: null - section: main universe multiverse + section: main suite: trusty type: deb uri: http://10.20.0.2:8080/mirrors/ubuntu/ - - name: ubuntu-updates - priority: null - section: main universe multiverse - suite: trusty-updates - type: deb - uri: http://10.20.0.2:8080/mirrors/ubuntu/ - - name: ubuntu-security - priority: null - section: main universe multiverse - suite: trusty-security - type: deb - uri: http://10.20.0.2:8080/mirrors/ubuntu/ - name: mos priority: 1050 section: main restricted