fuel, docs: fuel-menu: Align defaults with OPNFV

Since Armband alters the bootstrap package list and/or kernel version,
hanlded previously by fuel_bootstrap_cli.yaml, we need to refactor our
patches a bit.

NOTE: deploy-cache: We no longer include bootstrap package list in the
bootstrap fingerprint; we only rely on the repo mirrors fingerprints,
since bootstrap package list is mostly static across release cycles.

Upstream-bug: https://bugs.launchpad.net/fuel/+bug/1679636

JIRA: FUEL-266
JIRA: ARMBAND-225

Change-Id: I6c1b090df6eff69d60fa7ca779b3dde527e77879
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>

2 files changed, 269 insertions, 11 deletions

diff --git a/patches/opnfv-fuel/upstream-backports/0000-fuel-menu-Align-defaults-w-OPNFV-bootstrap-cfg.patch b/patches/opnfv-fuel/upstream-backports/0000-fuel-menu-Align-defaults-w-OPNFV-bootstrap-cfg.patch
new file mode 100644
index 00000000..a2f922c5
--- /dev/null
+++ b/patches/opnfv-fuel/upstream-backports/0000-fuel-menu-Align-defaults-w-OPNFV-bootstrap-cfg.patch
@@ -0,0 +1,256 @@
+From: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+Date: Sun, 9 Apr 2017 14:10:21 +0200
+Subject: [PATCH] fuel-menu: Align defaults w/ OPNFV bootstrap cfg
+
+Instead of overwriting fuel_bootstrap_cli.yaml during post-install,
+with a static config which we need to manually sync in OPNFV after
+each upstream change, make the whole thing dynamic by syncing (via
+patch) fuel-menu's default settings with OPNFV bootstrap.
+
+This allows us to drop the static fuel_bootstrap_cli.yaml.
+
+JIRA: FUEL-266
+
+Change-Id: I132f2a272c764ad0fc84b3e0e0e1222b1fab1cd2
+Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
+---
+ .../f_bootstrap/post-scripts/03_install_repo.sh    |  5 --
+ build/f_isoroot/f_repobuild/Makefile               |  3 +-
+ build/f_isoroot/f_repobuild/config.mk              |  2 +-
+ .../f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml  | 74 ----------------------
+ build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py |  7 +-
+ ...s-OPNFV-bootstrap-Default-to-local-mirror.patch | 66 +++++++++++++++++++
+ 6 files changed, 73 insertions(+), 84 deletions(-)
+ delete mode 100644 build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml
+ create mode 100644 build/f_repos/patch/fuel-menu/0001-settings-OPNFV-bootstrap-Default-to-local-mirror.patch
+
+diff --git a/build/f_isoroot/f_bootstrap/post-scripts/03_install_repo.sh b/build/f_isoroot/f_bootstrap/post-scripts/03_install_repo.sh
+index be17e7e..431577c 100755
+--- a/build/f_isoroot/f_bootstrap/post-scripts/03_install_repo.sh
++++ b/build/f_isoroot/f_bootstrap/post-scripts/03_install_repo.sh
+@@ -26,9 +26,4 @@ if [ $? -ne 0 ]; then
+   echo "Error removing /opt/opnfv/nailgun directory!"
+   exit 1
+ fi
+-mv /opt/opnfv/fuel_bootstrap_cli.yaml /etc/fuel-bootstrap-cli/fuel_bootstrap_cli.yaml
+-if [ $? -ne 0 ]; then
+-  echo "Error moving bootstrap image configuration!"
+-  exit 1
+-fi
+ echo "Done installing pre-build repo"
+diff --git a/build/f_isoroot/f_repobuild/Makefile b/build/f_isoroot/f_repobuild/Makefile
+index 891712b..c61d1ad 100644
+--- a/build/f_isoroot/f_repobuild/Makefile
++++ b/build/f_isoroot/f_repobuild/Makefile
+@@ -27,6 +27,8 @@ nailgun:
+ 	rm -Rf nailgun packetary opnfv_config && mkdir opnfv_config
+ 	# We will analyze fuel-web's fixture files for package lists
+ 	ln -sf ${F_SUBMOD_DIR}/fuel-web fuel-web
++	# Same for fuel-agent's bootstrap package list
++	ln -sf ${F_SUBMOD_DIR}/fuel-agent fuel-agent
+ 	git clone --quiet $(PACKETARY_REPO)
+ 	if [ -n $(PACKETARY_COMMIT) ]; then \
+ 		git -C packetary checkout $(PACKETARY_COMMIT); \
+@@ -48,7 +50,6 @@ release:nailgun
+ 	@rm -Rf ../release/opnfv/nailgun
+ 	@mkdir -p ../release/opnfv
+ 	@cp -Rp nailgun ../release/opnfv/nailgun
+-	@cp fuel_bootstrap_cli.yaml ../release/opnfv/
+
+ ############################################################################
+ # Cache operations - only used when building through ci/build.sh
+diff --git a/build/f_isoroot/f_repobuild/config.mk b/build/f_isoroot/f_repobuild/config.mk
+index 986ea9e..68101d6 100644
+--- a/build/f_isoroot/f_repobuild/config.mk
++++ b/build/f_isoroot/f_repobuild/config.mk
+@@ -19,5 +19,5 @@ export MIRROR_UBUNTU_ROOT_arm64=ubuntu-ports
+
+ # Merge all local mirror repo components/section into single "main"
+ # NOTE: When changing this, make sure to also update all consumer config, like:
+-# - fuel_bootstrap_cli.yaml
++# - fuel-menu/fuelmenu/settings.yaml
+ export MIRROR_UBUNTU_MERGE=true
+diff --git a/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml b/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml
+deleted file mode 100644
+index 81ca6eb..0000000
+--- a/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml
++++ /dev/null
+@@ -1,74 +0,0 @@
+-##############################################################################
+-# Copyright (c) 2015,2016 Ericsson AB and others.
+-# mskalski@mirantis.com
+-# All rights reserved. This program and the accompanying materials
+-# are made available under the terms of the Apache License, Version 2.0
+-# which accompanies this distribution, and is available at
+-# http://www.apache.org/licenses/LICENSE-2.0
+-##############################################################################
+----
+-  root_ssh_authorized_file: /root/.ssh/id_rsa.pub
+-  extend_kopts: "biosdevname=0 net.ifnames=1"
+-  ubuntu_release: xenial
+-  extra_dirs:
+-    - /usr/share/fuel_bootstrap_cli/files/xenial
+-  output_dir: /tmp/
+-  kernel_flavor: linux-image-generic-lts-xenial
+-  packages:
+-    - daemonize
+-    - fuel-agent
+-    - hwloc
+-    - i40e-dkms
+-    - linux-firmware
+-    - linux-headers-generic
+-    - live-boot
+-    - live-boot-initramfs-tools
+-    - mc
+-    - mcollective
+-    - msmtp-mta
+-    - multipath-tools
+-    - multipath-tools-boot
+-    - nailgun-agent
+-    - nailgun-mcagents
+-    - network-checker
+-    - ntp
+-    - ntpdate
+-    - openssh-client
+-    - openssh-server
+-    - puppet
+-    - squashfs-tools
+-    - ubuntu-minimal
+-    - vim
+-    - wget
+-    - xz-utils
+-    # NOTE(el): Packages required for new generation
+-    # network checker to be run without an access
+-    # to repositories.
+-    - sysfsutils
+-    - bridge-utils
+-    - ifenslave
+-    - irqbalance
+-    - iputils-arping
+-  bootstrap_images_dir: /var/www/nailgun/bootstraps
+-  active_bootstrap_symlink: /var/www/nailgun/bootstraps/active_bootstrap
+-  flavor: ubuntu
+-  http_proxy: ""
+-  https_proxy: ""
+-  repos:
+-    - name: ubuntu
+-      section: "main"
+-      uri: "http://127.0.0.1:8080/mirrors/ubuntu"
+-      priority:
+-      suite: xenial
+-      type: deb
+-    - name: mos
+-      section: "main restricted"
+-      uri: "http://127.0.0.1:8080/ubuntu/x86_64"
+-      priority: 1050
+-      suite: mos10.0
+-      type: deb
+-  skip_default_img_build: false
+-  direct_repo_addresses:
+-    - "127.0.0.1"
+-  # User can provide default hashed root password for bootstrap image
+-  # hashed_root_password: "$6$IInX3Cqo$5xytL1VZbZTusOewFnG6couuF0Ia61yS3rbC6P5YbZP2TYclwHqMq9e3Tg8rvQxhxSlBXP1DZhdUamxdOBXK0."
+diff --git a/build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py b/build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py
+index 1a603ee..7c38363 100755
+--- a/build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py
++++ b/build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py
+@@ -25,7 +25,7 @@
+ #    "main".
+ ##############################################################################
+ # Mirror build steps (for EACH architecture in UBUNTU_ARCH):
+-# 1. Collect bootstrap package deps from <fuel_bootstrap_cli.yaml>;
++# 1. Collect bootstrap package deps from fuel-agent's <settings.yaml.sample>;
+ # 2. Collect all fixture release packages from fuel-web's <openstack.yaml>;
+ # 3. Parse new "opnfv_config.yaml" list of packages (from old fuel-mirror);
+ # 4. Inherit enviroment variable(s) for mirror URLs, paths etc.
+@@ -156,7 +156,8 @@ MIRROR_UBUNTU_PATH = get_env('MIRROR_UBUNTU_OPNFV_PATH')
+ MIRROR_UBUNTU_TMP_PATH = '{0}.tmp'.format(MIRROR_UBUNTU_PATH)
+ MIRROR_UBUNTU_MERGE = get_env('MIRROR_UBUNTU_MERGE')
+ CFG_MM_UBUNTU = '{0}/ubuntu_mirror_local.yaml'.format(CFG_D)
+-FUEL_BOOTSTRAP_CLI_FILE = open('fuel_bootstrap_cli.yaml').read()
++FUEL_BOOTSTRAP_CLI_FILE = open('fuel-agent/contrib/fuel_bootstrap/'
++    'fuel_bootstrap_cli/fuel_bootstrap/settings.yaml.sample').read()
+ FUEL_BOOTSTRAP_CLI = yaml.load(FUEL_BOOTSTRAP_CLI_FILE)
+ FIXTURE_FILE = open('fuel-web/nailgun/nailgun/fixtures/openstack.yaml').read()
+ FIXTURE = yaml.load(FIXTURE_FILE)
+@@ -215,7 +216,7 @@ for arch in UBUNTU_ARCH.split(' '):
+         write_cfg_file(CFG_MM_UBUNTU, group_main_ubuntu[0])
+
+     # Collect package dependencies from:
+-    ## 1. fuel_bootstrap_cli.yaml (bootstrap image additional packages)
++    ## 1. fuel_bootstrap_cli (bootstrap image additional packages)
+     legacy_unresolved = legacy_diff(None, FUEL_BOOTSTRAP_CLI['packages'] + [
+             FUEL_BOOTSTRAP_CLI['kernel_flavor'],
+             FUEL_BOOTSTRAP_CLI['kernel_flavor'].replace('image', 'headers')],
+diff --git a/build/f_repos/patch/fuel-menu/0001-settings-OPNFV-bootstrap-Default-to-local-mirror.patch b/build/f_repos/patch/fuel-menu/0001-settings-OPNFV-bootstrap-Default-to-local-mirror.patch
+new file mode 100644
+index 0000000..456eea7
+--- /dev/null
++++ b/build/f_repos/patch/fuel-menu/0001-settings-OPNFV-bootstrap-Default-to-local-mirror.patch
+@@ -0,0 +1,66 @@
++From: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
++Date: Sun, 9 Apr 2017 13:49:28 +0200
++Subject: [PATCH] settings: OPNFV bootstrap: Default to local mirror
++
++JIRA: https://jira.opnfv.org/browse/FUEL-266
++
++Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
++---
++ fuelmenu/settings.yaml | 36 +++---------------------------------
++ 1 file changed, 3 insertions(+), 33 deletions(-)
++
++diff --git a/fuelmenu/settings.yaml b/fuelmenu/settings.yaml
++index e93309a..aaf07cd 100644
++--- a/fuelmenu/settings.yaml
+++++ b/fuelmenu/settings.yaml
++@@ -13,47 +13,17 @@ BOOTSTRAP:
++   http_proxy: ""
++   https_proxy: ""
++   repos:
++-  - name: "ubuntu"
++-    section: "main universe multiverse"
++-    uri: "http://archive.ubuntu.com/ubuntu"
+++  - name: "ubuntu-local"
+++    section: "main"
+++    uri: "http://127.0.0.1:8080/mirrors/ubuntu"
++     priority: null
++     suite: "${codename}"
++     type: "deb"
++-  - name: "ubuntu-updates"
++-    section: "main universe multiverse"
++-    uri: "http://archive.ubuntu.com/ubuntu"
++-    priority: null
++-    suite: "${codename}-updates"
++-    type: "deb"
++-  - name: "ubuntu-security"
++-    section: "main universe multiverse"
++-    uri: "http://archive.ubuntu.com/ubuntu"
++-    priority: null
++-    suite: "${codename}-security"
++-    type: "deb"
++   - name: "mos"
++     section: "main restricted"
++     uri: "http://127.0.0.1:8080/ubuntu/x86_64"
++     priority: 1050
++     suite: "mos${mos_version}"
++     type: "deb"
++-  - name: "mos-updates"
++-    section: "main restricted"
++-    uri: "http://mirror.fuel-infra.org/mos-repos/ubuntu/${mos_version}"
++-    priority: 1050
++-    suite: "mos${mos_version}-updates"
++-    type: "deb"
++-  - name: "mos-security"
++-    section: "main restricted"
++-    uri: "http://mirror.fuel-infra.org/mos-repos/ubuntu/${mos_version}"
++-    priority: 1050
++-    suite: "mos${mos_version}-security"
++-    type: "deb"
++-  - name: "mos-holdback"
++-    section: "main restricted"
++-    uri: "http://mirror.fuel-infra.org/mos-repos/ubuntu/${mos_version}"
++-    priority: 1100
++-    suite: "mos${mos_version}-holdback"
++-    type: "deb"
++ PRODUCTION: docker
++ FEATURE_GROUPS: []
diff --git a/patches/opnfv-fuel/upstream-backports/0005-CI-deploy-cache-Store-and-reuse-deploy-artifacts.patch b/patches/opnfv-fuel/upstream-backports/0005-CI-deploy-cache-Store-and-reuse-deploy-artifacts.patch
index 10a7f2c3..9997dedf 100644
--- a/patches/opnfv-fuel/upstream-backports/0005-CI-deploy-cache-Store-and-reuse-deploy-artifacts.patch
+++ b/patches/opnfv-fuel/upstream-backports/0005-CI-deploy-cache-Store-and-reuse-deploy-artifacts.patch
@@ -36,10 +36,10 @@ Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
  deploy/cloud/configure_settings.py                 |   4 +
  deploy/cloud/deployment.py                         |  12 +
  deploy/deploy.py                                   |  25 +-
- deploy/deploy_cache.py                             | 312 +++++++++++++++++++++
+ deploy/deploy_cache.py                             | 314 +++++++++++++++++++++
  deploy/deploy_env.py                               |  13 +-
  deploy/install_fuel_master.py                      |   9 +-
- 8 files changed, 470 insertions(+), 9 deletions(-)
+ 8 files changed, 472 insertions(+), 9 deletions(-)
  create mode 100644 build/f_repos/patch/fuel-main/0006-bootstrap_admin_node.sh-deploy_cache-install-hook.patch
  create mode 100644 deploy/deploy_cache.py
 
@@ -372,10 +372,10 @@ index 7648baf..ee3cb7a 100755
                'deploy_log': args.deploy_log}
 diff --git a/deploy/deploy_cache.py b/deploy/deploy_cache.py
 new file mode 100644
-index 0000000..7df43c6
+index 0000000..30bfe30
 --- /dev/null
 +++ b/deploy/deploy_cache.py
-@@ -0,0 +1,312 @@
+@@ -0,0 +1,314 @@
 +###############################################################################
 +# Copyright (c) 2016 Enea AB and others.
 +# Alexandru.Avadanii@enea.com
@@ -433,7 +433,7 @@ index 0000000..7df43c6
 +# Cache fingerprint covers:
 +# - bootstrap:
 +#   - local mirror contents
-+#   - package list (and everything else in fuel_bootstrap_cli.yaml)
++#   - FIXME(armband): [disabled] package list (old fuel_bootstrap_cli.yaml)
 +# - target image:
 +#   - local mirror contents
 +#   - package list (determined from DEA)
@@ -472,8 +472,8 @@ index 0000000..7df43c6
 +# Relative path for collecting target image(s) for deployed enviroment
 +NAILGUN_TIMAGES_SUBDIR = TARGETIMAGES
 +
-+# OPNFV Fuel bootstrap settings file that will be injected at deploy
-+ISO_BOOTSTRAP_CLI_YAML = '/opnfv/fuel_bootstrap_cli.yaml'
++# FIXME(armband): re-include package list (old fuel_bootstrap_cli.yaml)
++# ISO_BOOTSTRAP_CLI_YAML = '/opnfv/fuel_bootstrap_cli.yaml'
 +
 +# OPNFV Deploy Cache path on Fuel Master, where artifacts will be injected
 +REMOTE_CACHE_PATH = '/var/cache/opnfv'
@@ -522,10 +522,12 @@ index 0000000..7df43c6
 +    def __fingerprint_bootstrap(self, chroot_path):
 +        """Collect bootstrap image metadata fingerprints"""
 +        # FIXME(armband): include 'extra_dirs' contents
-+        cli_yaml_path = os.path.join(chroot_path, ISO_BOOTSTRAP_CLI_YAML[1:])
-+        bootstrap_cli_yaml = open(cli_yaml_path).read()
-+        bootstrap_data = yaml.load(bootstrap_cli_yaml)
-+        sorted_data = json.dumps(bootstrap_data, sort_keys=True)
++        sorted_data = ''
++        # FIXME(armband): re-include package list (old fuel_bootstrap_cli.yaml)
++        # cli_yaml_path = os.path.join(chroot_path, ISO_BOOTSTRAP_CLI_YAML[1:])
++        # bootstrap_cli_yaml = open(cli_yaml_path).read()
++        # bootstrap_data = yaml.load(bootstrap_cli_yaml)
++        # sorted_data = json.dumps(bootstrap_data, sort_keys=True)
 +        self.fingerprints[BOOTSTRAPS] = hashlib.sha1(sorted_data).hexdigest()
 +
 +    def __fingerprint_target(self, dea_file):