summaryrefslogtreecommitdiffstats
path: root/mcp/scripts
AgeCommit message (Collapse)AuthorFilesLines
2017-11-27Add pre-{install,purge} support for base imageAlexandru Avadanii2-5/+144
Extend <lib.sh> and its invocation from <ci.deploy.sh> with support for modifying foundation node VMs base image prior to using it with: - additional APT GPG keys; - additional APT repos; - packages to pre-install; - packages to pre-remove; - (non-configurable) cloud init datasource via NoCloud only, so VCP VMs won't wait for metadata service; While at it, re-use the resulting image as a base for another round of pre-patching (same operations as above are supported) to provide a base image for VCP VMs. Add AArch64-specific configuration based on new mechanisms: - pre-install linux-image-generic-hwe-16.04-edge (and headers) for foundation node and VCP (common) image (also requires new repo and its key); - pre-install cloud-init for VCP image (it should already be installed, but script needs non-empty config for VCP to create the VCP image and transfer it over to Salt Master); NOTE: cloud-init is required on VCP VMs for DHCP on 1st iface. JIRA: FUEL-309 Change-Id: I7dcaf0ffd9c57009133c6d339496ec831ab14375 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-23salt.sh: Rework `git -C` for older git compatAlexandru Avadanii1-1/+1
CentOS 7.4 provides git 1.8.3, while `git -C` support was introduced in 1.8.5, so rework <salt.sh> to not rely on this parameter. Change-Id: Ied88466b2b3d91a3317922fa08f60afe934358a0 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-21ci/deploy.sh: Add new `-E` arg for env eraseAlexandru Avadanii1-12/+38
NOTE: In order to undefine VCP VMs with NVRAM (e.g. AArch64 VMs using AAVMF), an additional parameter should be passed to libvirt by Salt virt core module (equivalent to `virsh undefine --nvram`). While at it, pass CI_DEBUG, ERASE_ENV enviroment variables to state execution, and stop force-applying patches. Also refactor the rsync between foundation node and Salt master, so the whole git repo is copied as </root/opnfv>, and <root/fuel> becomes a link to it; useful for Armband, where 'fuel' is a git submodule. Fix .git paths after rsync, so git submodules work as expected in cfg01 repos. JIRA: FUEL-307 Change-Id: Ic62f03e786581c019168c50ccc50107238021d7f Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-12salt formulas: Pin via reclass instead of salt.shAlexandru Avadanii1-8/+0
Previous commit pinned the salt-formula-package by installing a specific version after OPNFV-specific patches were applied by `patch.sh`. To prevent patch bypass, move the pin to reclass model. While at it, also move salt-formula-neutron pinning to reclass to bypass wrong minimum length of 2 for version-pinned formula pkgs. Change-Id: I35c2773c130d79ec919685cd771311d6c11e11a8 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-11salt.sh: Pin MaaS formula to last known stable verAlexandru Avadanii1-0/+3
Recent upstream changes broke MaaS machine status parsing. Pin the formula to the previous version until upstream provides a fixed package or a repo snapshot baseline. Change-Id: I26ebd60f40812ae47da0ca28d19c9fdcac7920da Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-08lib.sh: Cleanup previous /boot/efi contentsAlexandru Avadanii1-1/+2
Some UEFI firmwares insist on scanning removable drives, even when boot entries were deleted from UEFI boot list (board flash). To work around this, remove contents of </boot/efi/*>, so scanning won't identify any valid EFI binaries. Another option would be erasing partition tables, but identifying the underlying disk(s) is more complicated, especially when using LVM/RAID etc. Change-Id: I9949b99b139b1642e3bd8f04de3bd5ef74d1ecc5 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-07Revert "Freeze the galera formula to avoid update"Michael Polenchuk1-4/+0
This reverts commit f49340e1aac3f5e8bb009a9d0d35aa6fedba7384. Version has been set back to 5.6 in upstream. Change-Id: I06d6865e59cfbc5055322eb31cbb237075ebdcb1 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-10-31Freeze the galera formula to avoid updateMichael Polenchuk1-0/+4
Change-Id: Ibf03a7dc2ed58a749476ae0bdc6cad2bba8783fb Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-10-31Connect proxy nodes to public networkMichael Polenchuk1-0/+0
cloudify vnf requires public enpoints to be available from instance with manager, so connect nodes w/ nginx to routed public network. JIRA: FUEL-286 Change-Id: Id0caa699519b9b46dad51deac6955afe6a0050de Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-10-28lib.sh: rm Ubuntu boot entry on EFI systemsAlexandru Avadanii1-0/+10
On EFI-enabled systems, grub-install from grub-efi-* package installs a boot entry named "ubuntu". MaaS relies on IPMI to set boot order to PXE first; however on systems with buggy firmware or without full IPMI support, that fails, leading to booting Ubuntu from hard disk instead. Work around this by clearing any previous Ubuntu boot entry from board flash, before starting a new baremetal deploy. NOTE: This only runs against nodes that are online from a previous deploy. Closes: ARMBAND-47 Change-Id: I1c4ece09e42845ce2a1b7119ec69e46e5ca12376 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-27Merge "lib.sh: Fix compatibility with bash 4.2"Alexandru Avadanii1-16/+16
2017-10-27lib.sh: Fix compatibility with bash 4.2Alexandru Avadanii1-16/+16
JIRA: FUEL-296 Change-Id: Ide9f9333fe9b44ff6b78678064f8e67f05aabd42 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-26AArch64: Switch jumpserver VMs to virtio videoAlexandru Avadanii1-2/+2
Drop vgabios dependency by switching video from VGA to virtio for all VMs spawned on the jumpserver. NOTE: This requires virtualization packages on the jumpserver to be up to date (e.g. libvirt, QEMU). JIRA: ARMBAND-306 Change-Id: I73913e1ae8584f4e73b92994f78f7ec363cba3ec Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-20lib.sh: Fix locals override in nested invocationsAlexandru Avadanii1-11/+14
'wait_for' bash function is nested in another 'wait_for' call in some places, which leads to inner calls interfering with outer calls by overriding the locally scoped variables, including the 'attempt' internal counter. In some cases, the outer 'wait_for' would exit after a single attempt. Fix that by running all contents of `wait_for` inside a subshell, which inherits outer calls variables, but does not override them when the inner call is finished. Change-Id: I450eda3d023af2380c61ee930071fbfc393a5645 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-18Horizon: service: Default to v2 APIAlexandru Avadanii1-1/+1
Horizon service definition hardcodes the API version to v3, and also overrides the value in our reclass model during interpolation. Adjust the default to '2'. While at it, move 'reclass' patching later in `salt.sh`, after the salt formula packages are installed. JIRA: FUEL-284 Change-Id: I6f63b4d00ae85c82b076b3efef857e7cc4f3fd35 Signed-off-by: Ting Wu <ting.wu@enea.com> Signed-off-by: Junaid Ali <junaidali.yahya@gmail.com> Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-16Merge "salt.sh: DEBIAN_FRONTEND=noninteractive"Alexandru Avadanii1-0/+1
2017-10-16[ovs/dpdk] Return true on linux.network stateMichael Polenchuk1-1/+1
linux.network state is expected to be down until node reboot to activate required hugepages therefore return true in any case. Change-Id: I61deac12b9ac212c5f79cdcce30cda8e5d24413b Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-10-16salt.sh: DEBIAN_FRONTEND=noninteractiveAlexandru Avadanii1-0/+1
Change-Id: Ibb1e78563043298dffdb4b7e9091d3f582e189fc Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-15states: Break on error, retry states up to 5 timesAlexandru Avadanii5-12/+20
While applying scenario states, break on error, and retry failed state up to 5 times. Apply the same behavior for `salt.sh`. Add new deploy parameter, '-D', backed up by 'CI_DEBUG' env var, which gates deploy sh scripts logging (set -x). Also extend '-f' deploy parameter, allowing it to be specified more than once; the first occurence will skip infra VM creation, but still sync reclass & other config from local repo, while a second occurence will also disable config sync. To prevent glusterfs client state from failing due to non-existent nova user/group, move it after nova:compute's nova state is applied. Change-Id: I234e126e16be0e133d878957bd88fed946955de8 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-14Add license headers where missingAlexandru Avadanii7-0/+53
While at it, compact 'set' into bash shebang where possible and add `make patches-copyright` target to simplify adding patch license headers. Change-Id: I0c841de72e5709e5eef915a52c5ec4a7fc0f7c37 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-14Drop infinite loops in favor of finite wait_forAlexandru Avadanii2-4/+3
While at it, fix some shellcheck warnings, and s/fgrep/grep -F/g. Change-Id: I093b7b4c196731b1ecc0c27a4111955b2e412762 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-14Merge "states: Split virtual_control_plane from maas"Alexandru Avadanii1-10/+23
2017-10-14states: Split virtual_control_plane from maasAlexandru Avadanii1-10/+23
We should eventually also support baremetal deploys without a virtualized control plane (VCP), so decouple MaaS provisioning from VCP provisioning. While at it, move "wait_for" bash function from maas state to common library file, lib.sh. Change-Id: I32c33135655cb6aceae901a5f92b51265a8c84b4 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-13patch.sh: Exit on patch apply failureAlexandru Avadanii1-1/+2
Patches that fail to apply are silently skipped. Since retrying is not possible, deploy moves on, making the problem hard to observe. Change-Id: I3fe01cef6b245de53767a660b9b4fdfae0d5d0ff Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-05log.sh: Fix sourcing of globals.shAlexandru Avadanii1-2/+2
Fix sourcing of globals.sh from log.sh when running with a CWD different from scripts dir. Change-Id: I680531022d30b57f650a612493f40fdb3e073dc2 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-04Merge "ci/deploy.sh: Move global vars to separate file"Alexandru Avadanii2-0/+23
2017-10-04Merge "Identify jump host bridges based on IDF / PDF nets"Alexandru Avadanii3-9/+3
2017-10-03ci/deploy.sh: Move global vars to separate fileAlexandru Avadanii2-0/+23
log.sh should also work as standalone script, so factor out global variables from <ci/deploy.sh> into <mcp/scripts/globals.sh>, which should be sourced by log.sh when required variables are not set. Change-Id: I5a5bb85ee5835f567d6910b11c137d95e2147714 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-03Identify jump host bridges based on IDF / PDF netsAlexandru Avadanii3-9/+3
- minor refactor of runtime templates parsing to allow var expansion; - parse <pod_config.yml> into shell vars, match dynamically networks from PDF to IP addresses on bridges of current jumphost; - keep old '-B' parameter in <ci/deploy.sh>, use it for providing fallback values in case there's no bridge name specified via IDF and no IP on the jumphost for one or more of the PDF networks; - re-enable dry-run to ease testing of the above; - add sample 'idf-pod1.yaml' to <mcp/config/labs/local>; The new behavior will try to determine the jump host bridge names: 1. Based on IDF mapping, if available 2. Based on PDF network matching with IP addrs on jumphost; 3. Fallback to values passed via '-B'; 4. Fallback to default values hardcoded in the deploy script; Later, we will drop MaaS network env vars in favor of PDF vars, once the PDF template is generating them. Change-Id: If9cd65d310c02965b2e2bfa06a0d7e0f97f1dd48 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-03Apply neutron formula patch after formula updateMichael Polenchuk1-0/+1
Change-Id: I8dba3676adc8cc49731d91db7cc028a9c5b1627d Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-10-03salt.sh: Stick to older salt-formula-neutronAlexandru Avadanii1-0/+4
Recent changes to salt-formula-neutron, more precisely [1], broke the compatibility with our current reclass model [2], which was not updated to reflect the formula change. Instead of locally patching the reclass model, install the latest compatible version of the Neutron formula: - 2016.12.1+201709251458.f0607d9-1xenial1; [1] https://github.com/salt-formulas/salt-formula-neutron/commit/ 79ffa26858d682f404984175dc1ab93863a149ad [2] https://github.com/Mirantis/reclass-system-salt-model Change-Id: Ia1bc9de9a1543f7888688be76ff3513e6b4f062b Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-27pharos git submodule: Bump for latest PDF adapterAlexandru Avadanii1-0/+0
Change-Id: I658c1dfeb369adf74038abd0ce1b593d40303527 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-27Merge "PDF parsing support"Alexandru Avadanii2-5/+7
2017-09-27PDF parsing supportAlexandru Avadanii2-5/+7
- add new git submodule pointing to OPNFV Pharos; - use Pharos 'generate_config.sh' to parse the PDF using the Fuel installer adapter and generate <pod_config.yml> dynamically; - build <pod_config.yml> outside current git repo and sync separately to prevent sensitive data leak; - add <pod1.yaml> PDF sample based on LF-POD2, should be used with 'ci/deploy.sh -l local -p pod1'; Change-Id: I4e1b95f180bcd5ade5d86f516628eb8edbe64b1c Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-26lib.sh: Use host bridge in favor of virsh networkAlexandru Avadanii1-4/+5
If a linux bridge already exists on the jump host, with the same name as the value passed via `-B` or the default, use it instead of trying to create a virsh network. Change-Id: If7f0753ec366d429ff101c2b37dd68f895903404 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-25Merge "Fix or silence all yamllint warnings"Alexandru Avadanii1-1/+1
2017-09-25Run packages upgrade on openstack nodes onlyMichael Polenchuk1-2/+1
Change-Id: I53ac0be519df1bb39a6a56e236285fce95228bd4 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-09-25Fix or silence all yamllint warningsAlexandru Avadanii1-1/+1
Change-Id: Iface28ab770beee00374afb902ef4f9c983538f5 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-22Adjust memory allocation sizeMichael Polenchuk1-0/+2
* [baremetal] add memory to contollers & salt master * tune up sysctl vm.dirty* for compute nodes * upgrade packages to get the latest versions (https://bugs.launchpad.net/cinder/+bug/1641312) Change-Id: I9ad22206f2f3f11e1da3f93c7a0931c592adf1cf Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-09-13[virtual] Setup nfs for nova instancesMichael Polenchuk1-1/+2
Configure nfs shared storage for nova instances to support live migration feature. Change-Id: I777d5725e45d03325507d3ecc8dd59a28cbc188f Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-09-12Merge "salt-master-init.sh: Use stable 2016.11 salt ver"Alexandru Avadanii2-6/+4
2017-09-12reclass, states: Parametrize runtime configurationAlexandru Avadanii2-10/+10
mcpcontrol virsh network, as well as MaaS PXE network are installer specific, and not POD specific. Therefore, these should be easily parametrized without the PDF, using only installer inputs (e.g. env vars passed via Jenkins). - add new <all-mcp-ocata-common.opnfv.runtime> reclass class; - parametrize at runtime new reclass class based on global vars; - factor out MaaS deploy address / config using new mechanism; - parametrize at runtime virsh network definitions based on template; - add new "maas.pxe_route" sls for configuring routing on cfg01; - replace env vars with the new sls in "maas" state; NOTE: baremetal parametrization will be handled later. Change-Id: Ifd61143d818fb088b3f4395388ba769bbc49156e Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-12salt-master-init.sh: Use stable 2016.11 salt verAlexandru Avadanii2-6/+4
The <salt-master-init.sh> script default to using stable 2016.3 version of Salt via BOOTSTRAP_SALTSTACK_OPTS. However, since we manually add the 2016.11 repos beforehand, the correct version (2016.11) ends up being installed. This change merely drops one uselss repo from salt master apt sources. While at it, use HTTPS for repo.saltstack.com DEB repository. Since we already add the repo to apt sources via <user-data.sh>, pass "-r" on all architectures, skipping repo (re)addition. Change-Id: I7fbe0b646b2f2a8e952718a61ff8c180116f8f27 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-11salt master, maas: Move mcpcontrol to 10.20.0.0/24Alexandru Avadanii2-3/+4
Use INSTALLER_IP Jenkins param instead of SALT_MASTER_IP, allowing us to drop SALT_MASTER_IP completely from releng. mcpcontrol IP changes: - 192.168.10.100 becomes 10.20.0.2 (align with legacy Fuel master); - 192.168.10.3 becomes 10.20.0.3 (baremetal MaaS address); JIRA: FUEL-285 Change-Id: I6e2d44c3a8b43846196bd64191735214167a76ce Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-11reclass: Multiarch support for arch specific paramAlexandru Avadanii1-0/+2
Introduce a simple mechanism that simulates an 'if-arch-then' cond for reclass models: - add new <all-mcp-ocata-common> class hierarchy; - at runtime (via <salt.sh>) make 'all-mcp-ocata-common.arch' point to 'all-mcp-ocata-common.$(uname -i)' dynamically; - inherit new 'arch' class in all cluster models; - factor out current x86_64 default for "salt_control_xenial_image"; - add AArch64 default for param "salt_control_xenial_image"; Change-Id: I3b239b28d0fd1cc2ced8579e2e93b764eb71ffc3 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-11Merge "log.sh: Collect /var/log from cluster nodes"Michael Polenchuk2-2/+35
2017-09-09log.sh: Collect /var/log from cluster nodesAlexandru Avadanii2-2/+35
Bring back <ci/deploy.sh> parameter `-L`, which is already passed by releng deploy script. Since we need to enable file_recv on salt master, do so by adding a new class in <virtual-mcp-ocata-common/infra/config.yml>, which will be inherited by all scenario reclass models. JIRA: FUEL-288 Change-Id: I9b649315984fb8580b63883d0e06c1e1efd11057 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-08bash scripts: Fix remaining shellcheck warn/errsAlexandru Avadanii1-1/+1
Fix/silence all shellcheck errors, except for scripts in <prototypes/sfc_tacker>. Change-Id: Idc317cdba0f69b78299f2d3665e72ffc19dd8af5 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-08Merge "Move mcp.rsa to /var/lib/opnfv"Alexandru Avadanii1-1/+1
2017-09-08lib.sh: Remove all storage on undefineAlexandru Avadanii1-1/+4
While executing the deploy script via sudo will raise no issues with re-using old storage volume files, non-root deploys fail to change ownership of said files. Properly clean after ourselves when destroying and undefining VMs. JIRA: ARMBAND-311 Change-Id: I7fdd269dccbfd5ab5285d852ac7dacfc4dc5ccd7 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>