aboutsummaryrefslogtreecommitdiffstats
path: root/mcp/scripts/lib.sh
AgeCommit message (Collapse)AuthorFilesLines
2018-02-05[PDF] Switch to generate_config, unify templatesAlexandru Avadanii1-3/+53
- move bash template handling (previously expanded via `envsubst`) to lib.sh; - move j2 template handling to lib.sh; - move virsh network templates to 'mcp/scripts/virsh_net' subdir; - switch virsh network templates from `envsubst` expansion to j2 and leverage generate_config.py, similar to PDF Fuel installer adapter; - add relevant runtime env vars (e.g. SALT_MASTER, MAAS_IP) on the fly to PDF, to consume them in templates like params coming from PDF; - parameterize virsh network definitions based on PDF (mgmt, public); JIRA: FUEL-322 Change-Id: Ib94e78fc4f25797b9354a0552e884104da5d0003 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-02Merge "deploy.sh: Disable net.bridge.bridge-nf-call"Michael Polenchuk1-0/+9
2018-02-02Revert "[FN VMs] remove graphics"Alexandru Avadanii1-2/+10
RHEL family virtualization tools reserve 02:00 PCI slot for VGA, even if 'nographics' is specified when creating the VM (in case the user wants to later hook a video card, which usually *requires* PCI slot2). Debian systems do not follow this rule (tested with libvirt 1.x, 2.x, 3.x), hence 1st NIC lands on PCI slot 2 (and get eth name 'ens2'). To align the behavior across all possible jumpserver distros, bring back the virtio video. This reverts commit 738f6c3b68d1179de1ff790f9e72c25f10874da4. Change-Id: Ifd855c12e04aec1ff0ab047b13f8081365741889 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-01deploy.sh: Disable net.bridge.bridge-nf-callAlexandru Avadanii1-0/+9
JIRA: FUEL-334 Change-Id: I6d2499053dcfb7f99593fcd5c948b569bdcb9c9b Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-29[FN VMs] remove graphicsAlexandru Avadanii1-10/+2
Since VCP VMs (created via salt formula) do not have a video controller defined in their domain XMLs, network devices end on different PCI slots and hence have different names assigned (ens2+ vs foundation node VMs, which start with ens3). To align network interface names for VMs on jumpserver vs kvm nodes, and reduce confusion, remove the video controller from FN VMs. This allows some cleanup: - drop extra AArch64 args from virt-install; - unify 'opnfv_vcp_vm_*' and 'opnfv_fn_vm_*' variables; Change-Id: I0d108b00914b3eaaa03b67c652174f8ed4573118 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-25lib.sh: Reduce useless wget dot reportingAlexandru Avadanii1-1/+1
Downloading the base image (usually a few hundred MB) outputs a lot of useless dots to show progress. Switch to 1M per dot (from 1K). Change-Id: I8c525cad0b46e8ba3a7f6da4dd7f8277a49df91f Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-22Add support for different public network netmaskGuillermo Herrero1-0/+7
- Remove hardcoded /24 mask - Use PDF as source for public network, with reclass params: opnfv_net_public, _mask, _gw, _pool_start, _pool_end JIRA: FUEL-315 Change-Id: Idf3a4ed8f63f58fa90d9c1dcb7751ef3b1c9bd36 Signed-off-by: Guillermo Herrero <guillermo.herrero@enea.com>
2018-01-08lib.sh: base image: Prevent using incomplete imgAlexandru Avadanii1-4/+4
In case the previous deploy attempt already copied the base image as the VCP image in order to perform offline operations and failed, leaving an incomplete image in place, current code might try to use it instead of building it from scratch. Use the hash-agnostic link names as checkpoints for successful image handling. Change-Id: I1e99e515e18ba1dec534c520811c127b2b528afe Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-08lib.sh: Fix `modprobe -f` on UbuntuAlexandru Avadanii1-3/+4
For some reason, `modprobe -f` for a clean nbd module (from vanilla Ubuntu) fails with exec format error randomly, while a simple `modprobe` works. Change-Id: I79785e510cab757e2482baf442054be984c24019 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-07lib.sh: Fix stop condition in wait_forAlexandru Avadanii1-5/+5
Change-Id: Ida693b6dd328db283d6992ac33500f4dd1a73eb8 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-07lib.sh: Extend wait_for function to catch no respAlexandru Avadanii1-3/+11
wait_for function should be able to also check for minions that did not return or not respond, in addition to the return code. To keep it backwards compatible, condition the new check on the max attempt number being specified in decimal format (e.g. '10.0' unlike old '10'). Change-Id: If2512cf9121cdd795638efe7362ef0485d4e8d91 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-07user-data.template: Remove salt-minion installAlexandru Avadanii1-4/+1
salt-minion is now pre-provisioned inside the image using qemu-nbd. Revert "lib.sh: Limit envsubst to certain variables" This reverts commit 3a76d07dbd409b781abdb8520f55a1b20edf07db. Change-Id: Icceb8bcf439e28ab01c7731c3602c1113290454d Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-07lib.sh: Cache base image(s) between deploysAlexandru Avadanii1-13/+36
Fingerprint and re-use base image artifacts. Change-Id: Ic7a73c04e27d25addd50e4e9880619a0028956d3 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-04[centos jump] nbd build: Fix REQ_TYPE_SPECIALAlexandru Avadanii1-1/+5
Change-Id: Ia514418d2aae1b4f7e752d4610fa6c9829c67e51 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-04[centos jump] Add nbd build supportAlexandru Avadanii1-2/+70
RHEL distros do not maintain nbd, so add a best-effort function to build it on the fly. Change-Id: Ie0419f0fed8a0b12f6b878b3093d6ca34f72d140 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-12-27lib.sh: Add delay after `kpartx -av`Alexandru Avadanii1-0/+1
On rare occassions, mapper bindings created by kpartx take longer to show up, leading to errors when we try to mount them on. Bring back the hardcoded delay to bypass such issues. Change-Id: Ib386c04fc55cd85235a2156dba08fda378e4cdfd Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-12-18Merge "[baremetal] Move salt master IP to PXE/admin"Alexandru Avadanii1-4/+3
2017-12-18[baremetal] Move salt master IP to PXE/adminAlexandru Avadanii1-4/+3
Use PXE/admin network for salt traffic from/to all minions except cfg01, mas01. This allows us to drop the route to admin net from cfg01. Change-Id: Ic2526f1ff77afe5d92ced900971f4c8f78d2d8a2 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-12-15ci/deploy.sh: maas: cleanup_uefi on env eraseAlexandru Avadanii1-2/+3
Running `ci/deploy.sh -EE` should also perform an UEFI boot option cleanup, otherwise we risk booting the previously installed OS. While at it, reduce delay between nodes removal and fix a rare failure for `-EE` when no nodes are defined in MaaS. Change-Id: I789ffd3e22545921216f7d5ee3509c76354542eb Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-12-07Merge "base image: Explicitly call kpartx"Michael Polenchuk1-3/+7
2017-12-07Merge "lib.sh: persistent cfg01, mas01 mcpcontrol"Michael Polenchuk1-2/+2
2017-11-28base image: Explicitly call kpartxAlexandru Avadanii1-3/+7
qemu-nbd currently available in CentOS 7 does not add partition mappings automatically for NBD devices, so add explicit `kpartx` calls. Change-Id: Ifa79c89b82024602b782c449dbf4de10899e03b5 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-27lib.sh: persistent cfg01, mas01 mcpcontrolAlexandru Avadanii1-2/+2
cfg01, mas01 DHCP leases in mcpcontrol virtual network should be persistent (if cfg01 IP changes, minions can't find Salt Master). Change-Id: I497207ebe1537af94fd92de12491664d17ad3144 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-27AArch64: base image: pre-install salt-minionAlexandru Avadanii1-1/+7
While at it, rename apt repo in foundation node user-data template from "salt" to "saltstack", to align with reclass model naming. Change-Id: I5b216492349ae187b568884b1ab4046c52b1c6b2 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-27Add pre-{install,purge} support for base imageAlexandru Avadanii1-5/+132
Extend <lib.sh> and its invocation from <ci.deploy.sh> with support for modifying foundation node VMs base image prior to using it with: - additional APT GPG keys; - additional APT repos; - packages to pre-install; - packages to pre-remove; - (non-configurable) cloud init datasource via NoCloud only, so VCP VMs won't wait for metadata service; While at it, re-use the resulting image as a base for another round of pre-patching (same operations as above are supported) to provide a base image for VCP VMs. Add AArch64-specific configuration based on new mechanisms: - pre-install linux-image-generic-hwe-16.04-edge (and headers) for foundation node and VCP (common) image (also requires new repo and its key); - pre-install cloud-init for VCP image (it should already be installed, but script needs non-empty config for VCP to create the VCP image and transfer it over to Salt Master); NOTE: cloud-init is required on VCP VMs for DHCP on 1st iface. JIRA: FUEL-309 Change-Id: I7dcaf0ffd9c57009133c6d339496ec831ab14375 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-08lib.sh: Cleanup previous /boot/efi contentsAlexandru Avadanii1-1/+2
Some UEFI firmwares insist on scanning removable drives, even when boot entries were deleted from UEFI boot list (board flash). To work around this, remove contents of </boot/efi/*>, so scanning won't identify any valid EFI binaries. Another option would be erasing partition tables, but identifying the underlying disk(s) is more complicated, especially when using LVM/RAID etc. Change-Id: I9949b99b139b1642e3bd8f04de3bd5ef74d1ecc5 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-28lib.sh: rm Ubuntu boot entry on EFI systemsAlexandru Avadanii1-0/+10
On EFI-enabled systems, grub-install from grub-efi-* package installs a boot entry named "ubuntu". MaaS relies on IPMI to set boot order to PXE first; however on systems with buggy firmware or without full IPMI support, that fails, leading to booting Ubuntu from hard disk instead. Work around this by clearing any previous Ubuntu boot entry from board flash, before starting a new baremetal deploy. NOTE: This only runs against nodes that are online from a previous deploy. Closes: ARMBAND-47 Change-Id: I1c4ece09e42845ce2a1b7119ec69e46e5ca12376 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-27Merge "lib.sh: Fix compatibility with bash 4.2"Alexandru Avadanii1-16/+16
2017-10-27lib.sh: Fix compatibility with bash 4.2Alexandru Avadanii1-16/+16
JIRA: FUEL-296 Change-Id: Ide9f9333fe9b44ff6b78678064f8e67f05aabd42 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-26AArch64: Switch jumpserver VMs to virtio videoAlexandru Avadanii1-2/+2
Drop vgabios dependency by switching video from VGA to virtio for all VMs spawned on the jumpserver. NOTE: This requires virtualization packages on the jumpserver to be up to date (e.g. libvirt, QEMU). JIRA: ARMBAND-306 Change-Id: I73913e1ae8584f4e73b92994f78f7ec363cba3ec Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-20lib.sh: Fix locals override in nested invocationsAlexandru Avadanii1-11/+14
'wait_for' bash function is nested in another 'wait_for' call in some places, which leads to inner calls interfering with outer calls by overriding the locally scoped variables, including the 'attempt' internal counter. In some cases, the outer 'wait_for' would exit after a single attempt. Fix that by running all contents of `wait_for` inside a subshell, which inherits outer calls variables, but does not override them when the inner call is finished. Change-Id: I450eda3d023af2380c61ee930071fbfc393a5645 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-15states: Break on error, retry states up to 5 timesAlexandru Avadanii1-2/+3
While applying scenario states, break on error, and retry failed state up to 5 times. Apply the same behavior for `salt.sh`. Add new deploy parameter, '-D', backed up by 'CI_DEBUG' env var, which gates deploy sh scripts logging (set -x). Also extend '-f' deploy parameter, allowing it to be specified more than once; the first occurence will skip infra VM creation, but still sync reclass & other config from local repo, while a second occurence will also disable config sync. To prevent glusterfs client state from failing due to non-existent nova user/group, move it after nova:compute's nova state is applied. Change-Id: I234e126e16be0e133d878957bd88fed946955de8 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-14Add license headers where missingAlexandru Avadanii1-0/+7
While at it, compact 'set' into bash shebang where possible and add `make patches-copyright` target to simplify adding patch license headers. Change-Id: I0c841de72e5709e5eef915a52c5ec4a7fc0f7c37 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-14Drop infinite loops in favor of finite wait_forAlexandru Avadanii1-3/+2
While at it, fix some shellcheck warnings, and s/fgrep/grep -F/g. Change-Id: I093b7b4c196731b1ecc0c27a4111955b2e412762 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-14states: Split virtual_control_plane from maasAlexandru Avadanii1-10/+23
We should eventually also support baremetal deploys without a virtualized control plane (VCP), so decouple MaaS provisioning from VCP provisioning. While at it, move "wait_for" bash function from maas state to common library file, lib.sh. Change-Id: I32c33135655cb6aceae901a5f92b51265a8c84b4 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-03Identify jump host bridges based on IDF / PDF netsAlexandru Avadanii1-7/+1
- minor refactor of runtime templates parsing to allow var expansion; - parse <pod_config.yml> into shell vars, match dynamically networks from PDF to IP addresses on bridges of current jumphost; - keep old '-B' parameter in <ci/deploy.sh>, use it for providing fallback values in case there's no bridge name specified via IDF and no IP on the jumphost for one or more of the PDF networks; - re-enable dry-run to ease testing of the above; - add sample 'idf-pod1.yaml' to <mcp/config/labs/local>; The new behavior will try to determine the jump host bridge names: 1. Based on IDF mapping, if available 2. Based on PDF network matching with IP addrs on jumphost; 3. Fallback to values passed via '-B'; 4. Fallback to default values hardcoded in the deploy script; Later, we will drop MaaS network env vars in favor of PDF vars, once the PDF template is generating them. Change-Id: If9cd65d310c02965b2e2bfa06a0d7e0f97f1dd48 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-26lib.sh: Use host bridge in favor of virsh networkAlexandru Avadanii1-4/+5
If a linux bridge already exists on the jump host, with the same name as the value passed via `-B` or the default, use it instead of trying to create a virsh network. Change-Id: If7f0753ec366d429ff101c2b37dd68f895903404 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-25Fix or silence all yamllint warningsAlexandru Avadanii1-1/+1
Change-Id: Iface28ab770beee00374afb902ef4f9c983538f5 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-11salt master, maas: Move mcpcontrol to 10.20.0.0/24Alexandru Avadanii1-1/+2
Use INSTALLER_IP Jenkins param instead of SALT_MASTER_IP, allowing us to drop SALT_MASTER_IP completely from releng. mcpcontrol IP changes: - 192.168.10.100 becomes 10.20.0.2 (align with legacy Fuel master); - 192.168.10.3 becomes 10.20.0.3 (baremetal MaaS address); JIRA: FUEL-285 Change-Id: I6e2d44c3a8b43846196bd64191735214167a76ce Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-08bash scripts: Fix remaining shellcheck warn/errsAlexandru Avadanii1-1/+1
Fix/silence all shellcheck errors, except for scripts in <prototypes/sfc_tacker>. Change-Id: Idc317cdba0f69b78299f2d3665e72ffc19dd8af5 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-08Merge "Move mcp.rsa to /var/lib/opnfv"Alexandru Avadanii1-1/+1
2017-09-08lib.sh: Remove all storage on undefineAlexandru Avadanii1-1/+4
While executing the deploy script via sudo will raise no issues with re-using old storage volume files, non-root deploys fail to change ownership of said files. Properly clean after ourselves when destroying and undefining VMs. JIRA: ARMBAND-311 Change-Id: I7fdd269dccbfd5ab5285d852ac7dacfc4dc5ccd7 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-07Move mcp.rsa to /var/lib/opnfvAlexandru Avadanii1-1/+1
JIRA: FUEL-280 Change-Id: I1e07b0e1597b2a1e4a92e1274f89dda62cb81bb8 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-05ci/deploy.sh, lib.sh: Align SSH_KEY with JenkinsAlexandru Avadanii1-14/+21
Jenkins defines a build param also called SSH_KEY, which holds the full path to our MCP private RSA key, as opposed to Fuel@OPNFV, which uses the file basename as SSH_KEY. Switch to using full path in SSH_KEY, while also moving the key outside /tmp. NOTE: When running `ci/deploy.sh` with sudo, key will land in /root/opnfv/mcp.rsa unless SSH_KEY is set via env var. While at it, bring back VM image storage dir param, allowing us to drop one more reference to /tmp. JIRA: FUEL-280 Change-Id: Id1cfa6dbe7b2e6f3915b22281da5957333718401 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-04lib.sh: Limit envsubst to certain variablesAlexandru Avadanii1-1/+2
This will prevent envsubst from expanding the SALT_REPO local variable in user-data.template. Change-Id: Idcc3ea827293ac0d3b4059776f90bbbcf807daec Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-08-29lib.sh: Reuse /tmp/mcp.rsa if presentAlexandru Avadanii1-0/+5
Long-term, /tmp/mcp.rsa should be moved to a persistent location [1], and made configurable via env var / other mechanisms. This will allow us to: - use an existing keypair (provided by end-user in expected path); - login to previous deployment machines (e.g. to cleanup UEFI boot entries before destroying the cluster and rebuilding it); - split deploy in re-entrant stages (salt master only, cluster nodes only; similar to old Fuel, where we could reuse old Fuel VM); [1] https://jira.opnfv.org/browse/FUEL-280 Change-Id: I1e53321ed1cfc217ff95e809c867fa3370c479c9 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-08-29[virtual] Apply ceilometer/aodh servicesMichael Polenchuk1-1/+1
* run ceilometer/aodh states * wrap common virtual cluster options * get the source image based on timestamps Change-Id: I88f1d63ed4a94eba4ec0a9cf33d36d51c75ae355 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-08-17Merge "Bring in baremetal support"Alexandru Avadanii1-22/+28
2017-08-17lib.sh: Fix mcp key owner for OPNFV Jenkins jobsAlexandru Avadanii1-1/+4
In case of non-root deploys (i.e. running `ci/deploy.sh` without sudo), the OPNFV Jenkins slave executes it under a user shell, under a root shell, e.g.: $ sudo sudo -u jenkins bash $ echo $USER jenkins $ echo $SUDO_USER root Adjust lib.sh to only use SUDO_USER when it is not "root". Change-Id: I6dd1c8a5dd3a3b02cfb760818d66e99f49709a5b Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-08-17Bring in baremetal supportAlexandru Avadanii1-22/+28
- ci/deploy.sh: fail if default scenario file is missing; - start by copying reclass/classes/cluster/virtual-mcp-ocata-ovs as classes/cluster/baremetal-mcp-ocata-ovs; - add new state (maas) that will handle MaaS configuration; - Split PXE network in two for baremetal: * rename old "pxe" virtual network to "mcpcontrol", make it non-configurable and identical for baremetal/virtual deploys; * new "pxebr" bridge is dedicated for MaaS fabric network, which comes with its own DHCP, TFTP etc.; - Drop hardcoded PXE gateway & static IP for MaaS node, since "mcpcontrol" remains a NAT-ed virtual network, with its own DHCP; - Keep internet access available on first interfaces for cfg01/mas01; - Align MaaS IP addrs (all x.y.z.3), add public IP for easy debug via MaaS dashboard; - Add static IP in new network segment (192.168.11.3/24) on MaaS node's PXE interface; - Set MaaS PXE interface MTU 1500 (weird network errors with jumbo); - MaaS node: Add NAT iptables traffic forward from "mcpcontrol" to "pxebr" interfaces; - MaaS: Add harcoded lf-pod2 machine info (fixed identation in v6); - Switch our targeted scenario to HA; * scenario: s/os-nosdn-nofeature-noha/os-nosdn-nofeature-ha/ - maas region: Use mcp.rsa.pub from ~ubuntu/.ssh/authorized_keys; - add route for 192.168.11.0/24 via mas01 on cfg01; - fix race condition on kvm nodes network setup: * add "noifupdown" support in salt formula for linux.network; * keep primary eth/br-mgmt unconfigured till reboot; TODO: - Read all this info from PDF (Pod Descriptor File) later; - investigate leftover references to eno2, eth3; - add public network interfaces config, IPs; - improve wait conditions for MaaS commision/deploy; - report upstream breakage in system.single; Change-Id: Ie8dd584b140991d2bd992acdfe47f5644bf51409 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com> Signed-off-by: Guillermo Herrero <Guillermo.Herrero@enea.com> Signed-off-by: Charalampos Kominos <Charalampos.Kominos@enea.com> Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>