summaryrefslogtreecommitdiffstats
path: root/mcp/scripts/salt.sh
AgeCommit message (Collapse)AuthorFilesLines
2018-02-06Add NOVCP HA OVS scenario (baremetal, virtual)Alexandru Avadanii1-2/+2
Add a new class of scenarios, based on existing baremetal HA scenarios, but instead of having a virtualized control plane (VCP), all Openstack controller services will run directly on the cluster nodes. This change adds the common scaffolding, as well as the OVS scenario. The new scenario(s) can be used on full-baremetal clusters, soon on full-virtual clusters and later on hybrid (virt + bare) clusters. This change defines old (current) style scenario definitions for both baremetal and virtual, both named: - os-nosdn-nofeature-novcp-ha; Prerequisites: 1. Merge-able by name reclass.storage.node definitions Each cluster (e.g. database, telemetry) adds its own set of reclass storage node defitions, which for novcp scenarios should be merged into a single node (kvm) based on the 'name' property. This is not currently supported by upstream reclass 'node.sls' high state, so add support for it via an early patch (required before salt-master-init.sh tries to handle reclass.storage). 2. common reclass classes for novcp Some of the classes in `baremetal-...-common-ha` are not fit for novcp as they define VCP-specific config/inheritance, so add new versions of said classes with novcp in mind or adapt old classes: - parameterize ctl hostname in `openstack_compute.yml`; - new `openstack_control_novcp.yml`; - new `openstack_init_novcp.yml`; 3. Handle hard set names in state files for baremetal nodes Some of our state files (e.g. maas) hardcodes baremetal node names to 'kvm', 'cmp', so we need to align the names in novcp scenario with these values to re-use the maas state. As a future improvement we should parameterize these names in all state files. As a consequence, our baremetal controller nodes will also use 'kvm*' hostnames (instead of 'ctl*'). 4. Add 'noifupdown' to all interfaces on kvm nodes to prevent duplicate IPs/routes created at *any* ifup due to /etc/network/route-br-ex. Patch salt-formula-linux to skip network restart on 'noifupdown', also when routes are present on that interface. JIRA: FUEL-310 Change-Id: Ic67778f63e5ee0334dbfe9547c7109ec1a938d61 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-06[virtual] Add missing ODL dhcp_intAlexandru Avadanii1-0/+1
- add missing network definitions for ODL node's 1st interface; - add missing comments for `notify` global functions; - fix or silence shellcheck issues; JIRA: FUEL-322 Change-Id: Ie3341d29ab12ddf432db603ad865259afb54714e Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-06[apt-mk] Switch back to nightlyAlexandru Avadanii1-1/+1
Revert "salt: Use apt-mk 'stable' distribution" Revert "reclass: apt_mk_version: stable" This reverts commit d1b6119e288a31e015573363ce77790fec8684df. This reverts commit 4563ea7d62238e8273d840a8d9c6c1e179ca584e. Change-Id: I383db1f78a087045086096cbc674260b985fd913 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-05[FN VM] Reboot VMs on jump, wait for all onlineAlexandru Avadanii1-10/+17
- apply `linux` state on cfg01 first, so PXE/admin IP is added and FN VM minions are available; - add barrier and wait for all FN VMs to register with cfg01; - use batch-mode execution while applying `linux.network` on FN VMs; - retry all states executed via <salt.sh> on FN VMs; JIRA: FUEL-310 Change-Id: I72e1c565370072500df1d486fe76e6315f583c75 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-30[patch] Group APT calls for salt formula installAlexandru Avadanii1-2/+3
When installing salt-formula-* on cfg01, we used to call APT for each package. Instead, handle them all at once. While at it, stop using colored output on terminals that don't support it (e.g. 'vt220' used by OPNFV Jenkins). Change-Id: Ib8f2cee9638c43cdf648487bf05b07cd49802d3e Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-28[cfg01] apt.conf.d: Dpkg::Use-Pty: 0Alexandru Avadanii1-0/+1
This will silence all 'Reading database ...' noise during Salt formula installation. Change-Id: I734d727194e276443db1e1581f40ec494562196e Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-21reclass: apt_mk_version: stableAlexandru Avadanii1-1/+1
Although previous commit d1b6119 changed the first reference of apt-mk repos to 'stable' from 'nightly', it missed the cluster model. This fixes redeploys with `-f`, which fail due to conflicts between already installed 'stable' packages and 'nightly' ones. Fixes: d1b6119 Change-Id: I854bac86feaaa61da0b68d158e270eec1ee0ccb7 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-07salt.sh: Fix linux state apply on cfg01Alexandru Avadanii1-4/+4
cfg01 does not repond or is not connected while trying to apply linux state via salt.sh, use wait_for macro to account for this. JIRA: ARMBAND-315 Change-Id: I2d4c63d43f806b65f9ce327f4a00a6334be88750 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-12-30salt: Use apt-mk 'stable' distributionAlexandru Avadanii1-1/+1
'nightly' repo dist from apt-mk is broken, so switch to 'stable'. Change-Id: Ie12dfc2a499910b8b98a63886ba16044e66435f5 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-12-18salt.sh: Use salt-call to apply linux sls on cfg01Alexandru Avadanii1-1/+1
Also, retry applying linux high state up to twice, due to rare spurious failures with 'No reponse' status. Change-Id: Ic7839a5c9501673cb127412136afb91e05f87a7e Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-12-01salt.sh: Force ocata link creationAlexandru Avadanii1-1/+1
Required for re-deploying with `ci/deploy.sh -f`. Change-Id: I8e6d95f17f57b9dab3257fb4c8ccd1f962dacb45 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-28[virtual] Update openstack version to pikeMichael Polenchuk1-1/+5
Change-Id: I1df0228cb44bf9122aaf93dd25fc16a0d26a5240 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-11-27Add pre-{install,purge} support for base imageAlexandru Avadanii1-0/+12
Extend <lib.sh> and its invocation from <ci.deploy.sh> with support for modifying foundation node VMs base image prior to using it with: - additional APT GPG keys; - additional APT repos; - packages to pre-install; - packages to pre-remove; - (non-configurable) cloud init datasource via NoCloud only, so VCP VMs won't wait for metadata service; While at it, re-use the resulting image as a base for another round of pre-patching (same operations as above are supported) to provide a base image for VCP VMs. Add AArch64-specific configuration based on new mechanisms: - pre-install linux-image-generic-hwe-16.04-edge (and headers) for foundation node and VCP (common) image (also requires new repo and its key); - pre-install cloud-init for VCP image (it should already be installed, but script needs non-empty config for VCP to create the VCP image and transfer it over to Salt Master); NOTE: cloud-init is required on VCP VMs for DHCP on 1st iface. JIRA: FUEL-309 Change-Id: I7dcaf0ffd9c57009133c6d339496ec831ab14375 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-23salt.sh: Rework `git -C` for older git compatAlexandru Avadanii1-1/+1
CentOS 7.4 provides git 1.8.3, while `git -C` support was introduced in 1.8.5, so rework <salt.sh> to not rely on this parameter. Change-Id: Ied88466b2b3d91a3317922fa08f60afe934358a0 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-21ci/deploy.sh: Add new `-E` arg for env eraseAlexandru Avadanii1-12/+38
NOTE: In order to undefine VCP VMs with NVRAM (e.g. AArch64 VMs using AAVMF), an additional parameter should be passed to libvirt by Salt virt core module (equivalent to `virsh undefine --nvram`). While at it, pass CI_DEBUG, ERASE_ENV enviroment variables to state execution, and stop force-applying patches. Also refactor the rsync between foundation node and Salt master, so the whole git repo is copied as </root/opnfv>, and <root/fuel> becomes a link to it; useful for Armband, where 'fuel' is a git submodule. Fix .git paths after rsync, so git submodules work as expected in cfg01 repos. JIRA: FUEL-307 Change-Id: Ic62f03e786581c019168c50ccc50107238021d7f Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-12salt formulas: Pin via reclass instead of salt.shAlexandru Avadanii1-8/+0
Previous commit pinned the salt-formula-package by installing a specific version after OPNFV-specific patches were applied by `patch.sh`. To prevent patch bypass, move the pin to reclass model. While at it, also move salt-formula-neutron pinning to reclass to bypass wrong minimum length of 2 for version-pinned formula pkgs. Change-Id: I35c2773c130d79ec919685cd771311d6c11e11a8 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-11salt.sh: Pin MaaS formula to last known stable verAlexandru Avadanii1-0/+3
Recent upstream changes broke MaaS machine status parsing. Pin the formula to the previous version until upstream provides a fixed package or a repo snapshot baseline. Change-Id: I26ebd60f40812ae47da0ca28d19c9fdcac7920da Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-07Revert "Freeze the galera formula to avoid update"Michael Polenchuk1-4/+0
This reverts commit f49340e1aac3f5e8bb009a9d0d35aa6fedba7384. Version has been set back to 5.6 in upstream. Change-Id: I06d6865e59cfbc5055322eb31cbb237075ebdcb1 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-10-31Freeze the galera formula to avoid updateMichael Polenchuk1-0/+4
Change-Id: Ibf03a7dc2ed58a749476ae0bdc6cad2bba8783fb Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-10-18Horizon: service: Default to v2 APIAlexandru Avadanii1-1/+1
Horizon service definition hardcodes the API version to v3, and also overrides the value in our reclass model during interpolation. Adjust the default to '2'. While at it, move 'reclass' patching later in `salt.sh`, after the salt formula packages are installed. JIRA: FUEL-284 Change-Id: I6f63b4d00ae85c82b076b3efef857e7cc4f3fd35 Signed-off-by: Ting Wu <ting.wu@enea.com> Signed-off-by: Junaid Ali <junaidali.yahya@gmail.com> Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-16Merge "salt.sh: DEBIAN_FRONTEND=noninteractive"Alexandru Avadanii1-0/+1
2017-10-16[ovs/dpdk] Return true on linux.network stateMichael Polenchuk1-1/+1
linux.network state is expected to be down until node reboot to activate required hugepages therefore return true in any case. Change-Id: I61deac12b9ac212c5f79cdcce30cda8e5d24413b Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-10-16salt.sh: DEBIAN_FRONTEND=noninteractiveAlexandru Avadanii1-0/+1
Change-Id: Ibb1e78563043298dffdb4b7e9091d3f582e189fc Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-15states: Break on error, retry states up to 5 timesAlexandru Avadanii1-5/+7
While applying scenario states, break on error, and retry failed state up to 5 times. Apply the same behavior for `salt.sh`. Add new deploy parameter, '-D', backed up by 'CI_DEBUG' env var, which gates deploy sh scripts logging (set -x). Also extend '-f' deploy parameter, allowing it to be specified more than once; the first occurence will skip infra VM creation, but still sync reclass & other config from local repo, while a second occurence will also disable config sync. To prevent glusterfs client state from failing due to non-existent nova user/group, move it after nova:compute's nova state is applied. Change-Id: I234e126e16be0e133d878957bd88fed946955de8 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-14Drop infinite loops in favor of finite wait_forAlexandru Avadanii1-1/+1
While at it, fix some shellcheck warnings, and s/fgrep/grep -F/g. Change-Id: I093b7b4c196731b1ecc0c27a4111955b2e412762 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-13patch.sh: Exit on patch apply failureAlexandru Avadanii1-1/+2
Patches that fail to apply are silently skipped. Since retrying is not possible, deploy moves on, making the problem hard to observe. Change-Id: I3fe01cef6b245de53767a660b9b4fdfae0d5d0ff Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-03Apply neutron formula patch after formula updateMichael Polenchuk1-0/+1
Change-Id: I8dba3676adc8cc49731d91db7cc028a9c5b1627d Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-10-03salt.sh: Stick to older salt-formula-neutronAlexandru Avadanii1-0/+4
Recent changes to salt-formula-neutron, more precisely [1], broke the compatibility with our current reclass model [2], which was not updated to reflect the formula change. Instead of locally patching the reclass model, install the latest compatible version of the Neutron formula: - 2016.12.1+201709251458.f0607d9-1xenial1; [1] https://github.com/salt-formulas/salt-formula-neutron/commit/ 79ffa26858d682f404984175dc1ab93863a149ad [2] https://github.com/Mirantis/reclass-system-salt-model Change-Id: Ia1bc9de9a1543f7888688be76ff3513e6b4f062b Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-27PDF parsing supportAlexandru Avadanii1-5/+7
- add new git submodule pointing to OPNFV Pharos; - use Pharos 'generate_config.sh' to parse the PDF using the Fuel installer adapter and generate <pod_config.yml> dynamically; - build <pod_config.yml> outside current git repo and sync separately to prevent sensitive data leak; - add <pod1.yaml> PDF sample based on LF-POD2, should be used with 'ci/deploy.sh -l local -p pod1'; Change-Id: I4e1b95f180bcd5ade5d86f516628eb8edbe64b1c Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-25Run packages upgrade on openstack nodes onlyMichael Polenchuk1-2/+1
Change-Id: I53ac0be519df1bb39a6a56e236285fce95228bd4 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-09-22Adjust memory allocation sizeMichael Polenchuk1-0/+2
* [baremetal] add memory to contollers & salt master * tune up sysctl vm.dirty* for compute nodes * upgrade packages to get the latest versions (https://bugs.launchpad.net/cinder/+bug/1641312) Change-Id: I9ad22206f2f3f11e1da3f93c7a0931c592adf1cf Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-09-13[virtual] Setup nfs for nova instancesMichael Polenchuk1-1/+2
Configure nfs shared storage for nova instances to support live migration feature. Change-Id: I777d5725e45d03325507d3ecc8dd59a28cbc188f Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-09-12salt-master-init.sh: Use stable 2016.11 salt verAlexandru Avadanii1-5/+3
The <salt-master-init.sh> script default to using stable 2016.3 version of Salt via BOOTSTRAP_SALTSTACK_OPTS. However, since we manually add the 2016.11 repos beforehand, the correct version (2016.11) ends up being installed. This change merely drops one uselss repo from salt master apt sources. While at it, use HTTPS for repo.saltstack.com DEB repository. Since we already add the repo to apt sources via <user-data.sh>, pass "-r" on all architectures, skipping repo (re)addition. Change-Id: I7fbe0b646b2f2a8e952718a61ff8c180116f8f27 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-11reclass: Multiarch support for arch specific paramAlexandru Avadanii1-0/+2
Introduce a simple mechanism that simulates an 'if-arch-then' cond for reclass models: - add new <all-mcp-ocata-common> class hierarchy; - at runtime (via <salt.sh>) make 'all-mcp-ocata-common.arch' point to 'all-mcp-ocata-common.$(uname -i)' dynamically; - inherit new 'arch' class in all cluster models; - factor out current x86_64 default for "salt_control_xenial_image"; - add AArch64 default for param "salt_control_xenial_image"; Change-Id: I3b239b28d0fd1cc2ced8579e2e93b764eb71ffc3 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-09log.sh: Collect /var/log from cluster nodesAlexandru Avadanii1-2/+0
Bring back <ci/deploy.sh> parameter `-L`, which is already passed by releng deploy script. Since we need to enable file_recv on salt master, do so by adding a new class in <virtual-mcp-ocata-common/infra/config.yml>, which will be inherited by all scenario reclass models. JIRA: FUEL-288 Change-Id: I9b649315984fb8580b63883d0e06c1e1efd11057 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-05Merge "salt.sh: Execute deepclean before importing patches"Michael Polenchuk1-1/+1
2017-09-04salt.sh: Execute deepclean before importing patchesJunaid Ali1-1/+1
Deepcleaning can be useful when deploying from an existing local repo Change-Id: I3908d6533c744da7cc863a250f4158c6c0e8e169 Signed-off-by: Junaid Ali <junaidali.yahya@gmail.com>
2017-09-01salt.sh, user-data: Add Saltstack arm64 repoAlexandru Avadanii1-0/+4
These changes will only be applied on AArch64, based on `uname -m`. While at it, add arm64 suppport to salt-master-setup.sh. Upstream commit [1] broke Salt bootstrap on AArch64, by introducing an architecture condition that is too strict to allow Debian package installation (even if we provide our own repo). Add "arm64" to the list of supported architectures. This needs to be done on the fly, as the bootstrap script is fetched using `curl` from <salt-master-setup.sh>. [1] https://github.com/saltstack/salt-bootstrap/commit/caa6d7d Change-Id: Id706a74a52ffe2f8b8c9dd3b9f70c78f35b2f745 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-08-31mcp/deploy/scripts: Move to git submoduleAlexandru Avadanii1-4/+2
Change-Id: I65d1f5680000011493bde17a249a87738ebfdd96 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-08-30Update salt init scripts sourceMichael Polenchuk1-3/+2
Change-Id: I269e397b78d55794b1c49bf582cc0e663cbe9ca6 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-08-11Revert "Switch distrib revision to testing"Michael Polenchuk1-1/+1
There is no much difference between nightly vs testing repos. This reverts commit 052a1626da71d232431b3a9faf576a76db61f10e. Change-Id: I40f60ddc2c9e1e59c7c4eb0be59ae19b00b7fb9b Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-08-09Synchronize down all of the dynamic modulesMichael Polenchuk1-0/+1
Sync all of the salt dynamic modules to make sure custom modules, states, grains & etc. are on the target nodes. Change-Id: I04f2ac0465d5381c4c07623e9383619d9b9731de Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-08-08Switch distrib revision to testingMichael Polenchuk1-1/+1
In order to avoid bleeding edge issues with nightly version of salt formulas, switch to "safe" testing repo source. Change-Id: I9d12f337c3952711cb8843943c23315795064016 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-08-01salt.sh: Drop upstream clone in favor of local gitAlexandru Avadanii1-10/+34
salt.sh currently clones the full Fuel@OPNFV git repo from upstream public mirror, preventing us from testing locally edited or new patches. Instead, bring back git submodule handling from old f_repos, clone and patch each submodule locally, then copy the whole parent repo over to cfg01. This is also a first step towards implementing offline deploy support. NOTE: This adds new deploy prerequisite packages: - git (for submodule clone/update); - make (for submodule patching); - rsync (for parent repo replication to cfg01); NOTE: Parent repository is expected to be a git repo, in order to work with git submodules. While at it, perform some minor related changes: - add deploy artifacts (ISOs, qcow2 files) to .gitignore, also used to filter-out such files during rsync to cfg01; - remove obsolete Fuel patches (old f_repos mechanism); - rename "reclass-system-salt-model" submodule; Change-Id: I6210d80d41010b2802e4f1b31acf249a18db7963 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-07-31mcp/scripts: shellcheck cleanupAlexandru Avadanii1-1/+2
Change-Id: I280d540b461341e14de654186016248eba4d3521 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-07-12Apply reclass patches before salt master initMichael Polenchuk1-8/+7
Change-Id: I7bb984880e98b7bdec9aa0b3895a3be9fd75cac0 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-07-11Apply upstream patchesMichael Polenchuk1-0/+2
For opendaylight support: * neutron formula * reclass system model Plus missing apply of haproxy state. Change-Id: Ic9e3672b51d5331656d636c44c3f891e49437e23 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-07-05Turn on opendaylight scenarioMichael Polenchuk1-0/+15
* fix formula & reclass cluster model * bring in running states Change-Id: I8e66e69045f5c745f9aa6f59f7ce6d66b5bf1c95 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-07-03Rewrite infra deployment scriptsMichael Polenchuk1-0/+23
* bring in scenario files * shift infra code into functions Change-Id: I650a26d03d842c3afcc7fcb97b84ef4826827a38 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>