summaryrefslogtreecommitdiffstats
path: root/mcp/scripts
AgeCommit message (Collapse)AuthorFilesLines
2019-07-29[deploy] Explicitly set NS for resolvconf in VMsAlexandru Avadanii2-3/+5
With newer Ubuntu distros using netplan and systemd-resolve, we can't rely on /etc/resolv.conf found on the Jumphost being usable inside the guest VMs, so explicitly use the public network DNS servers configured in PDF/IDF. This will enable support for Jumpserver operating systems like Ubuntu 18.04. Change-Id: I0c7e02d5c1b822f809ce818e739c19d0344f39f5 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-07-22[iec] centos: Preinstall git into cloud imageAlexandru Avadanii1-2/+3
While at it, fix CentOS selinux preconfiguration on x86_64, which was previously limited (incorrectly) to AArch64. Change-Id: I2d6604d3eea2bfc11fdd5dd3aeb4e2c0c3ede4a2 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-07-10[virtual] Update OpenStack version to SteinMichael Polenchuk1-4/+4
Change-Id: I9c1e97144ffd46040d32a0edf8253fc393b73c89 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2019-07-01[lib] Limit cloud img partition resize to XenialAlexandru Avadanii1-1/+2
All cloud images except Ubuntu Xenial (CentOS 7, Ubuntu 18.04) already have enough free space on the predefined partitions, so skip the resize to avoid dealing with the newer e2fsprogs required by Ubuntu 18.04. Change-Id: I184590e631c76910e7c3169dc7bee3c5902ebaf1 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-06-29[virtual] Add Ubuntu 18.04 (Bionic) basic supportAlexandru Avadanii2-0/+23
Support Ubuntu 18.04 for virtual deployments (and implicitly for VCP VMs). Note that MaaS-provisioned systems will require the same changes being applied via curtin templates. Change-Id: I7cbd7e7c4421f6b970ce6ef97c10d269fec5fca3 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-06-28[iec] Add basic CentOS support (virtual only)Alexandru Avadanii6-27/+67
- reclass: iec: CentOS compatibility changes: * drop `proto: static` in favor of letting the linux formula set the appropiate default based on target OS; * replace `proto: manual` with `proto: none` on RHEL systems; * system.file: Avoid using non-existing `shadow` group for system files; * load br_netfilter kernel module to avoid `linux.network` state failures; * disable `at`, `cron` due to incomplete defaults in salt-formula-linux (since we don't use them on iec nodes anyway); - jumpserver/VCP VMs: centos: enable predictable interface names: * CentOS cloud image defaults to old 'eth' naming scheme; * add necessary kernel boot options via linux state; * cleanup auto-generated udev rules for old eth interface names; - salt-formula-linux: network: RHEL: Set bridge for member interfaces * Find the bridge containing the interface being currently configured (if any) and pass it to the `network.managed` Salt call; - deploy.sh: Add new deploy argument `-o` for specifying the operating system to preinstall on jumpserver and/or VCP VMs; * defaults to 'ubuntu1604'; * only iec scenarios will also support 'centos' for now; - user-data: minor tweaks for CentOS compatability: * use `systemctl` instead of `service` utility; * explicitly enable `salt-minion` service, since it defaults to disabled on RHEL systems; * explicitly call `ldconfig` to work around stale cache on RHEL, preventing `salt-minion` from using OpenSSL library; - states: virtual_init: Skip non-existing sysctl options on CentOS: * CentOS currently uses a 3.x kernel which lacks certain sysctl options that were only introduced in 4.x kernels, so skip them; - state: akraino_iec: Add centos support: * move iec repo to `/var/lib/akraino/iec` on both Salt Master and cluster nodes; - scenario defaults: Add CentOS configuration: * OS-dependent configuration split; * CentOS base image, default packages etc.; - AArch64 deploy requirements: Add `xz` dependency * CentOS AArch64 cloud image is archived using xz, install xz tools for decompression; - xdf_data: Make yaml parsing OS agnostic: * rename `apt` to `repo` where appropiate; * OS-dependent configuration parsing; - lib_jump_deploy: CentOS handling changes: * skip filesystem resize of cloud image for CentOS; * add repo handling, package intallation/removal handling for CentOS; * unxz base image if necessary (CentOS AArch64 cloud image); Change-Id: Ic3538bacd53198701ff4ef77db62218eabc662e7 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-05-16[lib] Add uninstall/cleanup optionAlexandru Avadanii1-0/+21
When multiple installers are used on the same jumpserver, it is useful to have the ability of automatic cleanup after a previous deploy. Change-Id: Ib3249f53ee9d6b1ba2409dd71bd13480536faedc Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-05-10[maas] Fix permissions on (partial) redeployAlexandru Avadanii2-4/+6
When redeploying a cluster only (keeping the infrastructure containers from a previous deploy), some things need to be adjusted: - /entrypoint.sh exec permission; - /etc/maas uid/gid re-align on new (fresh) deploy; - account for different location of /usr/sbin/tcpdump apparmor profile for CentOS jumpservers; Change-Id: If51db0bc95eff1a497e1df5d457e26a7b902aa5a Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-05-09[fdio] Bump compute RAM defaults for virtual PODsAlexandru Avadanii1-5/+5
Hugepage count has been recently bumped for virtual PODs via IDF changes in Pharos, so align our FDio scenarios with the new RAM requirements. While at it, fix wrong pod_config template evaluation by moving it after the templated scenario files are expanded, since pod_config relies on scenario node definition. Also, configure VPP to use decimal interface names by default to align with Pharos macro for the VPP interface name string. Change-Id: Ib3a89c294a3a2755567fdbe07e3be2b8ca1a5714 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-05-07Merge "[dpdk] Get back to shared memory model"Michael Polenchuk1-0/+0
2019-05-06Merge "[virtual] Parameterize scenarios based on PDF/IDF"Alexandru Avadanii3-29/+28
2019-05-06[dpdk] Get back to shared memory modelMichael Polenchuk1-0/+0
The per port model potentially requires an increase in memory resource requirements (which is limited by labs) to support the same number of ports and configuration as the shared port model. Set linux:network:openvswitch:per_port_memory explicitly to true to enable per port mempools support for DPDK devices. Change-Id: I130885afc50e7a047f8835113d370840827ad718 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2019-04-18mcpcontrol: Avoid duplicate ip rulesAlexandru Avadanii1-1/+2
Executing deploy.sh multiple times led to duplicating the ip rules. Change-Id: Iad5886a851970f166996226fa3d115a93113c6db Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-04-15mcpcontrol: policy based routing for INSTALLER_IPAlexandru Avadanii1-1/+2
To bypass Docker 'bridge'-backed network isolation, we previously added an extra routing hop, which broke access from inside the 'mcpcontrol' Docker network (typically 10.20.0.0/24) to its bridge address (10.20.0.1), leading to DNS issues on Salt Master. This change leverages policy based routing to only add the extra routing hop for connections originating from the default Docker bridge network ('docker0'). Note that other Docker networks using the 'bridge' driver are still isolated from 'mcpcontrol'. Fixes: d9b44acb Change-Id: Ib92901c3278ae9b815f28f26d4c26f82bcadacd6 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-04-11route mcpcontrol via PXE br to bypass isolationAlexandru Avadanii1-1/+2
Recent virsh/Docker network rework changed mcpcontrol (previously a virsh-managed network) into a Docker-controlled network using the 'bridge' driver. As a consequence, Docker now isolates traffic from 'mcpcontrol' network from the default Docker bridge network ('docker0') using iptables rules that check input/output interfaces. Yardstick (and any other Docker container hooked via 'docker0') will not be able to ssh into Salt master due to this isolation. One possible workaround would be to explicitly ACCEPT traffic from 'docker0' going to Salt master. However, this is only properly supported starting with Docker 17.06, while most CI hosts and end users are still using 17.05 or older. In older Docker releases, DOCKER-USER iptables table was not avaiable, so injecting custom iptables and making them persistent is not only complicated, it's also prone to subtle errors. Another way to bypass the iptables rules is to route the packets coming from our new Docker network via another bridge before letting them find their way into 'docker0'. This change adds a new route for the Salt master host (note that MaaS container will not benefit from this) via the PXE bridge on the jumphost (which can be either a real Linux bridge for baremetal deployments or a virsh-managed network); adding one extra network hop for each packet going between our 'mcpcontrol' Docker network and 'docker0', effectively bypassing the Docker-enforced iptables DROP. Change-Id: Id8ac7a638c778887b361c9b64c320664c88f59fd Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-04-08[virtual] Parameterize scenarios based on PDF/IDFAlexandru Avadanii3-29/+28
NOTE: only os-nosdn-nofeature-noha is parameterized for now. - move config drive & disk creation from prepare_vms to create_vms; - make default disk size(s) configurable based on scenario defaults and vPDF; * compute nodes require 2 disks to be defined in vPDF, since the pillar reclass model assumes /dev/vdb is reserved for cinder; * if multiple disks are defined in vPDF, they are created and attached accordinly (only ctl01 and cmp nodes are parameterized in this change; only for the os-nosdn-nofeature-noha scenario); - vCPU specifications are deduced based on vPDF (sockets, cores); * threads/core is hard set to 2 since vPDF does not have a key for it; * NUMA resources are distributed evenly based on the number of sockets configured in PDF; * no less than the mininum requirement for a scenario is allocated (e.g. if PDF specifies 2 cores, but the scenario requires at least 4 cores, the larger value will be used); - RAM is deduced based on PDF (but no less than the mininum req is allocated, e.g. if PDF specifies 2GB RAM for computes, but the scenario requires at least 8GB, the larger value will be used); Change-Id: I97188aa2a1006865b8429eb6483e10c76795f7d2 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-04-02[dpdk] Enable per port memory modelMichael Polenchuk1-0/+0
The per port memory model provides a more transparent memory usage model and avoids pool exhaustion due to competing memory requirements for interfaces. (http://docs.openvswitch.org/en/latest/topics/dpdk/memory/) Change-Id: I5add0f49cdcdf2fc3d24affee10a275abe3ca46a Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2019-03-29[akraino] Add IEC K8-calico scenariosAlexandru Avadanii3-4/+6
- bump Pharos git submodule to allow PODs with fewer nodes; - add `k8-calico-iec-noha` scenario definition for Akraino IEC basic configuration; - add `k8-calico-iec-vcp-noha` scenario definition for Akraino IEC nested (virtualized control plane) configuration; - add `akraino_iec` state, which will leverage the Akraino IEC bootstrap scripts from [1]; - replace system.reboot salt call with cmd.run 'reboot' as it's more reliable; - use kernel 4.15 for AArch64 K8 IEC scenarios; NOTE: These scenarios will not be released in OPNFV since don't rely on Salt formulas but instead of Akraino IEC scripts to install K8s. [1] https://gerrit.akraino.org/r/#/q/project:iec Change-Id: I4e538e0563d724cd3fd5c4d462ddc22d0c739402 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-03-18[lib] nbd: Explicitly map partitionsAlexandru Avadanii1-1/+5
Certain kernels (e.g. 4.4.0-101+ in Ubuntu) no longer automatically ack the partition table update after `kpartx -a /dev/nbdX`, see [1]. To avoid another dependency on `parted` packages, use `partx` from `util-linux`, which is already installed as a dependency of e2fsprogs. [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1743026 Change-Id: Ibd993fe210c1a11814e89a66759568d4d117d613 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-03-06[lib] Create veths using systemd opnfv-fuel unitsAlexandru Avadanii2-9/+43
Create 2 systemd services on the jumphost that will handle veth pairs creation, respectively adding them to virsh/real bridges. This allows us to set docker containers restart policy to 'always', enabling persistent Salt Master/MaaS containers across jumphost reboots. NOTE: libvirt creates virtual networks async, hence the need for retrying hooking veths to them. JIRA: FUEL-406 Change-Id: I1ca033cb5eb854b577b57bb2387a58bd9605a5bb Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-02-26Merge "[cfg01] Reduce mine_interval to 15 min"Michael Polenchuk1-1/+1
2019-02-22[lib] Add fatal validation of old kernel on UbuntuAlexandru Avadanii1-0/+8
As reported in [1], kernel 4.4 seems to break nested virtualization, add a fatal check against it. [1] https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1797332 Change-Id: I0aef8a7340dd82bfeb2e58c9642623b9ec13dca5 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-02-22[cfg01] Reduce mine_interval to 15 minAlexandru Avadanii1-1/+1
Some PODs are fast enough to get past installing, syncing and using MaaS to provision the OS on the baremetal nodes before the 1h mine refresh. Since mine.update operation is fast enough to go unnoticed and we only collect IP addresses, grains and pem entries, schedule it every 15 minutes. Due to reclass class inheritance, we can't easily override this via pillar data, so handle it via entrypoint.sh. Change-Id: I0d8ed2da838ad09c94e9327d0131d3e239de4f08 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-02-19Merge "[baremetal] Containerize MaaS"Alexandru Avadanii9-88/+145
2019-02-14[baremetal] Containerize MaaSAlexandru Avadanii9-88/+145
- replace mas01 VM with a Docker container; - drop `mcpcontrol` virsh-managed network, including special handling previously required for it across all scripts; - drop infrastructure VMs handling from scripts, the only VMs we still handle are cluster VMs for virtual and/or hybrid deployments; - drop SSH server from mas01; - stop running linux state on mas01, as all prerequisites are properly handled durin Docker build or via entrypoint.sh - for completeness, we still keep pillar data in sync with the actual contents of mas01 configuration, so running the state manually would still work; - make port 5240 available on the jumpserver for MaaS dashboard access; - docs: update diagrams and text to reflect the new changes; Change-Id: I6d9424995e9a90c530fd7577edf401d552bab929 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-02-14Rise up salt's gather job timeoutMichael Polenchuk1-1/+2
While the minions are working their jobs the CLI is waiting for the first initial timeout period (timeout) to start. When that hits, the CLI sends sends the first "find_job" query. This kicks off the gather_job_timeout timer. Sometimes a minion doesn't respond to the request within the gather_job_timeout time period (default is 10s), so rise up this value to give a chance for a minion to report actual status. Change-Id: Ic3756b82fdeb17718870ab30e9578263d25309f7 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2019-02-05Merge "[cfg01] Use ssh config to set default user & key"Michael Polenchuk1-0/+3
2019-02-04[cfg01] Use ssh config to set default user & keyAlexandru Avadanii1-0/+3
Change-Id: I7486569568207f7652f8bdfcf1060ce51a9dbb0e Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-02-04[submodule] Bump Pharos for arm-pod10 cmp changeAlexandru Avadanii1-0/+0
Change-Id: Ia7f8845017333e54db110bca5b3715702948b76b Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-01-29[deploy] Allow only operating system installAlexandru Avadanii1-1/+2
Extend one of the existing deployment arguments to allow the installation of only the operating system and infrastructure networks, skipping cloud setup. Change-Id: Ibc5d0f324ed15b66f809839cfce49a0324b6fe4d Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-01-19[uefi_cleanup] Use grain targetingAlexandru Avadanii1-8/+3
Alternating HA and no-HA scenario deployments on baremetal requires non-hostname targeting for UEFI cleanup (e.g. ctl01/gtw01/kvm01). Change-Id: I9f0e967b500856b65a69ea0ab6ea13e15b327d8b Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-01-17[submodule] Bump Pharos for arm-pod10 cmp NIC syncAlexandru Avadanii1-0/+0
Change-Id: I177598d4d20539e50aab5f283e8d10022a4f1a14 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-01-16[submodule] Bump Pharos for arm-pod10 NIC reorderAlexandru Avadanii1-0/+0
Change-Id: I79d3167432d48500346d5c8294d447c54e0cb6be Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-01-16Make shutdown only on physical nodesMichael Polenchuk2-2/+6
Change-Id: If167e7a6bdcdccd6b6df43bd5cac54250abec61a Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2019-01-15Merge "[baremetal] Shutdown nodes from previous deploy"Alexandru Avadanii1-0/+1
2019-01-14Merge "[centos] Update altarch kernel URL"Michael Polenchuk1-6/+2
2019-01-13[baremetal] Shutdown nodes from previous deployAlexandru Avadanii1-0/+1
When noha scenarios are scheduled on the same CI POD currently running a previously deployed HA scenario, one baremetal node might remain unused (kvm03), connect to the new Salt master and interfere with the deployment. To prevent that, shutdown all baremetal nodes at the begining of the deployment. Change-Id: Ia9bad8b5d8348433cefac9aa76eca0de664f187d Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-01-13[centos] Update altarch kernel URLAlexandru Avadanii1-6/+2
CentOS recently moved its kernel source RPM from the altarch subdir to the same directory x86_64 kernel sources used to reside, so update our script accordinly. Change-Id: I88010eabdfc15d6a79350dface29258cc37c4b95 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2019-01-03[MaaS] Implement aarch64 tags for kernel_optsAlexandru Avadanii1-0/+0
On AArch64, 1G hugepages need to be enabled via kernel cmdline before mounting hugetlbfs [1]. Leverage MaaS tags to apply custom kernel args to AArch64 nodes. [1] https://wiki.debian.org/Hugepages Change-Id: Ie68ddf805836ee62f725019b0b873082b1d40948 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-12-25Merge "[cfg01] salt-master: Sync cfg from reclass"Michael Polenchuk1-0/+4
2018-12-21Pull out rocky patchesMichael Polenchuk1-0/+7
This commit should be reverted once original formulas get required support of rocky version. Change-Id: Ia3458381bced0cae8dbfacc9781c90933ad5c822 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2018-12-19[cfg01] salt-master: Sync cfg from reclassAlexandru Avadanii1-0/+4
Sync predefined Salt Master configuration with reclass class system.salt.master.single (but limit worker_threads to 20). Change-Id: I760cdcb9ebbdab517011eccab0616abb36014cc1 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-12-14[maas] Adopt maas, maasng proposed functionsAlexandru Avadanii1-1/+1
JIRA: FUEL-364 Change-Id: Ia470fc8103713e7a06cd9647675b0edfb4342bf8 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-12-12Make MTU cluster-level configurable via IDFAlexandru Avadanii1-0/+0
JIRA: FUEL-336 Change-Id: I1c8d22b8322f700eb727d9077035ba4c9f9f9753 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-12-08[cfg01] reclass: Apply broken default patch firstAlexandru Avadanii1-0/+4
reclass settings.py should be patched before the salt-master service is started (since we can't restart it inside the container without killing the tini init). Fixes: 2de5348a Change-Id: Id62d8f9f12fd72ef60322dd9907f26907231c4a7 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-12-04submodule: Bump pharos to latestPaul Vaduva1-0/+0
Change-Id: I1bf4452e0f6e9aa5d2b9a002a1ec45c70fb8c2ab Signed-off-by: Paul Vaduva <Paul.Vaduva@enea.com>
2018-11-29[docker] compose: Switch ip_range to ipv4_addressAlexandru Avadanii3-9/+11
Explicitly set the ipv4_address for each network instead of relying on ip_range allocation, which seems to fail / not be picked up. While at it, use docker-compose 1.22 or newer to bypass slow Docker network creation with 'macvlan' driver [1]. [1] https://github.com/docker/compose/issues/5248 Change-Id: Ic31851522576ebb2407d869b7c3ed7bd06951922 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-11-23Merge "[cfg01] Honor idf.net_config.mgmt.ip-range"Alexandru Avadanii1-1/+1
2018-11-20[cfg01] Honor idf.net_config.mgmt.ip-rangeAlexandru Avadanii1-1/+1
Previously, cfg01 mgmt address did not consider the `ip-range` param, leading to a mismatching `cfg01` entry in /etc/hosts on cluster nodes. Change-Id: If6f605f4b2817c3751074bef60ebde298bc74b7d Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-11-09submodule: Bump salt-formula-maasAlexandru Avadanii1-0/+0
* bump salt-formula-maas git submodule; * sync AArch64 initial salt config with the x86_64 default config; * bump Pharos git submodule to sync `power_pass` MaaS configuration paramater naming; Change-Id: Ic59dd8becb6d83a9e67004c38d51681c88c4be7c Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>