summaryrefslogtreecommitdiffstats
path: root/ci/deploy.sh
AgeCommit message (Collapse)AuthorFilesLines
2018-08-21[deploy.sh] Install Docker if not presentAlexandru Avadanii1-0/+1
JIRA: FUEL-383 Change-Id: I19d27ca59a3f24d1bd66e39457a6ca267bccce19 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-08-21[jump req] Add build/deploy specific requirementsAlexandru Avadanii1-1/+1
Add support for different prerequisites depending on the current operation (docker build or cluster deploy). Leverage the new support to pre-install upcoming deps: - python-pip (build); - docker-compose (deploy); JIRA: FUEL-383 Change-Id: Ic3e6062b1943e3584f0b1f80d2e33b8812defced Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-08-19[deploy.sh] Prefix and export more env varsAlexandru Avadanii1-13/+14
Upcoming support for containerized Salt Master node requires access to certain variables during j2 interpolation, so also export: - MCP_STORAGE_DIR; - MCP_REPO_ROOT_PATH; JIRA: FUEL-383 Change-Id: I584c0bf8133b5ae6178d97da5b44d345e45a0222 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-08-13[jump] Enable jumbo frames for vnet devicesAlexandru Avadanii1-0/+1
By default, vnet devices have a MTU of 1500 on the host side, causing issue with larger packets traversing the bridges between guest VMs when guest VMs have jumbo frames enabled. JIRA: FUEL-336 JIRA: FUEL-367 JIRA: FUEL-382 [1] http://linuxaleph.blogspot.com/2013/01/ how-to-network-jumbo-frames-to-kvm-guest.html [2] https://packetpushers.net/udev/ Change-Id: I941ac9cf764e3b3fa2d6463be5363b5459775f29 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-08-03[lib.sh] Reset virtual nodes after MaaS installAlexandru Avadanii1-0/+5
For hybrid PODs (e.g. x86_64 jumpserver + control nodes, aarch64 baremetal compute nodes), the virtual nodes rely on MaaS DHCP to be up when the OS boots, so issue a `virsh reset` accordingly. Instead of checking for online nodes using `test.ping`, use `saltutil.sync_all` to also sync Salt state modules to the virtual nodes (usually handled by baremetal_init state in HA deploys). JIRA: FUEL-338 Change-Id: If689d057dc4438102c3a7428a97b9638e21bfdc5 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-06-25[noha] Adopt vPDF usageAlexandru Avadanii1-5/+3
Previously, all virtual deployments relied on lab configuration from <./mcp/config> using 'local-virtual1' as POD name. To keep using that default sample config, one can still pass the appropiate arguments to <ci/deploy.sh> as in: $ ./ci/deploy.sh -l local -p virtual1 [...] This changed introduces support for vPOD-specific lab config, e.g.: $ ./ci/deploy.sh -l ericsson -p virtual4 Change-Id: I6ecb8e71d7e923e4b29b4ab0693c1afa5e19b582 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-05-08[virtual] Bring in OpenDaylight DPDK scenarioMichael Polenchuk1-1/+2
Change-Id: If77ac85fa86e0a1a18c0cc2abff77d876cdb9e93 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2018-03-07[novcp] Add deploy argument `-N` (experimental)Alexandru Avadanii1-4/+11
Instead of duplicating scenarios for NOVCP, allow it to be specified using a new deploy argument, `-N`. Things are getting convoluted, so instead of creating dedicated '*_pdf.yml.j2' files for each group of similar features, apply the templating in-place and rename all affected files to ".yml.j2". This breaks .gitignore assumption of hiding only "*_pdf.yml" files, so extend (manually) the <mcp/reclass/classes/cluster/.gitignore> with `git ls-files --exclude-standard -o` after an expansion. - ha: move nfv.cpu_pinning to j2, conditioned by 'baremetal'; - ha: add cmp00* vnode definitions (hugepages need more RAM); - labs/local: enable hugepage params for non-dpdk noha; - salt.sh: add route_wrapper to all non-infra VMs; This change extends novcp support to all HA scenarios. Change-Id: I7a80415ac33367ab227ececb4ffb1bc026546d36 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-03-07[vnode parsing] Move core logic to j2Alexandru Avadanii1-47/+13
j2/python is easier to read and manipulate strings, although it does need some special care about undefined dict keys. With this in place, deploy.sh only contains the higher level logic for the deployment process. - merge arch-specific default configuration files into a singular file with arch name as main dict key of old config (also avoids creating duplicate 'virtual' YAML keys in $LOCAL_PDF); - move template handling to separate <lib_template.sh>; - decouple tight bash ordering of scenario expansion -> parse_yaml -> variable export (e.g. CLUSTER_DOMAIN) -> re-use in cluster j2s; however we can't parse *all* j2s in one go, as scenario j2s might expand to YAMLs needed while expanding cluster j2; - split `do_templates` into separate functions for each stage, with no coupling between them other then call order; Change-Id: I4b5e804094c00e5e918caf769fd85fa52181ad76 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-03-04[jump req] Move pkg installation to lib.shAlexandru Avadanii1-12/+1
Change-Id: I687b73b256aca78c9d41d4bcd49bfbde51278b51 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-03-04[deploy.sh] Factor out jump bridge parsing to j2Alexandru Avadanii1-14/+2
Change-Id: Ie4d8e70866d533d195a6e80cdfecbdb00a3027ce Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-13[patch] MaaS: mcp.rsa.pub to auth keys via pillarAlexandru Avadanii1-1/+4
Drop one questionable patch responsible for MaaS node authorized keys to include mcp.rsa.pub by reading the contents of authorized keys on mas01, assuming mcp.rsa.pub will be on the first line. Instead, export the contents of the public key using a shell env var during deploy, which gets expanded via maas_pdf j2 template into a reclass param, leveraging existing salt-formula-maas sshprefs mechanism for delivering the key to MaaS. Since we require the public key to exist before expanding templates, move `generate_ssh_key` call outside the current infrastructure handling block, allowing it to execute during all `deploy.sh` calls, even for dry-runs. Change-Id: I0f53b0f764a2fafd292e0ffd399c284acf61bd30 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-07[jump] Add simple check for required Linux bridgesAlexandru Avadanii1-0/+3
- MaaS requires PXE/admin to be a Linux bridge; - if virtual nodes are present, they should be hooked to a proper Linux bridge for the Public network, but only throw a warning if not (and create a mock public virsh network instead); - if both virtual and baremetal nodes are present, Public bridge is indirectly mandatory (we can't mock it); JIRA: FUEL-339 Change-Id: Idfe99d66c49eadc56cb3d94ca4db3467fb76d388 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-06[hybrid] Merge config/scenario/{baremetal,virtual}Alexandru Avadanii1-5/+3
Instead of classifying scenarios by underlying machine type, switch to HA/NOHA differentiantion only. This allows us to add support for hybrid scenarios (with some virtual and some baremetal nodes in the same cluster). To facilitate this, we will template the scenario files, which is a small step towards SDF (Scenario Descriptor File) definition and adoption later. JIRA: FUEL-338 Change-Id: If5787991869a3105d82c27ffa0a86ac79b4b08ba Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-06[virtual] Add missing ODL dhcp_intAlexandru Avadanii1-2/+2
- add missing network definitions for ODL node's 1st interface; - add missing comments for `notify` global functions; - fix or silence shellcheck issues; JIRA: FUEL-322 Change-Id: Ie3341d29ab12ddf432db603ad865259afb54714e Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-05[virtual] Split 'pxebr' from 'mcpcontrol' netAlexandru Avadanii1-5/+6
- add new virsh managed network 'pxebr' (to mimic baremetal behavior on virtual PODs, this will be the equivalent of PXE/admin network); - connect 'pxebr' to 3rd interface for cfg01, mas01 for all deploys (used to be baremetal-specific), replacing 'internal'; - keep 'mcpcontrol' connected only to 'cfg01' (+ 'mas01' if present) for initial infrastructure bring-up (1st interface); - switch all virtual cluster nodes to 'pxebr' (1st interface); - use 'pxebr' for all Salt cluster nodes traffic, 'mcpcontrol' only for mas01<=>cfg01 Salt traffic; - convert <user-data.template> to jinja2 and expand it based on PDF instead of using `envsubst`; - split <user-data.sh.j2> into two versions, one for each network used for Salt traffic; - ci/deploy.sh: Read scenario data before template parsing for cluster domain variable, needed in virsh network def; - leave docs diagram refresh to later after all possible deploy types have settled; - limit keyserver proxy usage to nodes where the configured http proxy matches the first nameserver (true for all MaaS-provisioned nodes), so we can re-use the same pillar for FN VMs and baremetal nodes; - add PXE/admin IP on cfg01's 3rd interface and switch other vnodes `salt_master_host` to point to it; JIRA: FUEL-322 Change-Id: Ie4f7aedddf2ef81046f1127b377d88dce79f0fda Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-05[FN VM] Reboot VMs on jump, wait for all onlineAlexandru Avadanii1-1/+1
- apply `linux` state on cfg01 first, so PXE/admin IP is added and FN VM minions are available; - add barrier and wait for all FN VMs to register with cfg01; - use batch-mode execution while applying `linux.network` on FN VMs; - retry all states executed via <salt.sh> on FN VMs; JIRA: FUEL-310 Change-Id: I72e1c565370072500df1d486fe76e6315f583c75 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-05[PDF] Switch to generate_config, unify templatesAlexandru Avadanii1-37/+8
- move bash template handling (previously expanded via `envsubst`) to lib.sh; - move j2 template handling to lib.sh; - move virsh network templates to 'mcp/scripts/virsh_net' subdir; - switch virsh network templates from `envsubst` expansion to j2 and leverage generate_config.py, similar to PDF Fuel installer adapter; - add relevant runtime env vars (e.g. SALT_MASTER, MAAS_IP) on the fly to PDF, to consume them in templates like params coming from PDF; - parameterize virsh network definitions based on PDF (mgmt, public); JIRA: FUEL-322 Change-Id: Ib94e78fc4f25797b9354a0552e884104da5d0003 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-05deploy.sh: Move notify() to globals.shAlexandru Avadanii1-46/+24
Extend `notify` to 4 variants: - notify_i = inline (no newline) colored output; - notify = `notify_i` + trailing '\n'; - notify_n = `notify` + extra '\n' before and after; - notify_e = `notify` + stderr output + exit; This allows us to remove '\n' and cleanup the code a bit. While at it, fix some 'NOTE' messages going to stderr instead of stdout. Change-Id: I682e3344ae9e307c4a68ab31c7766bc91b12ee58 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-05deploy.sh: Make PDF, IDF mandatory for all deploysAlexandru Avadanii1-18/+13
- hard requiremenet of PDF/IDF configuration for all deployments; - expand j2 templates for virtual deploys too; Since until now we used the same model for *all* virtual PODs, one of the PDF/IDF sets for existing vPODs (e.g. ericsson-virtual3) can be re-used practically on any host, without defining new vPODs. JIRA: FUEL-322 Change-Id: Iac6aab91b6958d0e5e175ed142da6aafadc6fac6 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-05[vPDF] Use local-virtual1, unify pkg requirementsAlexandru Avadanii1-1/+4
Until PDF/IDF land in Pharos for all our virtual PODs, use a common vPDF we already provide as an example to mimic the old hardcoded behavior while leveraging PDF/IDF parameterization. As a consequence, python requirements previously only needed for baremetal should now also be installed for virtual deploys too. JIRA: FUEL-322 Change-Id: Ied1c907275285a9086450a15491ae516a0db1be2 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-05[baremetal] Retire example pod_config.yamlAlexandru Avadanii1-2/+3
It is easier to just generate the `pod_config.yaml` file than to maintain it, so let's remove it. While at it, link sample PDF/IDF inside pharos git submodule, so we don't have to pass a different lab-config URI to use the sample. To generate pod_config.yml for the sample PDF/IDF: $ ./ci/deploy.sh -l local -p pod1 -s os-odl-nofeature-ha -d $ cat mcp/deploy/images/pod_config.yml JIRA: FUEL-322 Change-Id: If5898f92ef54bebc31d57f9632959e9093a89250 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-05[PDF] pod1: Refresh PDF, IDF examplesAlexandru Avadanii1-0/+1
Sync latest changes from pharos git repo for our sample PDF/IDF: - move net_config from PDF to IDF; - minor cleanup; JIRA: FUEL-322 Change-Id: If6865ac61a4942a1dd5daf7081fd8faa67e0e7bf Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-01deploy.sh: Disable net.bridge.bridge-nf-callAlexandru Avadanii1-0/+1
JIRA: FUEL-334 Change-Id: I6d2499053dcfb7f99593fcd5c948b569bdcb9c9b Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-27deploy.sh: Retire `-B` argumentAlexandru Avadanii1-46/+6
All bridge information should be defined via IDF. Keeping `-B` support in deploy.sh only creates confusion. While at it, remove automatic bridge detection based on network addr. Change-Id: Id8b54e4887a97092daed5beb31fde4edd13057f1 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-25Merge "[lab-config] Use Pharos submodule by default"Alexandru Avadanii1-4/+10
2018-01-24[lab-config] Use Pharos submodule by defaultAlexandru Avadanii1-4/+10
- switch from securedlab to pharos as lab-config structure; - accomodate the move net_config from PDF to IDF in j2 templates; Change-Id: Ib04e4fb384568a6efd9e78a080857b663521ae88 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-23deploy.sh: Retire simplest scenario fallbackAlexandru Avadanii1-7/+2
Previously, when a requested scenario was missing, we used to default to the simplest scenario. Now that we have a critical mass of scenario definitions, retire this fallback mechanism. Change-Id: I3ba6b04290806ff78b3ec22b90fa054d45602869 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-06deploy.sh: Print sysinfoAlexandru Avadanii1-0/+3
JIRA: FUEL-323 Change-Id: I0dcbcfbedc7e9ef013ca50dcc08f804323f91701 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-05deploy.sh: Silence cleanup outputAlexandru Avadanii1-14/+1
Cleanup sequence might print some warnings/errors that are misleading for new users, so let's silence them. While at it, remove unused `cleanup` function stub. Change-Id: Ic4225f7d4b701cd15d96e1bfc5eecfe2c4291eb6 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-12-13deploy.sh: Fix missing space in dependencies listAlexandru Avadanii1-1/+1
Change-Id: I4d41ff8b9f7c1b59a2c1a400b9335f17b8c542be Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-12-12deploy.sh: Factor out pkg deps into yaml filesAlexandru Avadanii1-21/+27
Instead of hardcoding the packages to install on the jumphost at deploy time in `ci/deploy.sh`, move the requirements list to a dedicated yaml file, so it is decoupled from actual code. While at it, add missing deps: - ipxe-qemu (AArch64 Ubuntu only); Change-Id: I9ee5c0796b70f9d29427d608b6415333c8841e1e Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-12-11[baremetal] Move all MaaS PXE net config to PDFAlexandru Avadanii1-5/+1
- s/opnfv_maas_pxe_/opnfv_infra_maas_pxe_/g to align with other vars; - patches: pharos: Add MaaS PXE network to installer adapter; - runtime.yml{,.template}: move to installer adapter, update pod_config.yml example; - drop MAAS_PXE_NETWORK global env var, now read strictly from PDF; JIRA: FUEL-313 Change-Id: I46d7510bd53fba7890c411d36bc28fd6ff6f3648 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-12-08[baremetal] PDF: Use net_admin for MaaS PXE netAlexandru Avadanii1-1/+5
While at it, fix another small issue where a variable (parameters__param_opnfv_maas_pxe_address) was used without being populated (defined in 'runtime.yml', which is not read using `parse_yaml`, like the scenario yml or the PDF expanded class, pod_config.yml). JIRA: FUEL-313 Change-Id: Iee88601d5420f55572bf90c8cf330afbedfb2e21 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-12-07Merge "base image: Explicitly call kpartx"Michael Polenchuk1-2/+2
2017-12-04Rename reclass models to add "-ha" or "-noha"Alexandru Avadanii1-3/+1
Parse all reclass j2 templates, not only common + current scenario (useful when adding new scenarios later). JIRA: FUEL-310 Change-Id: I8e87af702f83c42cb8f766bf6f121449aa5f2c26 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-29Merge "AArch64: base image: pre-install salt-minion"Michael Polenchuk1-2/+2
2017-11-28[virtual] Update openstack version to pikeMichael Polenchuk1-1/+1
Change-Id: I1df0228cb44bf9122aaf93dd25fc16a0d26a5240 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2017-11-28base image: Explicitly call kpartxAlexandru Avadanii1-2/+2
qemu-nbd currently available in CentOS 7 does not add partition mappings automatically for NBD devices, so add explicit `kpartx` calls. Change-Id: Ifa79c89b82024602b782c449dbf4de10899e03b5 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-27AArch64: base image: pre-install salt-minionAlexandru Avadanii1-2/+2
While at it, rename apt repo in foundation node user-data template from "salt" to "saltstack", to align with reclass model naming. Change-Id: I5b216492349ae187b568884b1ab4046c52b1c6b2 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-27Add pre-{install,purge} support for base imageAlexandru Avadanii1-2/+21
Extend <lib.sh> and its invocation from <ci.deploy.sh> with support for modifying foundation node VMs base image prior to using it with: - additional APT GPG keys; - additional APT repos; - packages to pre-install; - packages to pre-remove; - (non-configurable) cloud init datasource via NoCloud only, so VCP VMs won't wait for metadata service; While at it, re-use the resulting image as a base for another round of pre-patching (same operations as above are supported) to provide a base image for VCP VMs. Add AArch64-specific configuration based on new mechanisms: - pre-install linux-image-generic-hwe-16.04-edge (and headers) for foundation node and VCP (common) image (also requires new repo and its key); - pre-install cloud-init for VCP image (it should already be installed, but script needs non-empty config for VCP to create the VCP image and transfer it over to Salt Master); NOTE: cloud-init is required on VCP VMs for DHCP on 1st iface. JIRA: FUEL-309 Change-Id: I7dcaf0ffd9c57009133c6d339496ec831ab14375 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-21ci/deploy.sh: Add new `-E` arg for env eraseAlexandru Avadanii1-4/+14
NOTE: In order to undefine VCP VMs with NVRAM (e.g. AArch64 VMs using AAVMF), an additional parameter should be passed to libvirt by Salt virt core module (equivalent to `virsh undefine --nvram`). While at it, pass CI_DEBUG, ERASE_ENV enviroment variables to state execution, and stop force-applying patches. Also refactor the rsync between foundation node and Salt master, so the whole git repo is copied as </root/opnfv>, and <root/fuel> becomes a link to it; useful for Armband, where 'fuel' is a git submodule. Fix .git paths after rsync, so git submodules work as expected in cfg01 repos. JIRA: FUEL-307 Change-Id: Ic62f03e786581c019168c50ccc50107238021d7f Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-10AArch64: deploy.sh: Add AAVMF install prereqAlexandru Avadanii1-0/+6
Change-Id: I7b3bb3eac1b4665dc9c08eaf6f5af336cde31f87 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-10deploy.sh: Check hypervisor after prereq installAlexandru Avadanii1-7/+9
On a clean system (i.e. no libvirt installed), the prerequisites should be installed first, before checking for hypervisor access. While at it, add 'libvirt' to the list of prerequisites, as it is not automatically required by the others. Add a note in the documentation about libvirtd service not being automatically started after installation on CentOS. Change-Id: I29d94d4a56cd823362811e8fc9c076c83fb6c04c Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-09[baremetal] package: install python-yamlTaseer Ahmed1-2/+2
- Required for PDF parsing Change-Id: I95205e62efab3e9920687866d3a53a135828a218 Signed-off-by: Taseer Ahmed <taseer94@gmail.com>
2017-10-27Merge "ci/deploy.sh: Add PDF parsing dependencies"Alexandru Avadanii1-0/+8
2017-10-27Merge "lib.sh: Fix compatibility with bash 4.2"Alexandru Avadanii1-8/+9
2017-10-27ci/deploy.sh: Add PDF parsing dependenciesAlexandru Avadanii1-0/+8
The following packages are required for PDF parsing: - python - python-ipaddress - python-jinja2 Note that `python-ipaddress` is not available on older Ubuntu distributions (e.g. Trusty), in which case `pip` should be used to install it. An alternative would be to use python3 packages. Update the installation instructions to specify minimum expected requirements on the jump server. JIRA: FUEL-292 Change-Id: I19fdcc163dcfdee45ac6fa3f5e09a28638d6035c Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-10-27Merge "AArch64: Switch jumpserver VMs to virtio video"Alexandru Avadanii1-6/+0
2017-10-27lib.sh: Fix compatibility with bash 4.2Alexandru Avadanii1-8/+9
JIRA: FUEL-296 Change-Id: Ide9f9333fe9b44ff6b78678064f8e67f05aabd42 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>