aboutsummaryrefslogtreecommitdiffstats
path: root/tools
AgeCommit message (Collapse)AuthorFilesLines
2017-06-30Allow volumes in puppet_config containers specJames Slagle1-1/+1
Mounting host volumes when running containers via puppet_config already works and is supported with docker-puppet.py. However, the validation in yaml-validate.py does not allow it. This patch makes it allowed by the validation. It is sometimes necessary since some puppet modules expect to make persistent file system changes other than just configuration data under /etc. In particular, ironic inspector expects to configure a http and tftp root director with an ipxe configuration. See: https://github.com/openstack/puppet-ironic/blob/master/manifests/inspector.pp These changes would be lost if the value for those directories are not mounted as host volumes. Change-Id: Ie51c653f4c666fbaaef0ea80990e2e61f4b1353b
2017-06-28Add heat parameter for all of config_volume imagesMartin André1-0/+17
This commit consistently defines a heat template parameter in the form of DockerXXXConfigImage where XXX represents the name of the config_volume that is used by docker-puppet. The goal is to mitigate hard to debug errors where the templates would set different defaults for the image docker-puppet.py uses to run, for the same config_volume name. This fixes a couple of inconsistencies on the way. Change-Id: I212020a76622a03521385a6cae4ce73e51ce5b6b Closes-Bug: #1699791
2017-06-27Add validation check for divergent parameter definitionsBen Nemec1-4/+71
Many of our parameters are defined in multiple templates, but currently there is no easy way of checking that all of those definitions match. It can be confusing when a parameter is defined one way in one file and another way in a different file. For example, the NovaWorkers description is: Number of workers for Nova API service. and Number of workers for Nova Placement API service. and Number of workers for Nova Conductor service. Which is it actually? All of them. That one parameter controls the workers for all of the nova services, and its description should reflect that, no matter which template you happen to look at. This change adds a check to yaml-validate.py to catch these sorts of inconsistencies and allow us to eventually prevent new ones from getting into the templates. An exclusion mechanism is included because there are some parameter definitions we probably can't/shouldn't change. In particular, this includes the network cidrs which are defaulted to ipv4 addresses in the ipv4 net-iso templates and ipv6 in the ipv6 templates. It's possible a user would be relying on one of those defaults in their configuration, so if we change it they might break. To get around that, the tool explicitly ignores the default field of those parameters, while still checking the description and type fields so we maintain some sanity. There may be other parameters where this is an issue, but those can be added later as they are found. For the moment any inconsistencies are soft-fails. A failure message will be printed, but the return value will not be affected so we can add the tool without first having to fix every divergent parameter definition in tripleo-heat-templates (and there appear to be plenty). This will allow us to gradually fix the parameters over time, and once that is done we can make this a hard-fail. Change-Id: Ib8b2cb5e610022d2bbcec9f2e2d30d9a7c2be511 Partial-Bug: 1700664
2017-06-19Merge "Support config dir for env generator input files"Jenkins1-3/+10
2017-06-12Support config dir for env generator input filesBen Nemec1-3/+10
We're not going to want to list every single sample environment in a single file, so let's also take a directory and just read every yaml file in it. This commit adds support for that as well as some initial environments to demonstrate its use. Change-Id: If2c608f2a61fc5e16784ab594d23f1fa335e1d3c
2017-06-09Remove duplicate docker/puppet services.yamlSteven Hardy1-4/+2
Move to one common services.yaml not only reduces the duplication, but it should improve performance for the docker/services.yaml case, because we were creating two ResourceChains with $many services which we know can be really slow (especially since we seem to be missing concurrent: true on one) Change-Id: I76f188438bfc6449b152c2861d99738e6eb3c61b
2017-05-15Add role specific information to the service templateSaravanan KR1-1/+2
When a service is enabled on multiple roles, the parameters for the service will be global. This change enables an option to provide role specific parameter to services and other templates. Two new parameters - RoleName and RoleParameters, are added to the service template. RoleName provides the role name of on which the current instance of the service is being applied on. RoleParameters provides the list of parameters which are configured specific to the role in the environment file, like below: parameters_default: # Default value for applied to all roles NovaReservedHostMemory: 2048 ComputeDpdkParameters: # Applied only to ComputeDpdk role NovaReservedHostMemory: 4096 In above sample, the cluster contains 2 roles - Compute, ComputeDpdk. The values of ComputeDpdkParameters will be passed on to the templates as RoleParameters while creating the stack for ComputeDpdk role. The parameter which supports role specific configuration, should find the parameter first in in the RoleParameters list, if not found, then the default (for all roles) should be used. Implements: blueprint tripleo-derive-parameters Change-Id: I72376a803ec6b2ed93903cc0c95a6ffce718b6dc
2017-04-22Merge "Replace six.iteritems() with .items()"Jenkins1-1/+1
2017-04-21Merge "Add network_data.yaml to encapsulate list of networks for j2"Jenkins1-3/+13
2017-04-14Rework DOCKER_PUPPET_CONFIG validate toolBogdan Dobrelya1-7/+11
* Split it to REQUIRED/OPTIONAL * Move puppet_tags to OPTIONAL as it already has a default set of tags that need not to be repeated explicitly. Change-Id: Ib70176f1edf61228771c983b0c3231fb7939a316 Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2017-04-07Replace six.iteritems() with .items()loooosy1-1/+1
1.As mentioned in [1], we should avoid using six.iteritems to achieve iterators. We can use dict.items instead, as it will return iterators in PY3 as well. And dict.items/keys will more readable. 2.In py2, the performance about list should be negligible, see the link [2]. [1] https://wiki.openstack.org/wiki/Python3 [2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html Change-Id: I7c8f540eced0731f4dfb1dfd045828b5a9bb6c67
2017-03-29Qpid dispatch router composable roleJohn Eckersberg1-1/+3
Note: since it replaces rabbitmq, in order to aim for the smallest amount of changes the service_name is called 'rabbitmq' so all the other services do not need additional logic to use qdr. Depends-On: Idecbbabdd4f06a37ff0cfb34dc23732b1176a608 Change-Id: I27f01d2570fa32de91ffe1991dc873cdf2293dbc
2017-03-20Change kolla_config from required to optional in pep8.Ian Main1-3/+3
We've decided to use volumes for configuration wherever possible. This means moving away from kolla_config blocks in the templates. Update pep8 to reflect this. Change-Id: If1ec40d0e5a515eed35e0cd04711079294f358c3
2017-03-16docker: Add metadata_settings to optional parameters for yaml validateJuan Antonio Osorio Robles1-1/+2
This section will be needed for TLS-everywhere. So it should be added as optional in the yaml-validate. Change-Id: Ic6ea563b6c8e454cb51f640bb5aaa3adda82a5dd
2017-03-13Tasks hook for preparing BM host for deploying containerized servicesJiri Stransky1-1/+1
This implements a host_prep_tasks hook where we can specify Ansible tasks to perform on the host before deploying containerized services. The hook runs in a single step, the assumption is that we will mostly use the hook for creating per-service directories on the host to ensure we are able to mount them into the containers. (We cannot do this operation via Puppet because all containerized services run their Puppet within a config container, so Puppet doesn't have access to host's filesystem.) Change-Id: I7d8bac39e0cd422fd651eefe29f7d10941ab4a1a
2017-03-10Add pep8 tests on docker/services/*Dan Prince1-0/+61
This patch adds the beginning of a set of unit tests for the new docker services templates. This should help us the new interfaces as they evolve. Change-Id: I98a73cf090ebab4593a682f5f34c0950d37e010c
2017-03-05Add network_data.yaml to encapsulate list of networks for j2Steven Hardy1-3/+13
This moves the hard-coded networks from the default environment, and provides the first step towards enabling composable networks. Co-Author: Dan Sneddon <dsneddon@redhat.com> Partial-Bug: #1633090 Depends-On: I9f818912bd8e2a3220e41c8ccbbab3d9063b4d72 Change-Id: I7793b8badede5450b05437c84d9b40c28de7546b
2017-02-28Align hyperconverged-ceph.yaml environment and adds some validationGiulio Fidente1-0/+18
Until bug #1635409 is fixed we'll have to keep the default list of services deployed by hyperconverged-ceph.yaml in sync with the ServicesDefault list provided in roles_data.yaml This change adds some logic in the templates validation script to ensure that is preserved with future updates. Change-Id: Ib767f9a24c3541b16f96bd6b6455cf797113fbd8
2017-02-17Make the DB URIs host-independent for all servicesMichele Baldessari1-5/+3
When fixing LP#1643487 we added ?bind_address to all DB URIs. Since this clashes with Cellsv2 due to the URIs becoming host dependent, we need a new approach to pass bind_address to pymysql that leaves the DB URIs host-independent. In change Iff8bd2d9ee85f7bb1445aa2e1b3cfbff1f397b18 we first create a /etc/my.cnf.d/tripleo.cnf file with a [tripleo] section with the correct bind-address option. In this change we make sure that the DB URIs will point to the added file and to the specific section containing the necessary bind-address option. We do introduce a new MySQLClient profile which will hold all this more client-specific configuration so that this change can fit better in the composable roles work. Also, in the future it might contain the necessary configuration for SSL for example. Note that in case the /etc/my.cnf.d/tripleo.cnf file does not exist (because it is created via the mysqlclient profile), things keep on working as usual and the bind-address option simply won't be set, which has no impact on hosts where there are no VIPs. Co-Authored-By: Damien Ciabrini <dciabrin@redhat.com> Change-Id: Ieac33efe38f32e949fd89545eb1cd8e0fe114a12 Related-Bug: #1643487 Closes-Bug: #1663181 Closes-Bug: #1664524 Depends-On: Iff8bd2d9ee85f7bb1445aa2e1b3cfbff1f397b18
2017-02-07Stop setting bind_address on nova db uri.Oliver Walsh1-1/+4
This reverts the changes in https://review.openstack.org/414629 for nova as they are incompatible with cell_v2. This is a temporary fix for HA while a long-term solution is developed. Change-Id: I79d30a2d76a354999152c0c997ea77f104c51027 Related-bug: #1643487 Closes-bug: #1662344
2017-02-02Moving the validation for using the template alias version for all templatesCarlos Camacho1-4/+7
Currently we are applying this validation for the services templates, this submission moves it to run with all templates. Also fixed those templates not using the alias name. Change-Id: I3a2c0ce6adcc8061fdc51f73fdc6b9748c0fead9
2017-01-27Add support for Jinja2 includesOliver Walsh1-1/+8
This replicates the behavior of the custom Jinja2 loader from tripleo-common to allow template validation on the local filesystem using tox. Change-Id: I27683ab31187c6334dc5b4b5363a3347874b9a90 Partially-Implements: blueprint overcloud-upgrades-per-service Depends-On: Idc5c3f49c7a2fc7f3622c76da001992cc657384e
2017-01-24Specify what to do if endpoint map environment don't matchJuan Antonio Osorio Robles1-2/+2
this attempts to make the error message more useful. This error message happens if the environment files containing endpoint map overrides haven't been updated to match the base endpoint map (or the defaults). Change-Id: If53d3a9d7848aed62ebb235afe8b14c18d1b284d
2017-01-23Merge "Allow for specifying the output_dir"Jenkins1-5/+28
2017-01-23Merge "Ignore hidden files in directories"Jenkins1-0/+12
2017-01-23Allow for specifying the output_dirFlavio Percoco1-5/+28
Allow for passing the output_dir in the process-templates scripts so that it doesn't overwrites the templates in the src dir. This is a desired feature when running the script from a t-h-t installed system-wide. Change-Id: I47994d34f47a4084a11124bc9075cb2f457889ea
2017-01-23Ignore hidden files in directoriesFlavio Percoco1-0/+12
Don't walk through hidden files. This avoids going through the .git, .tox and other hidden directories that we don't care about. Change-Id: I34b83229775d221299c8b572a7049175debac99d
2017-01-18Check that all templates are using the release aliasCarlos Camacho1-0/+4
Quick verification to check that the release name is used instead the date. Im also adding here all the updated templates required to pass the check and merge this check as soon as possible. Change-Id: Ifdc9ac4a9d0a4872d3e21672c93fc87da2e68a4e
2017-01-17Add endpoint map matching validation to validate-yaml.pyJuan Antonio Osorio Robles1-0/+62
This validation checks that the TLS-related environment files contain all of the services defined in the base endpoint map. This will hopefully help to keep them updated. Change-Id: I58df72e104d8eb74e577484405f15e0a6f92d0ce
2017-01-13Add ReNo supportEmilien Macchi2-0/+58
Add ReNo support to manage release notes. http://docs.openstack.org/developer/reno/ Change-Id: Ie5154d909e616e4e7e813052f9c121d6ac5b0875
2017-01-11Fix error path for tools/process-templates.pySteven Hardy1-0/+1
The error path for this fails because we don't import six or install it in the templates tox venv Change-Id: Ie9f46332f2b03d48a1b0a4a432e9721757833569
2017-01-03DB connection: prevent src address from binding to a VIPDamien Ciabrini1-0/+45
When a service connects to the database VIP from the node hosting this VIP, the resulting TCP socket has a src address which is by default bound to the VIP as well. If the VIP is failed over to another node while the socket's Send-Q is not empty, TCP keepalive won't engage and the service will become unavailable for a very long time (by default more than 10m). To prevent failover issues, DB connections should have the src address of their TCP socket bound to the IP of the network interface used for MySQL traffic. This is achieved by passing a new option to the database connection URIs. This option is available starting from PyMySQL 0.7.9-2. We use a new intermediate variable in hiera to hold the IP to be used as a source address for all DB connections. All services adapt their database URI accordingly. Moreover, a new YAML validation check is added to guarantee that new services will construct their database URI appropriately. Change-Id: Ic69de63acbfb992314ea30a3a9b17c0b5341c035 Closes-Bug: #1643487
2016-11-30Add local template generation tox taskDan Prince1-0/+125
This patch adds a local version of our template processing routine so that developers can more quickly view the templates that are actually getting generated. I've noticed multiple developers now do a full deployment with 'overcloud deploy' only to download the swift container with the generated templates. This simple task avoids that step by allowing developers to generate it locally. It also aims to preserve the ability to use t-h-t templates directly with Heat (instead of going through Mistral) should users wish to do that. The new undercloud heat installer requires the ability to generate templates without requiring Mistral and Swift to do so. Ideally the Mistral API workflow would use this same code so perhaps in the future we might modify that routine to: -download swift tarball containing the templates -run this local routine that lives in t-h-t -re-upload the tarball of templates to the swift container Change-Id: Ie664c9c5f455b7320a58a26f35bc403355408d9b
2016-11-28Apply os-net-config with a script instead of elementDan Prince1-0/+219
Wire in os-net-config via a normal script heat deployment, which has the following advantages: 1. Improved error path, currently o-a-c deployments don't report any errors, thus hang and eventually the deployment times out 2. It's far more hackable from a deployer perspective, e.g it's much easier to change the os-net-config options or include a mapping file 3. Reduces our dependencies on o-a-c (it's only os-net-config and hiera which requires it), although the script does currently still use oac to get the metadata IP. 4. May enable passing os-net-config yaml via a json parameter in future, reducing the need for resource_registry mappings (although we'll have to support that for backwards compatibility) The script used is based directly on 20-os-net-config (from t-i-e at cf94c5e, we can probably improve this now that we have an error path, but for this initial commit it's a straight copy other than the changes to replace o-a-c for rendering the json config file. Co-Authored-By: Steven Hardy <shardy@redhat.com> Change-Id: I0ed08332cfc49a579de2e83960f0d8047690b97a
2016-09-09Convert overcloud.yaml to support jinja2 templatingSteven Hardy1-10/+11
The first step of generating the Service chain resources via j2, we'll then incrementally convert other resources to be created in a similar way. Partially-Implements: blueprint custom-roles Depends-On: I81239991f36ed5f6453184bf9cffe930832cb68b Change-Id: Iafa9b2afddf18a5a9833ec472a552fb256338b38
2016-08-26Skip warning for required parametersBrent Eagles1-1/+4
Skips the unused parameter warning for required parameters. Change-Id: I71ad4ab9f6e6c63e3f01b8cc9c72262f1958331e
2016-08-18Add DefaultPasswords to composable servicesDan Prince1-1/+1
This patch adds a new DefaultPasswords parameter to composable services. This is needed to help provide access to top level password resources that overcloud.yaml currently manages (passwords for Rabbit, Mysql, etc.). Moving the RandomString resources into composable services would cause them to regenerate within the stack. With this approach we can leave them where they are while we deprecate the top level mechanism and move the code that uses the passwords into the composable services. Change-Id: I4f21603c58a169a093962594e860933306879e3f
2016-08-18Pass ServiceNetMap to servicesGiulio Fidente1-0/+7
This will be needed to pick the network where the service has to bind to from within the service template. Change-Id: I52652e1ad8c7b360efd2c7af199e35932aaaea8c
2016-07-28Convert service_name to underscore syntaxSteven Hardy1-1/+2
Currently we use hyphens, e.g cinder-api, but in overcloud.yaml we have a lot of references to services (e.g for AllNodesConfig) by underscore, e.g cinder_api. To enable dynamic generation of this data, we need the service name in underscore format. Change-Id: Ief13dfe5d8d7691dfe2534ad5c39d7eacbcb6f70
2016-07-22Add 'service_name' to composable servicesDan Prince1-3/+29
This patch adds a new service_name section to each composable service. We now have an explicit unit test check to ensure that service_name exists in tools/yaml-validate.py. This patch also wires service_names into hieradata on each of the roles so that tools can access the deployed services locally during deployment and upgrades. Change-Id: I60861c5aa760534db3e314bba16a13b90ea72f0c
2016-03-23Add simple parameter test to yaml-validate.pySteven Hardy1-1/+10
Output a warning for parameters which look unused, this should help developers clean up the template a bit, and eventually could maybe be developed further into something we can use for gating. Change-Id: Ide4fbe3c85854cbddee44801d39ae73003d63bb8
2016-01-28Update yaml-validate.py to accept files or directoriesSteven Hardy1-11/+30
For developer usage it's helpful to have the choice to provide either an individual files, list of files, or some mix of files and directories as you don't necessarily want to walk everything all the time. Change-Id: I050de123bba51402a0dbb42d71e97fd27d7ce4bc
2015-09-15Add YAML sanity checkBen Nemec1-0/+46
Adds a "validate" tox env for basic sanity checking of templates. Currently it just validates that all of the .yaml files are in fact valid YAML. In the future we might want to add more, but this seemed like a reasonable start. Change-Id: I8091bbad0003b150e23dae5de4f465053c982229