summaryrefslogtreecommitdiffstats
path: root/docker/services/pacemaker
AgeCommit message (Collapse)AuthorFilesLines
2017-09-14Retry if the pacemaker_resource commands failedMathieu Bultel6-0/+36
Add a retry when the pacemaker_resource command wasn't apply correctly, more info here: https://bugzilla.redhat.com/show_bug.cgi?id=1482116 This is the same approach puppet-pacemaker uses and provides eventual consistency when multiple nodes change the cluster CIB concurrently. This change depends-on : https://review.gerrithub.io/375982 The return code is not available in the current ansible-pacemaker package. Change-Id: I8da03f5c4a6d442617b81be5793a9724cc8842bf (cherry picked from commit e92430d8d03fc2ce2d0ce192b96209f2c5c04169)
2017-09-11Enable redis TLS proxy in HA deploymentsMartin André1-26/+67
Redis does not have TLS out of the box. Let's use a proxy container for TLS termination. This commit enables redis TLS proxy for the HA deployment. bp tls-via-certmonger Change-Id: I45e539872a03878337def33c681c4577c1a5629e (cherry picked from commit c6d8df01d7aa8b44af9ac152b3bb08f07e2e02b7)
2017-09-07Merge "Support HA for OVN DBs containers using pacemaker bundle" into ↵Jenkins1-0/+140
stable/pike
2017-09-06Mount public certificate in haproxy init containerJuan Antonio Osorio Robles1-0/+1
It's being mounted on the actual haproxy container, but not the init one. Change-Id: I66b69e0bb3642dbfeec767ef5216d515786b5b19 Closes-Bug: #1715132 (cherry picked from commit 03622e89ac3037b4d69d913586823e689b210688)
2017-08-31Add --wsrep-provider=none to the mysql_bootstrap containerMichele Baldessari1-2/+2
Depending on the version of mariadb/galera installed the mysql_bootstrap command might fail. With the following unrevealing error: openstack-mariadb-docker:2017-08-28.10 "bash -ec 'if [ -e /v" 3 hours ago Exited (124) 3 hours ago The timeout is actually due to the fact that the following snippets does not complete within 60 seconds: """ if [ -e /var/lib/mysql/mysql ]; then exit 0; fi kolla_start mysqld_safe --skip-networking --wsrep-on=OFF & timeout ${DB_MAX_TIMEOUT} /bin/bash -c ''until mysqladmin -uroot -p"${DB_ROOT_PASSWORD}" ping 2>/dev/null; do sleep 1; done'' mysql -uroot -p"${DB_ROOT_PASSWORD}" -e "CREATE USER ''clustercheck''@''localhost'' IDENTIFIED BY '${DB_CLUSTERCHECK_PASSWORD}'';" mysql -uroot -p"${DB_ROOT_PASSWORD}" -e "GRANT PROCESS ON *.* TO ''clustercheck' """ The problem is that with older mariadb versions: galera-25.3.16-3.el7ost.x86_64 mariadb-5.5.56-2.el7.x86_64 The mysqld_safe process starts in galera mode (as opposed as to single local mode): 170830 17:03:05 [Note] WSREP: Start replication 170830 17:03:05 [Note] WSREP: GMCast version 0 ... 170830 17:03:05 [ERROR] WSREP: wsrep::connect() failed: 7 170830 17:03:05 [ERROR] Aborting That means that even though we specified --wsrep-on=OFF it is still starting in cluster mode. Let's add the extra --wsrep-provider=none which older versions required. Let's also add a '-x' to this transient container as that would have helped a bit because we would have understood right away that it was mysqld_safe that was not starting. I tested this successfully on an environment that showed the problem. The new option is still accepted by newer DB versions in any case. Closes-Bug: #1714057 Change-Id: Icf67fd2fbf520e8a62405b4d49e8d5169ff3925b Co-Authored-By: Mike Bayer <mbayer@redhat.com> (cherry picked from commit c19968ca852ab608513fe692aab958af25276220)
2017-08-31Support HA for OVN DBs containers using pacemaker bundleNuman Siddique1-0/+140
ovn-dbs pacemaker bundle resources are created for supporting Master/Slave HA. puppet-tripleo already supports creating ovn-dbs bundle resources. The heat template added in this patch makes use of this. Closes-bug: #1699085 Change-Id: I23c2d312cfb144f9afc14f0982a92670dc29d74c (cherry picked from commit 444a39f5983e71e3222b6b7f8f523fce60aeece7)
2017-08-30Remove src_ceph from manila kolla_configJan Provaznik1-5/+0
Pacemaker puppet module takes care of mounting /etc/ceph into manila-share container (I23b6890b4cf7f1e6fe84b6be280dde82218275fc). Change-Id: I1026b2436275b17cfe3ac85192d42c5268f0a630 Related-To: I23b6890b4cf7f1e6fe84b6be280dde82218275fc (cherry picked from commit 0d8040ca33d42dbb7e06162f2b659ff6cbc0316f)
2017-08-18Tag the ha containers with 'pcmklatest' at deploy timeMichele Baldessari7-18/+221
We need to tag the HA containers with a special tag so that the RA definition never changes. We do this step in THT as opposed to puppet because we need to guarantee that all images are tagged on all nodes *before* step 2 where the bundle gets created. NB: Getting the image name without the tag will require some more yaql work to get all the cases right. Right now this works only if we enforce that the image has a ':tag' at the end of the name. So far this is always the case. If things change we will need to amend this code. Co-Authored-By: Damien Ciabrini <dciabrin@redhat.com> Co-Authored-By: Sofer Athlan-Guyot <sathlang@redhat.com> Change-Id: I362e6cf26fba77d3f949b7d2fc4b35a3eab9087e
2017-08-18Merge "Containerize Manila Share for HA"Jenkins1-0/+142
2017-08-17Merge "Enable TLS configuration for containerized RabbitMQ"Jenkins1-0/+15
2017-08-17Merge "Enable TLS configuration for containerized HAProxy"Jenkins1-5/+52
2017-08-16Containerize Manila Share for HAVictoria Martinez de la Cruz1-0/+142
This service allows configuring and deploying manila-share containers in a HA overcloud managed by pacemaker. The containers are managed and run by pacemaker. Pacemaker runs the standard Kolla image but overrides the initial command so that it explicitely calls manila-share. This way, we shield ourselves from any unexpected future change in Kolla. This container needs to use the 'docker_config' section to invoke puppet (as opposed to 'docker_puppet_tasks'), because due to the HA composability each resource creation needs to happen on the bootstrap node of that service and 'docker_puppet_tasks' will only run on the controller/primary role. Based on work done in fdb233e64e3d78014dd7e351abfed5aec5035866 Partial-Bug: #1668922 Change-Id: Ifa94c506db5eb667690a19d594115a93d2a790b2 Depends-On: I797eea2f7788f65411964ccb852b5707e916416f
2017-08-15Merge "Do not run clustercheck on the host after O->P upgrade"Jenkins1-0/+6
2017-08-14Merge "Enable TLS configuration for containerized Galera"Jenkins1-0/+35
2017-08-11Enable TLS configuration for containerized GaleraDamien Ciabrini1-0/+35
In non-containerized deployments, Galera can be configured to use TLS for gcomm group communication when enable_internal_tls is set to true. Fix the metadata service definition and update the Kolla configuration to make gcomm use TLS in containers, if configured. bp tls-via-certmonger-containers Change-Id: Ibead27be81910f946d64b8e5421bcc41210d7430 Co-Authored-By: Juan Antonio Osorio Robles <jaosorior@redhat.com> Closes-Bug: #1708135 Depends-On: If845baa7b0a437c28148c817b7f94d540ca15814
2017-08-09Enable TLS configuration for containerized HAProxyDamien Ciabrini1-5/+52
In non-containerized deployments, HAProxy can be configured to use TLS for proxying internal services. Fix the creation of the of the haproxy bundle resource to enable TLS when configured. The keys and certs files are all passed as configuration files and must be copied by Kolla at container startup. For the time being, disable the use of the CRL file until we find a means of restarting the containerized HAProxy service when that file expires. Change-Id: If307e3357dccb7e96bdb80c9c06d66a09b55f3bd Depends-On: I4b72739446c63f0f0ac9f859314a4d6746e20255 Closes-Bug: #1709563
2017-08-09Enable TLS configuration for containerized RabbitMQDamien Ciabrini1-0/+15
In non-containerized deployments, RabbitMQ can be configured to use TLS for serving and mirroring traffic. Fix the creation of the rabbitmq bundle resource to enable TLS when configured. The key and cert are passed as other configuration files and must be copied by Kolla at container startup. Change-Id: I8af63a1cb710e687a593505c0202d717842d5496 Depends-On: Ia64d79462de7012e5bceebf0ffe478a1cccdd6c9 Closes-Bug: #1709558
2017-08-08Merge "MariaDB: create clustercheck user at container bootstrap"Jenkins1-1/+22
2017-07-31MariaDB: create clustercheck user at container bootstrapDamien Ciabrini1-1/+22
In HA overclouds, the helper script clustercheck is called by HAProxy to poll the state of the galera cluster. Make sure that a dedicated clustercheck user is created at deployment, like it is currently done in Ocata. The creation of the clustercheck user happens on all controller nodes, right after the database creation. This way, it does not need to wait for the galera cluster to be up and running. Partial-Bug: #1707683 Change-Id: If8e0b3f9e4f317fde5328e71115aab87a5fa655f
2017-07-27Generate MySQL client config if service requires databaseDamien Ciabrini2-2/+16
Services that access database have to read an extra MySQL configuration file /etc/my.cnf.d/tripleo.cnf which holds client-only settings, like client bind address and SSL configuration. The configuration file is thus used by containerized services, but also by non-containerized services that still run on the host. In order to generate that client configuration file appropriately both on the host and for containers, 1) the MySQLClient service must be included by the role; 2) every containerized service which uses the database must include the mysql::client profile in the docker-puppet config generation step. By including the mysql::client profile in each containerized service, we ensure that any change in configuration file will be reflected in the service's /var/lib/config-data/{service}, and that paunch will restart the service's container automatically. We now only rely on MySQLClient from puppet/services, to make it possible to generate /etc/my.cnf.d/tripleo.cnf on the host, and to set the hiera keys that drive the generation of that config file in containers via docker-puppet. We include a new YAML validation step to ensure that any service which depends on MySQL will initialize the mysql::client profile during the docker-puppet step. Change-Id: I0dab1dc9caef1e749f1c42cfefeba179caebc8d7
2017-07-27Do not run clustercheck on the host after O->P upgradeDamien Ciabrini1-0/+6
Once an Ocata overcloud is upgraded to Pike, clustercheck should only be running in a dedicated container, and xinetd should no longer manage it on the host. Fix the mysql upgrade_task accordingly. Change-Id: I01acacc2ff7bcc867760b298fad6ff11742a2afb Closes-Bug: #1706612
2017-07-26Merge "Open up firewall for the control-ports in the bundles"Jenkins3-1/+26
2017-07-21Open up firewall for the control-ports in the bundlesMichele Baldessari3-1/+26
This is required when the bundles run on pacemaker remote nodes otherwise the cluster won't be able to connect to the control-ports of each bundle. The only services that need this are rabbit, redis and galera because those run pacemaker_remote inside the container (A/P resources and haproxy do not) Change-Id: I6a56d79319ef3d14973a0586dcda4d523adda7aa Co-Authored-By: Damien Ciabrini <dciabrin@redhat.com>
2017-07-20Remove non-containerized pacemaker resources on upgrademarios6-10/+140
Adds upgrade_tasks to remove the pacemaker resources using the ansible-pacemaker module. Resources are disabled and removed in step2 (called only on bootstrap node) and then the cluster stop is moved to step3 The existing systemd/service call is kept but only to disable services after they are disabled/deleted from the cluster. Related-Bug: 1701485 Co-Authored-By: Damien Ciabrini <dciabrin@redhat.com> Change-Id: Ia597d240ea5834c50a8f6c4fac0b6ed417b8535c
2017-07-15Merge "Use a single configuration file for specifying docker containers."Jenkins7-101/+14
2017-07-14Use a single configuration file for specifying docker containers.Ian Main7-101/+14
This removes the default container names from all the templates and uses a single environment file to specify the full container name and registry from which to pull. Also does away with most of DockerNamespace. Change-Id: Ieaedac33f0a25a352ab432cdb00b5c888be4ba27 Depends-On: Ibc108871ebc2beb1baae437105b2da1d0123ba60 Co-Authored-By: Dan Prince <dprince@redhat.com> Co-Authored-By: Steve Baker <sbaker@redhat.com>
2017-07-14Adds network/cidr mapping into a new service propertyGiulio Fidente7-0/+35
Makes it possible to resolve network subnets within a service template; the data is transported into a new property ServiceData wired into every service which hopefully is generic enough to be extended in the future and transport more data. Data can be consumed in service templates to set config values which need to know what is the subnet where a deamon operates (for example the Ceph Public vs Cluster network). Change-Id: I28e21c46f1ef609517175f7e7ee19e28d1c0cba2
2017-07-12Merge "Bind mount needed cert for haproxy for HA too"Jenkins1-12/+26
2017-07-10Bind mount needed cert for haproxy for HA tooMartin André1-12/+26
haproxy needs the deployed SSL cert file to function when TLS is enabled. It is also required for the docker-puppet haproxy container since the haproxy puppet module uses a validate_cmd to check the generated config file is valid that fails when the required SSL cert is not present. There is no clean way to disable this feature [1] so we need to bind mount the cert into the container. This commit applies the same change that was applied in Id2df144b678769def204961236624091d4e5c457 for the non-ha case. [1] https://github.com/puppetlabs/puppetlabs-haproxy/blob/4753ea5b2506ee093e9b4c8af6e91201d476d426/manifests/config.pp#L53-L57 Change-Id: I93e1ee86197bcf271f18a62a27c2f350ed3966ea Co-Authored-By: Damien Ciabrini <dciabrin@redhat.com>
2017-07-10Copy only generated puppet files into the containerMartin André5-33/+30
This solves a problem with bind-mounts when the containers are holding files descriptors open. At the same time this makes the template more robust to puppet changes since new config files will be available in the containers without needing to update the templates. Partial-Bug: #1698323 Change-Id: Ia4ad6d77387e3dc354cd131c2f9756939fb8f736
2017-06-28Add heat parameter for all of config_volume imagesMartin André7-12/+45
This commit consistently defines a heat template parameter in the form of DockerXXXConfigImage where XXX represents the name of the config_volume that is used by docker-puppet. The goal is to mitigate hard to debug errors where the templates would set different defaults for the image docker-puppet.py uses to run, for the same config_volume name. This fixes a couple of inconsistencies on the way. Change-Id: I212020a76622a03521385a6cae4ce73e51ce5b6b Closes-Bug: #1699791
2017-06-26Merge "Containerize Cinder-backup for HA"Jenkins1-0/+152
2017-06-15Merge "Containerize Cinder-volume for HA"Jenkins1-0/+170
2017-06-12Generate HAproxy iptables rules for containerized HA deploymentsDamien Ciabrini1-10/+13
The containerized HAproxy service can only specify steps to be run in containers, i.e. it cannot runs the regular puppet steps on bare metal at the same time. A side effect is that the dedicated HAproxy iptables rules are no longer generated. Update the docker_config step to fix the creation of iptables rules for HAproxy and persist them on-disk as before. Co-Authored-By: Michele Baldessari <michele@acksyn.org> Closes-Bug: 1697387 Change-Id: Ib5a083ba3299a82645f1a0f9da0d482c6b89ee23
2017-06-08Containerize Cinder-volume for HADamien Ciabrini1-0/+170
This service allows configuring and deploying cinder-volume containers in a HA overcloud managed by pacemaker. The containers are managed and run by pacemaker. Pacemaker runs the standard Kolla image but overrides the initial command so that it explicitely calls cinder-volume. This way, we shield ourselves from any unexpected future change in Kolla. This container needs to use the 'docker_config' section to invoke puppet (as opposed to 'docker_puppet_tasks'), because due to the HA composability each resource creation needs to happen on the bootstrap node of that service and 'docker_puppet_tasks' will only run on the controller/primary role. Co-Authored-By: Michele Baldessari <michele@acksyn.org> Partial-Bug: #1668920 Depends-On: I95ad4dd89b47396bea672813d87de35e64c04b2d Change-Id: Ib6396219c3d9484c533f6f9995d565091a197bbb
2017-06-08Containerize Cinder-backup for HADamien Ciabrini1-0/+152
This service allows configuring and deploying cinder-backup containers in a HA overcloud managed by pacemaker. The containers are managed and run by pacemaker. Pacemaker runs the standard Kolla image but overrides the initial command so that it explicitely calls cinder-backup. This way, we shield ourselves from any unexpected future change in Kolla. This container needs to use the 'docker_config' section to invoke puppet (as opposed to 'docker_puppet_tasks'), because due to the HA composability each resource creation needs to happen on the bootstrap node of that service and 'docker_puppet_tasks' will only run on the controller/primary role. Co-Authored-By: Michele Baldessari <michele@acksyn.org> Partial-Bug: #1668920 Depends-On: If53495ff75d4832cc6be80dc0dc9bd540ab6583b Change-Id: Ieec823e10667592bd775bb2642f0c3790a83e85f
2017-06-04Merge "Containerize Redis for HA"Jenkins1-0/+140
2017-06-04Containerize Redis for HADamien1-0/+140
This service allows configuring and deploying Redis containers in a HA overcloud managed by pacemaker. The containers are managed and run by pacemaker. Inside there is pacemaker_remote which will invoke the resource agent managing galera. The resources themselves are created via puppet-pacemaker inside a short-lived container used for this purpose (mysql_init_bundle). This container needs to use the 'docker_config' section to invoke puppet (as opposed to 'docker_puppet_tasks'), because due to the HA composability each resource creation needs to happen on the bootstrap node of that service and 'docker_puppet_tasks' will only run on the controller/primary role. Co-Authored-By: Michele Baldessari <michele@acksyn.org> Closes-Bug: #1692924 Depends-On: Ia1131611d15670190b7b6654f72e6290bf7f8b9e Change-Id: Ie045954fcc86ef2b3e4562b6f012853177f03948
2017-06-03Merge "Containerize clustercheck galera monitor for HA deployments"Jenkins1-0/+103
2017-06-03Merge "Containerize HAProxy for HA"Jenkins1-0/+116
2017-06-03Merge "Containerize RabbitMQ for HA"Jenkins1-0/+159
2017-05-24Containerize RabbitMQ for HADan Prince1-0/+159
This service allows configuring and deploying RabbitMQ containers in a HA overcloud managed by pacemaker. The containers are managed and run by pacemaker. Inside there is pacemaker_remote which will invoke the resource agent managing galera. The resources themselves are created via puppet-pacemaker inside a short-lived container used for this purpose (mysql_init_bundle). This container needs to use the 'docker_config' section to invoke puppet (as opposed to 'docker_puppet_tasks'), because due to the HA composability each resource creation needs to happen on the bootstrap node of that service and 'docker_puppet_tasks' will only run on the controller/primary role. Co-Authored-By: Michele Baldessari <michele@acksyn.org> Co-Authored-By: John Eckersberg <jeckersb@redhat.com> Closes-Bug: #1692909 Depends-On: I0722e4a4d4716f477e8304cfa1aadd3eef7c2f31 Change-Id: I942737134385af775cade40c2d69516d4fe31a99
2017-05-24Containerize MySQL for HADamien Ciabrini1-0/+180
This service allows configuring and deploying MySQL/galera containers in a HA overcloud managed by pacemaker. The containers are managed and run by pacemaker. Inside there is pacemaker_remote which will invoke the resource agent managing galera. The resources themselves are created via puppet-pacemaker inside a short-lived container used for this purpose (mysql_init_bundle). This container needs to use the 'docker_config' section to invoke puppet (as opposed to 'docker_puppet_tasks'), because due to the HA composability each resource creation needs to happen on the bootstrap node of that service and 'docker_puppet_tasks' will only run on the controller/primary role. Co-Authored-By: Michele Baldessari <michele@acksyn.org> Closes-Bug: #1692842 Depends-On: I3b4d8ad2eec70080419882d5d822f78ebd3721ae Change-Id: I790dbc30b3de1c1a3fe76d3d8f060e4d7f95e2e7
2017-05-24Containerize HAProxy for HADamien Ciabrini1-0/+116
This service allows configuring and deploying HAProxy containers in a HA overcloud managed by pacemaker. The containers are managed and run by pacemaker. Pacemaker runs the standard Kolla image but overrides the initial command so that it explicitely calls HAProxy. This way, we shield ourselves from any unexpected future change in Kolla. This container needs to use the 'docker_config' section to invoke puppet (as opposed to 'docker_puppet_tasks'), because due to the HA composability each resource creation needs to happen on the bootstrap node of that service and 'docker_puppet_tasks' will only run on the controller/primary role. Co-Authored-By: Michele Baldessari <michele@acksyn.org> Closes-Bug: #1692908 Depends-On: Ifcf890a88ef003d3ab754cb677cbf34ba8db9312 Change-Id: I2f679bfe195733f4507e9b9e920b678e1370bb82
2017-05-23Containerize clustercheck galera monitor for HA deploymentsDamien Ciabrini1-0/+103
In HA overcloud deployments, HAProxy makes use of a helper service called "clustercheck", to check whether galera nodes are available for serving traffic. This change implements a dedicated service for clustercheck, which was originally part of the pacemaker mysql service. The service is configured by tripleo and the container's lifecycle is managed by docker, like other containerized services. Closes-Bug: #1692969 Change-Id: I8a5b30429f8ec3e484256a62a29ab7dee33ab291 Co-Authored-By: Michele Baldessari <michele@acksyn.org> Depends-on: I1aabe34fa6a9c8c705a4405f275b66502c313cf2