summaryrefslogtreecommitdiffstats
path: root/extraconfig/tasks
AgeCommit message (Collapse)AuthorFilesLines
2016-08-26M/N upgrade set scheduler_host_manager right.Sofer Athlan-Guyot1-0/+2
scheduler_host_manager doesn't take nova.scheduler.host_manager.HostManager as a value anymore. This fix it before restarting the service. Change-Id: Ia9adcfd5a898f0c712b4a37ae33db88a44630f0d Closes-Bug: 1615035
2016-08-17Upgrade scripts to migrate aodh alarm dataPradeep Kilambi2-0/+52
In Newton, Aodh will be using its own mysql DB rather than using ceilometer's mongo instance. This means we need to migrate any existing alarm and alrm history data from ceilometer DB to aodh mysqlDB. Upstream aodh provides us with a aodh-data-migration utility. We need to invoke this during the mitaka->newton upgrade procedure so data is migrated as expected and aodh mysql backend takes over. Closes-bug: #1611794 Change-Id: I17888b57ecf98cd83e92af2f9cdbead066b03aa3
2016-07-28Merge "Allow to manually disable post-puppet restarts"Jenkins2-13/+30
2016-07-04Merge "Dump and restore galera db during major upgrades"Jenkins3-0/+184
2016-06-29Dump and restore galera db during major upgradesMichele Baldessari3-0/+184
When the overcloud is upgraded we do a yum update of the packages. This step might introduce a newer galera version. In such a situation we need to dump the db and restore it. The high-level workflow should be the following: 1) During the main upgrade step, before shutting down the cluster we need to dump the db 2) We upgrade the packages 3) We briefly start mysql on a single node while making sure that /root/.my.cnf is briefly moved out of the way (because it contains a password) and import the data. After the import we shutdown this mysql instance 4) We let the cluster start up normally The above steps will take place in the following scenarios. Given a locally installed mariadb version X.Y.Z and release R, we will dump and restore the DB under the following conditions: A) MySqlMajorUpgrade template parameter is set to 'auto' and the upgraded package differs in X, Y *or* Z. We basically don't dump automatically if the release field changes. B) MySqlMajorUpgrade template parameter is set to 'yes' When MySqlMajorUpgrade is set to 'no', no dumping will be performed. Note that this will give a non functional upgrade if a major mariadb upgrade is taking place. Partial-Bug: #1587449 Co-Author: Damien Ciabrin <dciabrin@redhat.com> Co-Author: Mike Bayer <mbayer@redhat.com> Depends-On: I8cb4cb3193e6b823aad48ad7dbbbb227364d2a58 Depends-On: I38dcacfabc44539aab1f7da85168fe44a1b43a51 Change-Id: I374628547aed091129d0deaa29764bfc998d76ea
2016-06-29Increase cluster sync timeout for M->N major upgradesDamien Ciabrini1-1/+1
Since the Liberty release, the number of services managed by pacemaker on HA Overcloud has increased. This has an impact on major_upgrade_controller_pacemaker_1.sh, where cluster sync timeout value tuned for older releases is now becoming too low. Raise the cluster sync timeout value to a sensible limit to give pacemaker enough time to stop the cluster during major upgrade. Change-Id: I821d354ba30ce39134982ba12a82c429faa3ce62 Closes-Bug: #1597506
2016-06-24Disable stonith temporarily during upgradesMichele Baldessari1-0/+14
It is best if we disable stonith if a cluster has it configured and on, before we call "pcs cluster stop --all", because should a service fail to stop for whatever reason, pacemaker will fence the node where it happened. This is something that we unlikely want during an upgrade as it will make things worse. Once the cluster is stopped we can reenable stonith (if it was enabled to start with) in the CIB while the cluster is shut down. Closes-Bug: #1596065 Change-Id: I38dcacfabc44539aab1f7da85168fe44a1b43a51
2016-06-14Allow to manually disable post-puppet restartsJiri Stransky2-13/+30
Restarting services after Puppet is vital to ensure that config changes go applied. However, it can be sometimes desirable to prevent these restarts to avoid downtime, if the operator is sure that no config changes need applying. This can be a case e.g. when scaling compute nodes. Passing the puppet-pacemaker-no-restart.yaml environment file *in addition* to puppet-pacemaker.yaml should allow this. This is a stop gap solution before we have proper communication between Puppet and Pacemaker to allow selective restarts. Change-Id: I9c3c5c10ed6ecd5489a59d7e320c3c69af9e19f4
2016-05-04Merge "Disable VIPs before stopping cluster during version upgrade"Jenkins2-0/+11
2016-05-04Merge "Fix distinguishing between stack-create and stack-update"Jenkins1-1/+1
2016-05-02Disable VIPs before stopping cluster during version upgradeIan Pilcher2-0/+11
If "pcs cluster stop --all" is executed on a controller that happens to have a VIP on the internal network, pcs may use the VIP as the source address for communication with another cluster node. When pacemaker is stopped this VIP goes away, and pcs never receives a response from the other node. This causes pcs to hang indefinitely; eventually the upgrade times out and fails. Disabling the VIPs before stopping the cluster avoids this situation. Change-Id: I6bc59120211af28456018640033ce3763c373bbb Closes-Bug: 1577570
2016-04-26Merge "Update .sh references from openstack-keystone to openstack-core"Jenkins4-115/+8
2016-04-20Merge "Increase galera sync timeout in yum_update.sh"Jenkins1-1/+1
2016-04-13Make sure openstack services are dependent on openstack-coreJiri Stransky1-0/+36
Previously ceilometer-notification, aodh-listener and sahara-engine didn't have constraints that would anchor them under openstack-core dummy resource. Such constraints are added now. (sahara-engine starting after sahara-api, aodh-listener after aodh-evaluator, and ceilometer-notification after openstack-core.) Openstack-core -> heat-api constraint has been removed because heat-api depends on ceilometer-notification, so there's a transitive dependency on openstack-core already. Change-Id: Ided7321ebbf2c3556726343b4bb466fd8759b43a Closes-Bug: #1569444
2016-04-11Fix distinguishing between stack-create and stack-updateJiri Stransky1-1/+1
Previously we tried to use UpdateIdentifier for two different things: tell whether to perform package update, and also to tell whether the top-level stack is being created or updated (which was incorrect and resulted in bug 1567384, and an attempt to work around that bug resulted in bug 1567385). We cannot use Heat's "action" conditionals in some cases, because they refer to the direct parent stack, which can yield undesirable results when introducing new nested stacks or temporarily no-opping something and then adding it back (in both these cases, "action" would be considered "CREATE", even though the top-level stack is in "UPDATE"). So tripleoclient passes a new parameter StackAction to tell whether the top-level stack is being created or updated, and we make use of that. (It seems there's no better way of getting this info from within the nested Heat stacks.) Change-Id: Ie14ddbff15e7ed21aaa3fcdacf36e0040f912382 Depends-On: I9dc3b4cd8a6a71df34d8babf0e4c6505041f5311 Closes-Bug: #1567384 Related-Bug: #1567385
2016-04-11Update .sh references from openstack-keystone to openstack-coreGiulio Fidente4-115/+8
The update and upgrade shell scripts were still referencing the old openstack-keystone service which got removed with Ie26908ac9bfc0b84b6b65ae3bda711236b03d9d4 Also removes kilo and liberty specific workarounds and config changes. Change-Id: Icc80904908ee3558930d4639a21812f14b2fd12e
2016-04-08Replace extraconfig/tasks/noop.yaml w/ Heat::NoneDan Prince1-26/+0
Removes the old noop nested stack template for extraconfig tasks and instead uses OS::Heat::None. This should avoid a few extra resource checks on create and update. Change-Id: I5a42fc78ece2553e86385236e214aa1e3c91cd85
2016-04-07Add removal of the /etc/resolv.conf.save file for +bug/1567004marios1-0/+3
The change at https://review.openstack.org/#/c/302352/ should stop the if up/down scripts from making changes to resolv.conf as discussed in that review and the related bug below. However during upgrades, as we are moving from a version of the ifcfg-vlanXX files that don't have the PEERDNS=no added by /#/c/302352 the if up script will restore the /etc/resolv.conf.save to /etc/resolv.conf and overwrite it. This removes the .save file during the upgrade init command which gets delivered to all nodes as the first stage of a major upgrade. Change-Id: I91dd139f43be4912c20d8661691bee2b662964d4 Related-Bug: 1567004
2016-04-04Filter for local nodes in check_resource functionRaoul Scarazzini1-1/+2
While having extra customizations inside a TripleO deployed Pacemaker environment, say you have instance HA with pacemaker_remoted or you need to configure an external arbitrator for something, then the status of the resources for remote nodes is "Stopped". This leads to failures while, for example, scaling up. This fixes the way status is checked, filtering just local nodes. Co-Authored-By: Giulio Fidente <gfidente@redhat.com> Change-Id: I8dc25f5d7031c265858afd5a266fda5315ae37a0
2016-04-01Restart haproxy after configuring SSL certsBen Nemec2-7/+21
If a certificate expires, the user will need to update it. However, because we only restart services at the end of a stack-update the new certificate doesn't take effect until after puppet has run. This is a problem because puppet makes OpenStack calls, which will fail if the certificate is expired. In that case we never get to the service restart so the stack is wedged until the user manually restart haproxy. This patch addresses the problem by reloading haproxy before puppet runs. This is done in a pre-puppet script for pacemaker after pacemaker is maintenance mode because we need to make sure it happens after all of the certs have been installed on the controllers, but before puppet runs. For non-pacemaker, haproxy is simply reloaded. Change-Id: Id5ed05b3a20d06af8ae7a3d6f859b03399b0d77d
2016-03-24Set UpdateIdentifier for upgrade converge, to prevent services downMathieu Bultel1-6/+0
We'd like to let the post puppet pacemaker controller services restart to happen for the convergence step so set the UpdateIdentifier. However also set the PackageUpdate to noop so the yum_update.sh doesn't happen. Since a full haproxy restart is expected, we no longer need the systemctl reload added at Iae3bad745ecdf952a7a0314fe1375d07eb47c454 so remove that too. Some more context at https://bugzilla.redhat.com/show_bug.cgi?id=1321036 Co-Authored-By: marios <marios@redhat.com> Change-Id: I31c2d97d68c97b435f63863fae2c89f18f99681d
2016-03-24Merge "Add systemctl reload haproxy to the pacemaker_resource_restart.sh"Jenkins1-0/+6
2016-03-24Merge "Deploy Aodh services, replacing Ceilometer Alarm"Jenkins2-2/+31
2016-03-23Add systemctl reload haproxy to the pacemaker_resource_restart.shmarios1-0/+6
As discussed in the related bug below, after upgrading your environment to latest liberty the haproxy config isn't picked up. This adds a systemctl reload haproxy in the pacemaker resource restart we run as part of the post-puppet-pacemaker. Related-Bug: 1561012 Change-Id: Iae3bad745ecdf952a7a0314fe1375d07eb47c454
2016-03-20Deploy Aodh services, replacing Ceilometer AlarmPradeep Kilambi2-2/+31
Ceilometer Alarm is deprecated in Liberty by Aodh. This patch: * manage Aodh Keystone resources * deploy Aodh API under WSGI, Notifier, Listener and Evaluator * manage new parameters to customize Aodh deployment * uses ceilometer DB for the upgrade path * pacemaker config * Add migration logic to remove pcs resources Depends-On: I5333faa72e52d2aa2a622ac2d4b60825aadc52b5 Depends-On: Ib6c9c4c35da3fb55e0ca8e2d5a58ebaf4204d792 Co-Authored-By: Emilien Macchi <emilien@redhat.com> Change-Id: Ib47a22884afb032ebc1655e1a4a06bfe70249134
2016-03-11Merge "Upgrades: quiet yum upgrade on cinder nodes"Jenkins1-1/+1
2016-03-10Merge "Upgrades: initialization command/snippet"Jenkins1-0/+52
2016-03-10Merge "Add a ceph-storage node upgrade script for the upgrade workflow"Jenkins2-4/+50
2016-03-10Merge "Upgrades: object storage node upgrade fix"Jenkins1-3/+4
2016-03-10Upgrades: quiet yum upgrade on cinder nodesJiri Stransky1-1/+1
Yum update on cinder nodes should be quiet, as it is on controllers, because results of these updates are sent to Heat. I mistakenly left this out in the first patch because i used one of the standalone node upgrade scripts as a copy/paste base for the cinder node upgrade script. Change-Id: Id13190dc4d242317829c7994088183f52d21461d
2016-03-10Merge "Upgrade of Cinder block storage nodes"Jenkins2-1/+23
2016-03-09Upgrades: object storage node upgrade fixJiri Stransky1-3/+4
The variables in the heredoc should be escaped because they should evaluate only when the inner script runs, not when the outer "writer" script runs. Python-zaqarclient is installed for os-collect-config to work, as we do on the other node types. Swift-proxy is removed from list of services to stop/start, as swift-proxy isn't supposed to run on the swift storage nodes. Change-Id: I8426b859d11378ebdc3da94dcc090133dab0c628
2016-03-09Fixup systemctl_swift stop/start during the controller upgrademarios1-4/+17
During the controller upgrade in major_upgrade_controller_pacemaker_1.sh we use systemctl to stop all swift services and then start them again in _pacemaker_2.sh In the case of stand-alone swift nodes the deployer may have used the ControllerEnableSwiftStorage: false so that only the swift-proxy service is left on controllers (wrt swift). The systemctl_swift function used during upgrades is changed to factor this in. Change-Id: Ib22005123429f250324df389855d0dccd2343feb
2016-03-09Upgrades: initialization command/snippetJiri Stransky1-0/+52
This allows to run a command or a script snippet on all overcloud nodes at the beginning of the upgrade. The intended use is to switch to a new set of repositories on the overcloud. This is done differently in different contexts (e.g. upstream vs. downstream), but generally it should be simple enough to not warrant creation of switchable "UpgradeInit" resource in the resource registry, and a string command/snippet parameter should suffice. Change-Id: I72271170d3f53a5179b3212ec9bae9a6204e29e6
2016-03-08Add a ceph-storage node upgrade script for the upgrade workflowmarios2-4/+50
This adds delivery of an upgrade script to any ceph-storage nodes during the script delivery that comes first during the upgrade workflow. The controllers have the ceph-mon whilst the ceph-osds are on the ceph-storage nodes. The ceph-mons will be updated first as part of the heat-driven controller upgrade, and ceph-osds on ceph nodes are upgraded with the upgrade-non-controller.sh tripleo-common script as with compute and swift nodes. Also slight rename for the ObjectStorageConfig/Deployment here for consistency. Change-Id: I12abad5548dcb019ade9273da06fe66fd97f54cc
2016-03-07Merge "Revert "Deploy Aodh services, replacing Ceilometer Alarm""Jenkins1-7/+2
2016-03-07Merge "Function library for major upgrades"Jenkins2-0/+16
2016-03-07Merge "Introduce a UpgradeScriptDeliveryWorfklow as part of tripleo upgrades"Jenkins3-25/+103
2016-03-04Revert "Deploy Aodh services, replacing Ceilometer Alarm"James Slagle1-7/+2
This just a revert to see if reverting this gets back to a normal CI run time. This reverts commit f72aed85594f223b6f888e6d0af3c880ea581a66. Change-Id: I04a0893f6cf69f547a4db26261005e580e1fc90b
2016-03-03Merge "Deploy Aodh services, replacing Ceilometer Alarm"Jenkins1-2/+7
2016-03-03Deploy Aodh services, replacing Ceilometer AlarmEmilien Macchi1-2/+7
Ceilometer Alarm is deprecated in Liberty by Aodh. This patch: * manage Aodh Keystone resources * deploy Aodh API under WSGI, Notifier, Listener and Evaluator * manage new parameters to customize Aodh deployment * uses ceilometer DB for the upgrade path * pacemaker config Depends-On: I9e34485285829884d9c954b804e3bdd5d6e31635 Depends-On: I891985da9248a88c6ce2df1dd186881f582605ee Depends-On: Ied8ba5985f43a5c5b3be5b35a091aef6ed86572f Co-Authored-By: Pradeep Kilambi <pkilambi@redhat.com> Change-Id: I58d419173e80d2462accf7324c987c71420fd5f6
2016-03-03Function library for major upgradesJiri Stransky2-0/+16
This commit introduces a bash file to be sourced into major upgrade scripts. Into this file we can put specific pieces of migration logic in the form of bash functions, which can then be called from the upgrade scripts. Change-Id: Ibf7aa84d3880e9218c488dec9d707300e1784744
2016-03-03Introduce a UpgradeScriptDeliveryWorfklow as part of tripleo upgradesmarios3-25/+103
This splits the upgrade script delivery out of the UpgradeWorkflow and into a new task which delivers the upgrade script for compute and object-storage nodes. This is intended to be the first part of the upgrades process, since we need to upgrade swift nodes before the controllers and then only one at a time. So this will deliver the upgrade script which can be invoked by the operator using the existing script in tripleo-common 'upgrade-non-controller.sh'. This can be invoked by passing the -e environments/major-upgrade-script-delivery.yaml (added here) to the openstack overcloud deploy command. Change-Id: I20a0d4978e907111404f8108c502ab53b69a3296
2016-03-03Upgrade of Cinder block storage nodesJiri Stransky2-1/+23
This introduces upgrades for Cinder block storage nodes. Currently Cinder doesn't support upgrade level pinning and cannot safely deal with version skew. This means that we have to upgrade Cinder storage nodes in sync with controller nodes (after they were taken down for upgrade, before they are brought back up) to ensure that Cinder services perform AMQP communication only within the same major version of Cinder. According to our current knowledge, Cinder block storage nodes are the only node type that will have to be upgraded in sync with controllers. Change-Id: Icec913c015eff744b0f31b513176b4b657df43af
2016-03-02Moves the swift start/stop into the common_functions.sh filemarios3-10/+11
Since swift isn't managed by pacemaker we need to manually (systemctl) stop and start the swift services. This moves the duplicate blocks for start/stop into a common function (we already include that pacemaker_common_functions.sh here so may as well) Change-Id: Ic4f23212594c1bf9edc39143bf60c7f6d648fd1d
2016-03-02Upgrades: install zaqarclientJiri Stransky2-0/+3
Old overcloud images don't have python-zaqarclient installed, and new overclouds' os-collect-config are configured with Zaqar support. This together means that on upgrade we need to install python-zaqarclient, otherwise os-collect-config will be restarted during yum update and crash due to trying to import missing Python module from zaqarclient. Change-Id: I3e875e14cb60b1b78aec0d9ddc412ccf865abd01
2016-03-02Upgrades: quiet yum updateJiri Stransky1-1/+1
Quiet down yum during major upgrades to reduce the output size. This is consistent with what was introduced into minor updates in change I517271e8465885421a78b73c5af756816c37a977. Change-Id: Ie6b470e383fdf42870ac6f60ca43e44b4c446ebe
2016-02-29Merge "Write the compute upgrade script for tripleo major upgrade workflow"Jenkins2-0/+50
2016-02-26Merge "Add meta notify=true to rabbitmq resource"Jenkins1-0/+3
2016-02-26Write the compute upgrade script for tripleo major upgrade workflowmarios2-0/+50
As part of the major upgrade workflow non-controller nodes are to be updated by the operator, out-of-band and only after an initial heat stack-update that invokes the upgrade of the controller nodes. This review adds a ComputeDeliverUpgradeConfigDeployment_Step3 SoftwareDeploymentGroup to be applied only to compute nodes, and that depends on the controllers having been upgraded after ControllerPacemakerUpgradeConfig_Step2. Its purpose is to deliver but not invoke the upgrade script on compute nodes to /root/tripleo_upgrade_node.sh . The non-controller nodes will then be upgraded later by an operator that will run the script provided for that purpose, like at https://review.openstack.org/#/c/284722/1 for example. Change-Id: Ic6115fc8cf5320abfcf500112ff563bde8b88661