aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorAlexandru Avadanii <Alexandru.Avadanii@enea.com>2017-09-23 20:48:59 +0200
committerAlexandru Avadanii <Alexandru.Avadanii@enea.com>2017-09-23 21:27:05 +0200
commita6feb158fa5bc16b8d496c785c461c73ca715322 (patch)
tree1f08ad5800b2697dedf8842b95d8f5bf381f3a7e
parentee4d89f2803085bdd4a7f3b365e40ec9366c25a9 (diff)
Workaround VCP minions timeout post pkg.upgrade
pkg.upgrade was enabled for all salt minions, including VCP VMs, which take longer to perform the operation, probably due to an older set of packages in the Ubuntu disk image we use. One way to work around this is to switch to UCA Xenial image, and let Salt pre-provising salt minion on it, but that adds deploy time delay and has caused issues in the past (should be ok now). Alternatively, we can retry the pkg.upgrade until all minions respond, before moving on with the state execution. This prevents silently skipping the next salt calls (e.g. installing keepalived). Note that the issue did not manifest for OVD-DPDK, where after pkg.upgrade, DPDK is installed, giving VCP VMs enough time to return. While at it, retry 'salt.control' state apply too (non-critical, but it fails every once in a while). Fixes: 87310fb Change-Id: I97acc2b23206a55d72f7e6583ca42127fdbacc16 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
-rwxr-xr-xmcp/config/states/maas6
1 files changed, 4 insertions, 2 deletions
diff --git a/mcp/config/states/maas b/mcp/config/states/maas
index e1ffa8733..de9d699b9 100755
--- a/mcp/config/states/maas
+++ b/mcp/config/states/maas
@@ -91,7 +91,8 @@ salt -C 'kvm*' state.sls libvirt
salt -C '* and not cfg01* and not mas01*' state.apply salt
salt -C 'kvm*' saltutil.sync_all
-salt -C 'kvm*' state.sls salt.control
+wait_for 10 "! salt -C 'kvm*' state.sls salt.control | " \
+ "tee /dev/stderr | fgrep -q 'Not connected'"
vcp_nodes=$(salt --out yaml 'kvm01*' pillar.get salt:control:cluster:internal:node | \
awk '/\s+\w+:$/ {gsub(/:$/, "*"); print $1}')
@@ -115,4 +116,5 @@ wait_for 10 "salt -C 'E@^(?!cfg01|mas01|kvm|cmp00).*' ssh.set_auth_key ${SUDO_US
$(awk 'NR==1{print $2}' "$(eval echo "~${SUDO_USER}/.ssh/authorized_keys")")"
# Get the latest packages
-salt '*' pkg.upgrade refresh=False
+wait_for 10 "! salt '*' pkg.upgrade refresh=False | " \
+ "tee /dev/stderr | fgrep -q 'Not connected'"