aboutsummaryrefslogtreecommitdiffstats
path: root/mcp
diff options
context:
space:
mode:
authorAlexandru Avadanii <Alexandru.Avadanii@enea.com>2017-09-23 20:48:59 +0200
committerAlexandru Avadanii <Alexandru.Avadanii@enea.com>2017-09-23 21:57:55 +0000
commitd1d5c106c09790b85e146874122709e876769c12 (patch)
tree7381a2a3aed72d17e69c174f356b6d21e04ef191 /mcp
parent72d9201cb025e3c76df62892d03830863c2d751e (diff)
Workaround VCP minions timeout post pkg.upgrade
pkg.upgrade was enabled for all salt minions, including VCP VMs, which take longer to perform the operation, probably due to an older set of packages in the Ubuntu disk image we use. One way to work around this is to switch to UCA Xenial image, and let Salt pre-provising salt minion on it, but that adds deploy time delay and has caused issues in the past (should be ok now). Alternatively, we can retry the pkg.upgrade until all minions respond, before moving on with the state execution. This prevents silently skipping the next salt calls (e.g. installing keepalived). Note that the issue did not manifest for OVD-DPDK, where after pkg.upgrade, DPDK is installed, giving VCP VMs enough time to return. While at it, retry 'salt.control' state apply too (non-critical, but it fails every once in a while). Fixes: 87310fb Change-Id: I97acc2b23206a55d72f7e6583ca42127fdbacc16 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com> (cherry picked from commit a6feb158fa5bc16b8d496c785c461c73ca715322)
Diffstat (limited to 'mcp')
-rwxr-xr-xmcp/config/states/maas6
1 files changed, 4 insertions, 2 deletions
diff --git a/mcp/config/states/maas b/mcp/config/states/maas
index e1ffa8733..de9d699b9 100755
--- a/mcp/config/states/maas
+++ b/mcp/config/states/maas
@@ -91,7 +91,8 @@ salt -C 'kvm*' state.sls libvirt
salt -C '* and not cfg01* and not mas01*' state.apply salt
salt -C 'kvm*' saltutil.sync_all
-salt -C 'kvm*' state.sls salt.control
+wait_for 10 "! salt -C 'kvm*' state.sls salt.control | " \
+ "tee /dev/stderr | fgrep -q 'Not connected'"
vcp_nodes=$(salt --out yaml 'kvm01*' pillar.get salt:control:cluster:internal:node | \
awk '/\s+\w+:$/ {gsub(/:$/, "*"); print $1}')
@@ -115,4 +116,5 @@ wait_for 10 "salt -C 'E@^(?!cfg01|mas01|kvm|cmp00).*' ssh.set_auth_key ${SUDO_US
$(awk 'NR==1{print $2}' "$(eval echo "~${SUDO_USER}/.ssh/authorized_keys")")"
# Get the latest packages
-salt '*' pkg.upgrade refresh=False
+wait_for 10 "! salt '*' pkg.upgrade refresh=False | " \
+ "tee /dev/stderr | fgrep -q 'Not connected'"