From a6feb158fa5bc16b8d496c785c461c73ca715322 Mon Sep 17 00:00:00 2001 From: Alexandru Avadanii Date: Sat, 23 Sep 2017 20:48:59 +0200 Subject: Workaround VCP minions timeout post pkg.upgrade pkg.upgrade was enabled for all salt minions, including VCP VMs, which take longer to perform the operation, probably due to an older set of packages in the Ubuntu disk image we use. One way to work around this is to switch to UCA Xenial image, and let Salt pre-provising salt minion on it, but that adds deploy time delay and has caused issues in the past (should be ok now). Alternatively, we can retry the pkg.upgrade until all minions respond, before moving on with the state execution. This prevents silently skipping the next salt calls (e.g. installing keepalived). Note that the issue did not manifest for OVD-DPDK, where after pkg.upgrade, DPDK is installed, giving VCP VMs enough time to return. While at it, retry 'salt.control' state apply too (non-critical, but it fails every once in a while). Fixes: 87310fb Change-Id: I97acc2b23206a55d72f7e6583ca42127fdbacc16 Signed-off-by: Alexandru Avadanii --- mcp/config/states/maas | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) (limited to 'mcp/config/states') diff --git a/mcp/config/states/maas b/mcp/config/states/maas index e1ffa8733..de9d699b9 100755 --- a/mcp/config/states/maas +++ b/mcp/config/states/maas @@ -91,7 +91,8 @@ salt -C 'kvm*' state.sls libvirt salt -C '* and not cfg01* and not mas01*' state.apply salt salt -C 'kvm*' saltutil.sync_all -salt -C 'kvm*' state.sls salt.control +wait_for 10 "! salt -C 'kvm*' state.sls salt.control | " \ + "tee /dev/stderr | fgrep -q 'Not connected'" vcp_nodes=$(salt --out yaml 'kvm01*' pillar.get salt:control:cluster:internal:node | \ awk '/\s+\w+:$/ {gsub(/:$/, "*"); print $1}') @@ -115,4 +116,5 @@ wait_for 10 "salt -C 'E@^(?!cfg01|mas01|kvm|cmp00).*' ssh.set_auth_key ${SUDO_US $(awk 'NR==1{print $2}' "$(eval echo "~${SUDO_USER}/.ssh/authorized_keys")")" # Get the latest packages -salt '*' pkg.upgrade refresh=False +wait_for 10 "! salt '*' pkg.upgrade refresh=False | " \ + "tee /dev/stderr | fgrep -q 'Not connected'" -- cgit 1.2.3-korg