diff options
| author |   Alexandru Avadanii <Alexandru.Avadanii@enea.com> | 2017-09-23 20:48:59 +0200 |
|---|---|---|
| committer | Alexandru Avadanii <Alexandru.Avadanii@enea.com> | 2017-09-23 21:57:55 +0000 |
| commit | d1d5c106c09790b85e146874122709e876769c12 (patch) | |
| tree | 7381a2a3aed72d17e69c174f356b6d21e04ef191 /mcp | |
| parent | 72d9201cb025e3c76df62892d03830863c2d751e (diff) | |
Workaround VCP minions timeout post pkg.upgrade
pkg.upgrade was enabled for all salt minions, including VCP VMs,
which take longer to perform the operation, probably due to an
older set of packages in the Ubuntu disk image we use.
One way to work around this is to switch to UCA Xenial image, and
let Salt pre-provising salt minion on it, but that adds deploy
time delay and has caused issues in the past (should be ok now).
Alternatively, we can retry the pkg.upgrade until all minions
respond, before moving on with the state execution. This prevents
silently skipping the next salt calls (e.g. installing keepalived).
Note that the issue did not manifest for OVD-DPDK, where after
pkg.upgrade, DPDK is installed, giving VCP VMs enough time to return.
While at it, retry 'salt.control' state apply too (non-critical,
but it fails every once in a while).
Fixes: 87310fb
Change-Id: I97acc2b23206a55d72f7e6583ca42127fdbacc16
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
(cherry picked from commit a6feb158fa5bc16b8d496c785c461c73ca715322)
Diffstat (limited to 'mcp')
| -rwxr-xr-x | mcp/config/states/maas | 6 |
1 files changed, 4 insertions, 2 deletions
diff --git a/mcp/config/states/maas b/mcp/config/states/maas index e1ffa8733..de9d699b9 100755 --- a/mcp/config/states/maas +++ b/mcp/config/states/maas @@ -91,7 +91,8 @@ salt -C 'kvm*' state.sls libvirt salt -C '* and not cfg01* and not mas01*' state.apply salt salt -C 'kvm*' saltutil.sync_all -salt -C 'kvm*' state.sls salt.control +wait_for 10 "! salt -C 'kvm*' state.sls salt.control | " \ + "tee /dev/stderr | fgrep -q 'Not connected'" vcp_nodes=$(salt --out yaml 'kvm01*' pillar.get salt:control:cluster:internal:node | \ awk '/\s+\w+:$/ {gsub(/:$/, "*"); print $1}') @@ -115,4 +116,5 @@ wait_for 10 "salt -C 'E@^(?!cfg01|mas01|kvm|cmp00).*' ssh.set_auth_key ${SUDO_US $(awk 'NR==1{print $2}' "$(eval echo "~${SUDO_USER}/.ssh/authorized_keys")")" # Get the latest packages -salt '*' pkg.upgrade refresh=False +wait_for 10 "! salt '*' pkg.upgrade refresh=False | " \ + "tee /dev/stderr | fgrep -q 'Not connected'" |