1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
|
From: Josep Puigdemont <josep.puigdemont@enea.com>
Date: Wed, 4 May 2016 14:27:23 +0200
Subject: [PATCH] deployment.py: stdout not consumed when deploying changes
During the automatic deployment, when the environment is ready to be
deployed, the deploy.py script will spawn a shell process that will
perform the command "fuel deploy-changes". The standard output of this
process is then piped to a "tee" process, which redirects the output
to the standard output of the shell process, and to a file named
cloud.log. The file is monitored by the deploy script to find out the
status of the deployment, and print it to the log file of the automatic
deployment script, including percentages for each node being
provisioned. However, the deploy script never consumes the standard
output of the shell process. If the shell process produces enough
output, its standard output buffer will fill up, thus making the tee
process block trying to write to its standard output, and the cloud.log
file will not be updated. At this point, the deploy process, which is
monitoring cloud.log, will not detect any progress in the deployment,
and eventually it will time out and assume the deployment failed,
although it might have finished fine after that.
The solution here is to remove the "tee" process from the shell command,
and instead redirect standard output to the cloud.log file.
Another solution would be to actually parse the standard output of the
shell command from the deploy script itself, but that would require a
bit more work, as reading a line at a time might block the script.
Finally, with this patch the cloud.log file won't be deleted unless the
shell process has already finished.
Change-Id: I03a77be42d220b1606e48fc4ca35e22d73a6e583
Signed-off-by: Josep Puigdemont <josep.puigdemont@enea.com>
---
deploy/cloud/deployment.py | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/deploy/cloud/deployment.py b/deploy/cloud/deployment.py
index 306abf0..0127d2a 100644
--- a/deploy/cloud/deployment.py
+++ b/deploy/cloud/deployment.py
@@ -101,8 +101,8 @@ class Deployment(object):
LOG_FILE = 'cloud.log'
log('Starting deployment of environment %s' % self.env_id)
- run_proc('fuel --env %s deploy-changes | strings | tee %s'
- % (self.env_id, LOG_FILE))
+ p = run_proc('fuel --env %s deploy-changes | strings > %s'
+ % (self.env_id, LOG_FILE))
ready = False
for i in range(int(self.deploy_timeout)):
@@ -119,7 +119,13 @@ class Deployment(object):
break
else:
time.sleep(SLEEP_TIME)
- delete(LOG_FILE)
+
+ p.poll()
+ if p.returncode == None:
+ log('The process deploying the changes has not yet finished.')
+ log('''The file %s won't be deleted''' % LOG_FILE)
+ else:
+ delete(LOG_FILE)
if ready:
log('Environment %s successfully deployed' % self.env_id)
|
