diff options
| author |   Ross Brattain <ross.b.brattain@intel.com> | 2017-09-29 15:39:04 -0700 |
|---|---|---|
| committer | Ross Brattain <ross.b.brattain@intel.com> | 2017-10-03 11:01:44 -0700 |
| commit | 7496f13e0be433697a74d7156c47d950b2add0a9 (patch) | |
| tree | 27528bf08ce6d15a23c037c79506f348ca980969 /yardstick/network_services | |
| parent | 1d2c872c4524543a2440ee942734dc0aa9448ff9 (diff) | |
drain runner queue and undo cancel_join_thread
Sometimes the runners can hang. Initially
debugging lead to the queue join thread, so I thought
we could cancel all the join threads and everything would be okay.
But it turns out canceling the queue join threads can lead
to corruption of the queues, so when we go to drain the queues
the task hangs.
But it also turns out that we were not properly draining
the queues in the task process. We were waiting for all
the runners to exit, then draining the queues.
This is bad and will cause the queues to fill up and hang
and/or drop data or corrupt the queues.
The proper fix seems to be to draining the queues in a
loop before calling join with a timeout.
Also modified the queue drain loops to no block on queue.get()
Revert "cancel all queue join threads"
This reverts commit 75c0e3a54b8f6e8fd77c7d9d95decab830159929.
Revert "duration runner: add teardown and cancel all queue join threads"
This reverts commit 7eb6abb6931b24e085b139cc3500f4497cdde57d.
Change-Id: Ic4f8e814cf23615621c1250535967716b425ac18
Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
Diffstat (limited to 'yardstick/network_services')
4 files changed, 6 insertions, 0 deletions
diff --git a/yardstick/network_services/nfvi/collectd.py b/yardstick/network_services/nfvi/collectd.py index f2c9d40a7..e0027bbcb 100644 --- a/yardstick/network_services/nfvi/collectd.py +++ b/yardstick/network_services/nfvi/collectd.py @@ -35,6 +35,7 @@ class AmqpConsumer(object): self._consumer_tag = None self._url = amqp_url self._queue = queue + self._queue.cancel_join_thread() def connect(self): """ connect to amqp url """ diff --git a/yardstick/network_services/traffic_profile/prox_profile.py b/yardstick/network_services/traffic_profile/prox_profile.py index 896384d5e..170dfd96f 100644 --- a/yardstick/network_services/traffic_profile/prox_profile.py +++ b/yardstick/network_services/traffic_profile/prox_profile.py @@ -67,6 +67,7 @@ class ProxProfile(TrafficProfile): def init(self, queue): self.pkt_size_iterator = iter(self.pkt_sizes) self.queue = queue + self.queue.cancel_join_thread() def bounds_iterator(self, logger=None): if logger: diff --git a/yardstick/network_services/vnf_generic/vnf/prox_helpers.py b/yardstick/network_services/vnf_generic/vnf/prox_helpers.py index 02ae06170..ba4d44c41 100644 --- a/yardstick/network_services/vnf_generic/vnf/prox_helpers.py +++ b/yardstick/network_services/vnf_generic/vnf/prox_helpers.py @@ -878,6 +878,7 @@ class ProxResourceHelper(ClientResourceHelper): return self._test_type def run_traffic(self, traffic_profile): + self._queue.cancel_join_thread() self.lower = 0.0 self.upper = 100.0 diff --git a/yardstick/network_services/vnf_generic/vnf/sample_vnf.py b/yardstick/network_services/vnf_generic/vnf/sample_vnf.py index 91530860e..5cf234514 100644 --- a/yardstick/network_services/vnf_generic/vnf/sample_vnf.py +++ b/yardstick/network_services/vnf_generic/vnf/sample_vnf.py @@ -476,6 +476,9 @@ class ClientResourceHelper(ResourceHelper): self._queue.put(samples) def run_traffic(self, traffic_profile): + # if we don't do this we can hang waiting for the queue to drain + # have to do this in the subprocess + self._queue.cancel_join_thread() # fixme: fix passing correct trex config file, # instead of searching the default path try: |