aboutsummaryrefslogtreecommitdiffstats
path: root/yardstick
AgeCommit message (Collapse)AuthorFilesLines
2018-04-03[docs] update the user guide for "yardstick-operation"Emma Foley1-1/+1
* Add a new chapter: operation which shows how to use Yardstick and how to create test cases * Include details on how to reuse and keep VNFs between testcases JIRA: YARDSTICK-885 JIRA: YARDSTICK-886 Change-Id: I365152e8812427a3c54259adbc18d5ddb216d775 Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2018-04-03[docs] Move user guide chapters to make room for new chapter.Emma Foley1-0/+37
* Increment the numbering on chapters * Fix some of the changes flagged by doc8 Change-Id: Ie03d165297671dfd50b9de339612cdc8bf00b988 Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2018-03-29Merge "don't print exceptions in atexit handler"Ross Brattain1-1/+1
2018-03-29Merge "task: don't hide exceptions in Task"Ross Brattain1-6/+9
2018-03-29don't print exceptions in atexit handlerRoss Brattain1-1/+1
atexit handler calls terminate_all after the regular python execution path, and it looks like the traceback stack is None somehow. In this context log.debug("", exc_info=True) doesn't work it prints out NoneType on Python3 and causes other problems on Python2.7 remove the exc_info=True from the logging call JIRA: YARDSTICK-1107 Change-Id: Ida0a0ced7ff5e017e2f8608880e3bb531724af95 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2018-03-28Merge "restore Heat stack failure logs for CI"Ross Brattain2-1/+27
2018-03-28Merge "Replace neutron create security group rule with shade."Rex Lee2-38/+76
2018-03-27task: don't hide exceptions in TaskRoss Brattain1-6/+9
If the Task raised an exception we currently hide it and replace it with RuntimeError. This is bad. If an exception occured, then we don't have a result so re-raise the original exception. Or we could log the traceback and raise RuntimeError, but that doesn't seem to be a good idea. Sample traceback after re-raising original. Without this patch the ValueError is only written to _write_error_data 2018-03-25 22:57:56,511 yardstick.benchmark.contexts.node node.py:85 DEBUG BareMetals: [] 2018-03-25 22:57:56,511 yardstick.benchmark.contexts.node node.py:89 DEBUG Env: {} 2018-03-25 22:57:56,511 yardstick.cmd.commands.task task.py:57 INFO Task FAILED Traceback (most recent call last): File "/home/rbbratta/yardstick-upstream/yardstick/yardstick/cmd/commands/task.py", line 54, in do_start result = Task().start(param, **kwargs) File "/home/rbbratta/yardstick-upstream/yardstick/yardstick/benchmark/core/task.py", line 103, in start task_args_fnames) File "/home/rbbratta/yardstick-upstream/yardstick/yardstick/benchmark/core/task.py", line 321, in _parse_tasks task_args_fnames[i] File "/home/rbbratta/yardstick-upstream/yardstick/yardstick/benchmark/core/task.py", line 558, in parse_task context.init(cfg_attrs) File "/home/rbbratta/yardstick-upstream/yardstick/yardstick/benchmark/contexts/heat.py", line 131, in init server = Server(name, self, server_attrs) File "/home/rbbratta/yardstick-upstream/yardstick/yardstick/benchmark/contexts/model.py", line 210, in __init__ (name, p)) ValueError: server 'trafficgen_1', placement 'pgrp2' is invalid 2018-03-25 22:57:56,512 yardstick.cmd.commands.task task.py:62 INFO Task FAILED 2018-03-25 22:57:56,662 yardstick.benchmark.runners.base base.py:124 DEBUG Terminating all runners NoneType JIRA: YARDSTICK-1102 Change-Id: I7e6fa41fc1d36f6d438a1602ab60cb41ffbee1e9 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2018-03-27restore Heat stack failure logs for CIRoss Brattain2-1/+27
We need to see what Heat failures occured so that we can debug CI failures. The only way to do this seems to be to use the private event_utils.get_events function to query for all failures Also since we had a failure go ahead and dump the Heat template that failed. Sample output: 2018-03-25 23:50:08,765 [INFO] yardstick.orchestrator.heat heat.py:629 Creating stack 'yardstick-460ed969' START 2018-03-25 23:50:21,932 [ERROR] yardstick.orchestrator.heat heat.py:644 Resource CREATE failed: BadRequest: resources.yardstick-460ed969-xe1: Invalid input for operation: physical_network 'nosuch' unknown for flat provider network. Neutron server returns request_ids: ['req-6f981f1e-a9e2-4114-af84-1ee528aed51b'] 2018-03-25 23:50:21,933 [ERROR] yardstick.orchestrator.heat heat.py:644 BadRequest: resources.yardstick-460ed969-xe1: Invalid input for operation: physical_network 'nosuch' unknown for flat provider network. Neutron server returns request_ids: ['req-6f981f1e-a9e2-4114-af84-1ee528aed51b'] 2018-03-25 23:50:21,972 [ERROR] yardstick.orchestrator.heat heat.py:645 {'description': '\n' 'All referred generated resources are prefixed with the ' 'template\n' 'name (i.e. yardstick-460ed969).\n', 'heat_template_version': '2013-05-23', 'outputs': {'trafficgen_1.yardstick-460ed969': {'description': 'VM UUID', 'value': {'get_resource': 'trafficgen_1.yardstick-460ed969'}}, 'trafficgen_1.yardstick-460ed969-fip': {'description': 'floating ' 'ip ' JIRA: YARDSTICK-998 Change-Id: Ia8f4e5ba7e280fb9086519680d5ee90a2b442e6b Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2018-03-27Merge "Improve "Libvirt.virsh_destroy_vm" function"Rodolfo Alonso Hernandez2-8/+17
2018-03-27Merge "Improve "Libvirt.virsh_create_vm" function"Rodolfo Alonso Hernandez3-8/+26
2018-03-27Merge "Improve NSB Standalone XML generation"Rodolfo Alonso Hernandez6-188/+198
2018-03-27Merge "Improve OVS-DPDK setup process"Rodolfo Alonso Hernandez2-33/+58
2018-03-27Merge "Improve OVS-DPDK boot process"Rodolfo Alonso Hernandez5-165/+326
2018-03-27Replace neutron create security group rule with shade.Shobhi Jain2-38/+76
Rename create_secgroup_rule with create_security_group_rule. Function create_security_group_rule now uses shade client. JIRA: YARDSTICK-890 Change-Id: Ie0ebac67a281e55dc95c0e3e33ba43de80aba9ec Signed-off-by: Shobhi Jain <shobhi.jain@intel.com>
2018-03-27Merge "Replace neutron floating ip deletion with shade."Rodolfo Alonso Hernandez5-30/+93
2018-03-27Merge "Replace neutron floating ip creation with shade."Rodolfo Alonso Hernandez5-63/+98
2018-03-27Merge "Replace neutron router interface deletion with shade."Rodolfo Alonso Hernandez5-35/+101
2018-03-27Replace neutron floating ip deletion with shade.Shobhi Jain5-30/+93
Function delete_floating_ip now uses shade client. JIRA: YARDSTICK-890 Change-Id: I960630926b664266afbe7be00bb1352243b41be0 Signed-off-by: Shobhi Jain <shobhi.jain@intel.com>
2018-03-23Merge "TC042 bug fix"Ross Brattain3-31/+40
2018-03-23Merge "Add "Scenario" class wait methods for runners"Rodolfo Alonso Hernandez5-4/+51
2018-03-23Merge "Make "Scenario" class abstract"Rodolfo Alonso Hernandez2-8/+18
2018-03-22Merge "Make Sample VNF hugepages size configurable"Ross Brattain1-2/+2
2018-03-22Merge "Bugfix: Error when using API(v1) to run test suite"Rex Lee2-0/+35
2018-03-22Bugfix: Error when using API(v1) to run test suitechenjiankun2-0/+35
JIRA: YARDSTICK-1098 The reason is missing TasksHandler parameter to run test suite. Change-Id: I9dd45caa87d0e39afbf7485443a6e566317f5cea Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2018-03-22Merge "Running Test Case in Latest Yardstick Docker Image shows Error"Jack Chan1-3/+13
2018-03-21Add "Scenario" class wait methods for runnersRodolfo Alonso Hernandez5-4/+51
"Duration" and "Iteration" runners execute a passive wait during the execution of the work process. This wait time is done at the end of the scenario "run" method execution. This patch adds a pre-run and post-run wait period, which will depends on the Scenario executed. The wait time will be always the same, but the execution order (pre-wait time, run method, post-wait time) will depends on the Scenario. By default, any Scenario will execute the "run" method and them will wait the specified time. NetworkServicesTestCase Scenario will wait the specified time and them will execute the "run" method to retrieve the KPIs. JIRA: YARDSTICK-1067 Change-Id: I6ad6bfc6978815b6b2d4df63f2ac2f8815fb5b8a Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-03-21Make "Scenario" class abstractRodolfo Alonso Hernandez2-8/+18
All scenario child classes must implement "run" method. JIRA: YARDSTICK-1065 Change-Id: I35b78e380620967b49cd8cd23777a1aee6dfd140 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-03-21Improve "Libvirt.virsh_destroy_vm" functionRodolfo Alonso Hernandez2-8/+17
Read the command exit code and log a warning in case the VM destroy process went wrong. JIRA: YARDSTICK-943 Change-Id: I2750b8d4a8f67af081c1988510cf5aca848a2cf1 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-03-21Improve NSB Standalone XML generationRodolfo Alonso Hernandez6-188/+198
Delayed the generation of the XML file until the last step. The following functions will return a XML string insted: - Libvirt.build_vm_xml - SriovContext._enable_interfaces - OvsDpdkContext._enable_interfaces The XML file will be written just before copying the file to the compute hosting the VMs. JIRA: YARDSTICK-939 Change-Id: Icc80f4741903bbe335db4ebccab395b72fa87e82 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-03-21Improve OVS-DPDK setup processRodolfo Alonso Hernandez2-33/+58
Improved OVS-DPDK setup process: - Added log string for each command executed. - Removed unnecessary 'kill' command. - Check if commands are executed correctly. JIRA: YARDSTICK-922 Change-Id: I09fba96ffa39cde1e000a3c4382d474219c99985 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-03-21Improve "Libvirt.virsh_create_vm" functionRodolfo Alonso Hernandez3-8/+26
Read the command exit code and raise an exception in case the VM boot process went wrong. JIRA: YARDSTICK-941 Change-Id: Ibabaae8983213a33799ad00fa4d9541bbabea857 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-03-21Improve OVS-DPDK boot processRodolfo Alonso Hernandez5-165/+326
Check if hugepages are present in the system. Check if every command executed during the boot up process exits correctly. JIRA: YARDSTICK-917 Change-Id: Ic0c8a72199ad80e3a65488cc88d0692863751284 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-03-21Merge "Replace neutron router creation with shade."Rex Lee5-44/+115
2018-03-21TC042 bug fixliyin3-31/+40
JIRA:YARDSTICK-786 yardstick DPDK image have problems to build. We will fix this by using nsb image the pktgen DPDK testcase will use sampleVNF image Change-Id: I24b7b6a9acfe21ec956fa9c4ee7ebe7ae1787dbb Signed-off-by: liyin <liyin11@huawei.com>
2018-03-20Merge "Fix retrieving "options" section in "scenario""Ross Brattain2-2/+32
2018-03-20Replace neutron floating ip creation with shade.Shobhi Jain5-63/+98
Function create_floating_ip now uses shade client instead of neutron client. JIRA: YARDSTICK-890 Change-Id: I3defd691dad998cebf98442b52f0555b176f1af4 Signed-off-by: Shobhi Jain <shobhi.jain@intel.com>
2018-03-20Replace neutron router interface deletion with shade.Shobhi Jain5-35/+101
Function remove_interface_router now uses shade client instead of neutron client. JIRA: YARDSTICK-890 Change-Id: I6bd36e35a339cce64dfa8b69c1e7b56cd70af956 Signed-off-by: Shobhi Jain <shobhi.jain@intel.com>
2018-03-20Replace neutron router creation with shade.Shobhi Jain5-44/+115
Function create_neutron_router now uses shade client instead of neutron client. JIRA: YARDSTICK-890 Change-Id: I819d26f4edb3af5e50f777e8ddbadf4d24820b7f Signed-off-by: Shobhi Jain <shobhi.jain@intel.com>
2018-03-20Running Test Case in Latest Yardstick Docker Image shows Errorchenjiankun1-3/+13
JIRA: YARDSTICK-1080 This bug will only occur in kubernetes test case. In openstack, we use 'stack_name.context', but some kubernetes resources(pod, etc) do not support this format. So we use 'name-context'. Change-Id: I87d61c027d55449914bc283672eb98be3d3fe22f Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2018-03-19temp unblock unittestsRoss Brattain1-88/+0
====================================================================== ERROR: common.test_packages (unittest.loader._FailedTest) ---------------------------------------------------------------------- ImportError: Failed to import test module: common.test_packages Traceback (most recent call last): File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path module = self._get_module_from_name(name) File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name __import__(name) File "/home/jenkins/opnfv/slave_root/workspace/yardstick-verify-master/yardstick/tests/unit/common/test_packages.py", line 17, in <module> from pip.operations import freeze File "/home/jenkins/opnfv/slave_root/workspace/yardstick-verify-master/.tox/py3/lib/python3.5/site-packages/pip/__init__.py", line 45, in <module> from pip.vcs import git, mercurial, subversion, bazaar # noqa File "/home/jenkins/opnfv/slave_root/workspace/yardstick-verify-master/.tox/py3/lib/python3.5/site-packages/pip/vcs/mercurial.py", line 9, in <module> from pip.download import path_to_url File "/home/jenkins/opnfv/slave_root/workspace/yardstick-verify-master/.tox/py3/lib/python3.5/site-packages/pip/download.py", line 40, in <module> from pip._vendor import requests, six File "/home/jenkins/opnfv/slave_root/workspace/yardstick-verify-master/.tox/py3/lib/python3.5/site-packages/pip/_vendor/requests/__init__.py", line 98, in <module> from . import packages File "/home/jenkins/opnfv/slave_root/workspace/yardstick-verify-master/.tox/py3/lib/python3.5/site-packages/pip/_vendor/requests/packages.py", line 12, in <module> sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod] KeyError: 'pip._vendor.urllib3.contrib' File "/home/jenkins/opnfv/slave_root/workspace/yardstick-verify-master/yardstick/tests/unit/common/test_packages.py", line 16, in <module> from pip import exceptions as pip_exceptions ====================================================================== ERROR: common.test_packages (unittest.loader._FailedTest) ---------------------------------------------------------------------- ImportError: Failed to import test module: common.test_packages Traceback (most recent call last): File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path module = self._get_module_from_name(name) File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name __import__(name) File "/home/jenkins/opnfv/slave_root/workspace/yardstick-verify-master/yardstick/tests/unit/common/test_packages.py", line 16, in <module> from pip import exceptions as pip_exceptions File "/home/jenkins/opnfv/slave_root/workspace/yardstick-verify-master/.tox/py3/lib/python3.5/site-packages/pip/__init__.py", line 45, in <module> from pip.vcs import git, mercurial, subversion, bazaar # noqa File "/home/jenkins/opnfv/slave_root/workspace/yardstick-verify-master/.tox/py3/lib/python3.5/site-packages/pip/vcs/mercurial.py", line 9, in <module> from pip.download import path_to_url File "/home/jenkins/opnfv/slave_root/workspace/yardstick-verify-master/.tox/py3/lib/python3.5/site-packages/pip/download.py", line 40, in <module> from pip._vendor import requests, six File "/home/jenkins/opnfv/slave_root/workspace/yardstick-verify-master/.tox/py3/lib/python3.5/site-packages/pip/_vendor/requests/__init__.py", line 98, in <module> from . import packages File "/home/jenkins/opnfv/slave_root/workspace/yardstick-verify-master/.tox/py3/lib/python3.5/site-packages/pip/_vendor/requests/packages.py", line 12, in <module> sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod] KeyError: 'pip._vendor.urllib3.packages.backports.makefile' Change-Id: I43b1edc23cebe0db97b6b56a551acd376986dfea Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2018-03-16Merge "NSB PROX L3FWD Dropping packets"Abhijit Sinha1-0/+14
2018-03-16Merge "Replace assert statements in tests"Abhijit Sinha2-20/+20
2018-03-16Merge "Create API to get SUT information"Jack Chan5-0/+170
2018-03-15NSB PROX L3FWD Dropping packetsDanielMartinBuckley1-0/+14
JIRA: YARDSTICK-1037 NSB PROX L3FWD was incorrect. The routing table (ipv4.lua) was incorrect. Also a specific ipv4 for 2 port configuration is required. Change-Id: I3a5fd1ea1d1ddf74cbdd6a5d9e7ced42d3c1d6e9 Signed-off-by: Daniel Martin Buckley <daniel.m.buckley@intel.com> Signed-off-by: Abhijit Sinha <abhijit.sinha@intel.com>
2018-03-15Merge "support for Bottlenecks soak throughputs"Jing Lu1-1/+3
2018-03-15Create API to get SUT informationchenjiankun5-0/+170
JIRA: YARDSTICK-1072 We need to show SUT information in GUI. Change-Id: I40bcd513f3c6a443c82764687637ac5258b99584 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2018-03-15support for Bottlenecks soak throughputsYang Yu1-1/+3
JIRA: YARDSTICK-1049 Changes: 1. NetperfNode -> Netperf, it seems that yardstick-image has not correctly complied Netperf since cpuutil is not available, resulting that LOCAL_CPU_UTIL=-1 for the output. Will create another JIRA ticket for this. Change-Id: I26a85f4aafcc5d5e1eda3e04272ecf3b059238fc Signed-off-by: Yang Yu <Gabriel.yuyang@huawei.com>
2018-03-14Fix retrieving "options" section in "scenario"Rodolfo Alonso Hernandez2-2/+32
In [1] a new method to rename the "scenario" host names was implemented. This method parses all sections with host names and applies the "qualified" name. Some malformed test cases define the "options" section without content. This causes that [2] retrieves "None" value instead of an empty dictionary. This possibility must be handled in the new method. [1]I44da30dac562c1a4166e084645ae91c17798651d [2]https://github.com/opnfv/yardstick/blob/4b8b674b65830a24230faed71e8d9a1048139c89/yardstick/benchmark/core/task.py#L630 JIRA: YARDSTICK-1073 Change-Id: I8864b428734ead8c5aa39de5091d3a2a691be060 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-03-14Merge "Get SUT information"Jack Chan2-0/+69