aboutsummaryrefslogtreecommitdiffstats
path: root/yardstick/benchmark/core/task.py
AgeCommit message (Collapse)AuthorFilesLines
2017-07-04Bugfix: background scenario can't not update resultchenjiankun1-2/+4
JIRA: YARDSTICK-700 In this patch: https://gerrit.opnfv.org/gerrit/#/c/35257/ In yardstick/benchmark/core/task.py We update background result only when it be stopped abnormally. We need move it out of 'if' statement. Change-Id: I25b76e160273a87da66098565a16e4a143cec11f Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-06-30unify pod keywork so api can easily usedrexlee87761-1/+1
JIRA: YARDSTICK-695 Change-Id: I826cb2ea07949b1eaae4df4cb77175ec8f5df89b Signed-off-by: rexlee8776 <limingjiang@huawei.com>
2017-06-29Call core code directly in the API of run test casechenjiankun1-1/+8
JIRA: YARDSTICK-688 We need to call core code directly in the API of runTestCase. It would be more stable. Change-Id: I431a85ded7cd3b20da0462f947c25d91bb99decd Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-06-24Merge "Yardstick output format unified"Jing Lu1-10/+102
2017-06-22Merge "Acquire NSB specific data from Heat."Ross Brattain1-0/+3
2017-06-22Yardstick output format unifiedchenjiankun1-10/+102
JIRA: YARDSTICK-658 Currently the yardstick have three dispatcher: file, influxdb, mongodb. (influxdb using API to get result and mongodb using testAPI to get result) But their output format is different. It is hard to use. In this patch, make all dispatchers using the same data source. And make the output format of file and influxdb unified. As for mongodb, since it is related to testAPI, so I make it push data every test case. The unified output format is: http://paste.openstack.org/show/610125/ Change-Id: I854ac4f03e6f904469b07b0c924c7d850545ae5b Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-06-20Acquire NSB specific data from Heat.Edward MacGillivray1-0/+3
First we add mac_address, subnet_cidr to Heat template outputs Then we convert those into a form for NSB and add vld_id. NSB also requires PCI Bus ID, kernel driver and dpdk_port_num. We get this by ssh-ing into instance and dumping sysfs We also need to fix allow for ssh key auth, and implement relative path file loading so NSB can find all its YAML files JIRA: YARDSTICK-580 Change history: don't hide heat create tracebacks we need tracebacks for debug vnf_generic: add task_path to scenario so we can load relative paths for vnf_generic we want to be able to load yaml relative to the task path For example: traffic_profile: ../../traffic_profiles/fixed.yaml topology: ping_tg_topology.yaml # TODO: look in relative path where the tc.yaml is found These need to be relative to samples/vnf_samples/nsut/ping/tc_ping_heat_context.yaml Add a scenario["task_path"] entry heat: log actual exception vnf_generic: replace list with set and iterate over values() some general refactors to remove redundact lookups and type conversions heat: provide mac_address, device_id and network_id from outputs We may need more information to dynamically determine test topology. Towards this end return more info in the heat template. We can return mac_address, device_id and network_id. Once we have this info we can add it to the context_cfg as an interfaces dict. add sample vnf ping multi-network test this test requires 3 network, one for mgmt and the other two for NSB traffic tests We have to make sure we don't use DPDK on mgmt interface because DPDK unbinds the driver heat: convert networks to OrderedDict so we can lookups networks as well as iterate over them in consisitent order heat: and vld_id to networks for vnf_generic vnf_generic uses vld_id Virtual Link Descriptor ID to identify interfaces Add the key to the networks dict and store in Networks object implement relative path file loading in vnf_generic in multiple places we need to load a file relative to the task path, so add open_relative_file_path and modify load_vnf_model to include the scenario_cfg parameter so we have access to task_path DRAFT: heat timeout support Heat stack in CI job failed due to some Nova issue. But then apparently yardstick kept running and took 180mins to timeout https://build.opnfv.org/ci/view/bottlenecks/job/bottlenecks-compass-posca_stress_ping-baremetal-daily-master/16/console We can add a Heat create timeout and fail faster if there is an error. The question is how long should we wait for a Heat stack to deploy. We can set a default and allow override in the heat context config, if users make complicated stacks heat: get netmask and gateway from heat outputs we have do some tricky business with finding the subnet cidr and converting it into netmask vnf_generic: get vpci, driver and dpdk_port_num use a big old find command to dump all the sysfs netdev info nicely. This was re-used from autotest FCoE tests. r"""find /sys/devices/pci* -type d -name net -exec sh -c '{ grep -sH ^ \ +$1/ifindex $1/address $1/operstate $1/device/vendor $1/device/device \ +$1/device/subsystem_vendor $1/device/subsystem_device ; \ +printf "%s/driver:" $1 ; basename $(readlink -s $1/device/driver); } \ +' sh \{\}/* \; This finds all PCI devices that are network devices, then dumps all the relevant info using /bin/sh. Then we parse this into a 'netdevs' dict inside the node_dict and also convert into VNF fields we need. vnf_generic: set node name for kpis node is a dict, so we have to use node_name vnfdgen: we CANNOT use TaskTemplate.render because it does not allow for missing variables, we need to allow password for key_filename to be undefined remove default ssh password hack, once rendering is fixed add new example tc_external_ping_heat_context Change-Id: If1fe0c1a2ab0a5be17e40790a66f28f706fa44d6 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com> Signed-off-by: Edward MacGillivray <edward.s.macgillivray@intel.com>
2017-05-26Pass parameters between scenarioschenjiankun1-0/+17
JIRA: YARDSTICK-641 Allowing parameters to pass between scenarios so that the one test case can be combination of several scenarios. Change-Id: I55a00855e77d5b719a27a069a3ea195d6bbd0ef8 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-04-01Yardstick virtualenv supportchenjiankun1-1/+1
JIRA: YARDSTICK-620 Currently we recommend using docker to run yardstick. And it is hard to use virtualenv to install yardstick. So I modify install.sh in yardstick root path. It will support using virtualenv to install yardstick(including API) in linux. In this patch, I make yardstick support read yardstick configuration have priority over constants. Change-Id: I9ea1241b228532a6497451e6c8f232173ddb783e Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-03-22Merge "Yardstick: User interface for Yardstick."Jing Lu1-0/+4
2017-03-20Bugfix: netperf_bottleneck execute failed because is_same_heat_context ↵chenjiankun1-1/+0
return True JIRA: YARDSTICK-600 Bugfix: netperf_bottleneck execute failed because is_same_heat_context return True So it exit without warning. I will remove the 'return True' from it. Change-Id: I48d62bd15ec359e2121f236c48d9d349174f2c10 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-03-17Yardstick: User interface for Yardstick.rajesh_4k1-0/+4
Currently Yardstick doesnt have any UI which gives detail analysis of the test-results. This commit generates a HTML page after the execution of a command "yardstick report generate <task-ID> <TC-name>" which intern can be executed after the execution of test-case. Used: Highcharts.js for the graphs. JIRA: YARDSTICK-280 Change-Id: Ic98cc348719f3922bff178f52e7944a4a931763a Signed-off-by: Rajesh K <4k.rajesh@gmail.com>
2017-03-09Merge "move external_network auto-assign to Heat context"Rex Lee1-11/+1
2017-03-08Bugfix: yardstick will create stacks with the same name when run using API ↵chenjiankun1-98/+108
in parallel JIRA: YARDSTICK-575 Currently yardstick will create stacks with the same name when run using API in parallel. The reason is there is a global variable in context base and the core will always deploy the first context in Context.list. When run in parallel, it will run in the one process. So yardstick will deploy stacks with the same name. The solution is do not use Context.list in yardstick core. And using a local variable instead. BTW, if we use API to call yardstick core, we can not config the output way. So I parse yardstick.conf when task start. And I think we can include scenario_cfg, context_cfg, yardstick_cfg in one config object later so that we can get all config in one object. Change-Id: I1ada4ef486bd252e78c3a2e49c6a39b3f8f16a7c Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-02-28move external_network auto-assign to Heat contextRoss Brattain1-11/+1
if there are no external networks defined then assign the external network to the first network as determined by standard sorting Change-Id: I9d32eca258f8e7de3d44cec4124c0e8c020c0b85 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-02-21ci in huawei-pod1 do not run test case after tc074chenjiankun1-1/+5
JIRA: YARDSTICK-568 https://build.opnfv.org/ci/view/yardstick/job/yardstick-compass-baremetal-daily-master/814/consoleFull the test case tc075 and tc027 after tc074 does not run in the ci, which is not correct. the logs shows the suite quit after tc074 finishes. The reason is: In this patch: https://gerrit.opnfv.org/gerrit/#/c/26763/ I add a task_id to the context name, but in tc074, it do not have a name. It will cause a keyError. Change-Id: I09608675d9dadcc3ff0d825ca63db3be5e414120 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-02-16Merge "Context improvement: add support for configing node environment"Rex Lee1-2/+2
2017-02-16Context improvement: add support for configing node environmentchenjiankun1-2/+2
JIRA: YARDSTICK-556 Currently we do can not config the node environment before heat create a stack. But in lots of scene, we need to config node environment before heat. So I add support for it. Change-Id: Iac1b74dc780eb40e6ab2c9cf04ed14e2b8f91ca8 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-02-15Bugfix: AttributeError: 'dict' object has no attribute 'encode'chenjiankun1-3/+19
JIRA: YARDSTICK-557 If we run task with sample/ping.yaml We will encounter below error, here is the log: Traceback (most recent call last): File "/usr/local/bin/yardstick", line 11, in <module> load_entry_point('yardstick==0.1.dev0', 'console_scripts', 'yardstick')() File "/usr/local/lib/python2.7/dist-packages/yardstick/main.py", line 49, in main YardstickCLI().main(sys.argv[1:]) File "/usr/local/lib/python2.7/dist-packages/yardstick/cmd/cli.py", line 167, in main self._dispath_func_notask() File "/usr/local/lib/python2.7/dist-packages/yardstick/cmd/cli.py", line 145, in _dispath_func_notask func(CONF.category) File "/usr/local/lib/python2.7/dist-packages/yardstick/cmd/commands/task.py", line 45, in do_start Task().start(param, **kwargs) File "/usr/local/lib/python2.7/dist-packages/yardstick/benchmark/core/task.py", line 83, in start self._run(scenarios, run_in_parallel, args.output_file) File "/usr/local/lib/python2.7/dist-packages/yardstick/benchmark/core/task.py", line 131, in _run runner = run_one_scenario(scenario, output_file) File "/usr/local/lib/python2.7/dist-packages/yardstick/benchmark/core/task.py", line 410, in run_one_scenario if is_ip_addr(scenario_cfg["target"]): File "/usr/local/lib/python2.7/dist-packages/yardstick/benchmark/core/task.py", line 358, in is_ip_addr ipaddress.ip_address(addr.encode('utf-8')) AttributeError: 'dict' object has no attribute 'encode' Change-Id: Iba1570416bd8614e38c9e847de730a31d9ddedc2 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-01-20Record task status if running via CLIchenjiankun1-2/+0
JIRA: YARDSTICK-542 Currently we do not record task status when using CLI to run task. So I add this function. If status=0, task is not done. if status=1, task is done. if status=2, there is an error. Change-Id: Ib9b3c8abd233909c04f792115199250419fa8d7a Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-01-20Merge "Bugfix:can not run a test suite if not under yardstick root path"Jing Lu1-0/+2
2017-01-19Bugfix: context not found for server ''node1.LF''chenjiankun1-1/+2
JIRA: YARDSTICK-540 We are adding dynamically generated suffix to server name in the previous patch. But in that patch we just change the 'target', 'host' in nodes, but not 'node1', 'node2' ... This patch will change all this. Change-Id: Ic7c82c323dea24a40f1fe30871603d30c2689f67 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-01-16Use """ to replace ''' in docstringchenjiankun1-17/+17
JIRA: YARDSTICK-525 For consistency, we always use """triple double quotes""" around docstrings. Change-Id: I47a20bbd8b55bc544b4841ea4006929af0a044ac Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-01-14Bugfix:can not run a test suite if not under yardstick root pathchenjiankun1-0/+2
JIRA: YARDSTICK-530 Currently we can not run a test suite if we are not in yardstick root path. The reason is that the file in test suite config file use relative path. So I change it to absolute path when run. Change-Id: I62758bc67f466ac794d339b597562b3be05574fb Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-01-12Add support for Python 3Ross Brattain1-16/+19
Porting to Python3 using Openstack guidelines: https://wiki.openstack.org/wiki/Python3 This passes unittests on Python 3.5 and passes opnfv_smoke suite Updates: use six for urlparse and urlopen fix exception.message attribute removal run unittests on python3 use unitest.mock on python 3 fix open mock for vsperf fix float division by using delta/eplison comparison use unicode in StringIO use plugin/sample_config.yaml relative path from test case fixed apexlake unittests upgraded to mock 2.0.0 to match python3 unittest.mock features fixed flake8 issues implement safe JSON decode with oslo_serialization.jsonutils.dump_as_bytes() implement safe unicode encode/decode with oslo_utils.encodeutils heat: convert pub key file from bytes to unicode pkg_resources returns raw bytes, in python3 we have to decode this to utf-8 unicode so JSON can encode it for heat template JIRA: YARDSTICK-452 Change-Id: Ib80dd1d0c0eb0592acd832b82f6a7f8f7c20bfda Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-01-09Yardstick framework concurrent supportchenjiankun1-1/+28
JIRA: YARDSTICK-528 Currently yardstick framework can not support run the same test case at the same time. But actually we need to support it. The reason why framework can't support it is that openstack do not allow to create stack with the same name. So I use the task_id to make the stack different. Change-Id: I9e853793650066dfc56606464f7826f330a1401c Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2016-12-30Split Yardstick CLI with Yardstick core logicchenjiankun1-0/+484
JIRA: YARDSTICK-511 We need to unify yardstick entry. Now the solution is using CLI call API as nova do. This is the first step: coupling the yardstick core logic from CLI. Moving the core logic to yardstick/benchmark/core and the CLI using a object to call yardstick core logic. Change-Id: I84f10d2134635880c281cc63212a8533f2dd7d4e Signed-off-by: chenjiankun <chenjiankun1@huawei.com>