aboutsummaryrefslogtreecommitdiffstats
path: root/yardstick/benchmark/core
AgeCommit message (Collapse)AuthorFilesLines
2017-12-15pretty cli format: runner/scenario/testcase listrexlee87763-24/+22
JIRA: YARDSTICK-855 1. using prettytable to reformat cli below: yardstick runner list yardstick scenario list yardstick testcase list 2. remove redundant function print_hbar 3. fix pep8 problems, the framework using do_list dynamicly, so disable unused-argument check for now. Change-Id: I79e1226e21fca4624bf4436414917bd07ecabca0 Signed-off-by: rexlee8776 <limingjiang@huawei.com>
2017-11-07Create get_description and get_scenario_type for ScenarioRodolfo Alonso Hernandez1-4/+3
Created two new class functions for Scenario class: * "get_description" will return the first line of the docstring, if exists. * "get_scenario_type" will return the value of "__scenario_type__" variable. Change-Id: I8e3b47e33e0bae101b874c3975bb9c383397d188 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2017-10-13task: use CONF_FILE from constantsRoss Brattain1-2/+2
I'm not sure why we hardcoded config_file when the CONF_FILE constant will work. Change-Id: I45a1dc17ac2b185c659725f55462214acb74838d Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-10-11task: drain background runner queuesRoss Brattain1-6/+14
We were not draining the queues in the background runners. Modify the main runner_join loop to loop over all the runners and drain them. The runner join method does extra work for periodic actions so we can't change its behavior. Instead create a new poll() method and use that to check runner status Change-Id: I9466ba40a6a4c45c82cedff279cbb4817c6b66ad Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-10-02drain runner queue and undo cancel_join_threadRoss Brattain1-15/+16
Sometimes the runners can hang. Initially debugging lead to the queue join thread, so I thought we could cancel all the join threads and everything would be okay. But it turns out canceling the queue join threads can lead to corruption of the queues, so when we go to drain the queues the task hangs. But it also turns out that we were not properly draining the queues in the task process. We were waiting for all the runners to exit, then draining the queues. This is bad and will cause the queues to fill up and hang and/or drop data or corrupt the queues. The proper fix seems to be to draining the queues in a loop before calling join with a timeout. Also modified the queue drain loops to no block on queue.get() Revert "cancel all queue join threads" This reverts commit 75c0e3a54b8f6e8fd77c7d9d95decab830159929. Revert "duration runner: add teardown and cancel all queue join threads" This reverts commit 7eb6abb6931b24e085b139cc3500f4497cdde57d. Change-Id: Ic4f8e814cf23615621c1250535967716b425ac18 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-09-25task: fixup s/exe_info/exc_infoRoss Brattain1-1/+1
JIRA: YARDSTICK-785 Change-Id: Ib37498e8df6a520f1d03256b73346fcedab3a177 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-09-25Merge "Log each test case status in a task"Ross Brattain1-2/+5
2017-09-21Add service in kubernetes contextchenjiankun1-5/+5
JIRA: YARDSTICK-803 Currently kubernetes test case can only run in master node. We need to support it run in jump server. So I add service and use nodePort type. Then we can login the pod using nodePort. Change-Id: Ia7900d263f1c5323f132435addec27ad10547ef9 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-09-21Merge "Add host&targer in scenario['options']['server_name'] support"Jing Lu1-31/+37
2017-09-15Generate pod.yaml from current contextRoss Brattain1-0/+1
We want to generate pod.yaml from Heat contexts so we can re-use the context without destroying it. But we don't have node role information and it doesn't make sense in this case, so make the role optional. Since we changed Heat to use pkey instead of key_filename, we can embed the pkey into the pod.yaml, but we have to make sure to convert the pkey to string, in case it is a RSAKey object Change-Id: Ibefcfbd8236e68013a704c39964cb870da825da8 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-09-15Merge "improve logging, clear using print"Rex Lee2-22/+20
2017-09-15Add host&targer in scenario['options']['server_name'] supportchenjiankun1-31/+37
JIRA: YARDSTICK-810 Currently host, target is in scenario, but as a input, we prefer it in scenario['options']. So I add support for under scenario['options']['server_name'] If we write host in scenario['options']['server_name'], the host ip info will be written in context. Change-Id: I90df20467ef5da772d22e9f272a2cac250f822e0 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-09-13Heat: support non-mesh network toplogyRoss Brattain1-6/+4
Previsouly we added all servers to every network in Heat in a full mesh. To more closely replicate test topology and to limit then number of ports we need to all each server to specify which ports should be connected to each network. This should also allow for some kind of multiport setup. Add optional network_ports dict to each server with network to port_list mapping match inteface based on port name or vld_id replace vld_id matching with network name matching, since network_name == vld_id Change-Id: I5de46b8f673949e3c17d8df6fa96f055c43886ce Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-09-14improve logging, clear using printrexlee87762-22/+20
Change-Id: I744353f631cf1771d75f750543e8612f81be71ee Signed-off-by: rexlee8776 <limingjiang@huawei.com>
2017-09-11Log each test case status in a taskchenjiankun1-2/+5
JIRA: YARDSTICK-785 Currently if one test case failed, we will log error. But if one case success, we do not have any tips. We need log success when one case success. Change-Id: I0f41ac55f2569f44b787133e3f2594a5c5547f4a Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-08-28Kubernetes context adoption when run in CIchenjiankun1-15/+0
JIRA: YARDSTICK-801 We need kubernetes test case run in CI. We have one scenario k8-nosdb-lb-noha for kubernetes. But for now, we need to do some adoption in CI script. Also I move check OS_AUTH_URL in heat.py. Change-Id: Iff86caf77fd74e70469ad2dc43f2238097ce0001 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-08-17Merge "utils: create TASK_LOG_DIR if it doesn't exist"Ross Brattain1-0/+1
2017-08-15YAML fixesRoss Brattain3-7/+8
There are multiple issues wiht YAML loading. 1. Jinja2 renders None values as a string 'None'. This is not valid YAML we need to render None values to '~' or 'null' which is the native YAML None value. 2. Jinja2 renders dict and lists that contain unicode with u'foo' values. This is not value YAML syntax. Because we are serializing dict and lists into YAML, we need to encode them as valid YAML. We can override Jinja2 finalize to use yaml.dump to dump inline YAML. We use yaml.safe_dump(elem, default_flow_style=True).replace('\n', '') to generate valid single-line YAML dict and list values. But this problem highlights the general difficulties with templating and loading files. We could avoid this Python->Jinja2->YAML->Python issue by directly injecting the list or dict after the YAML is loaded. I'm not sure of the real utility of these templates. 3. On Python 2 YAML loader is rendering all strings as unicode. This does not work for Trex because Trex is broken and badly coded. Trex does type checking against str() which is different for Python 2 and Python 3. The default YAML loader will return native string types, str() or unicode() for Python 2 and Python 3 respectively. The bad Trex codes is in convert_val: https://github.com/cisco-system-traffic-generator/trex-core/blob/master/scripts/automation/trex_control_plane/stl/trex_stl_lib/trex_stl_packet_builder_scapy.py#L674 def convert_val (val): if is_integer(val): return val if type(val) == str: return ipv4_str_to_num (is_valid_ipv4(val)) raise CTRexPacketBuildException(-11,("init val invalid %s ") % val ); This code is doing type(val) == str. This is bad and broken. We can't fix Trex, so we have to render all strings as native str() types The bug here was that the Heat template loader template_format.py was overriding the global YAML loader to always return unicode. We don't want this global override. To fix this we have to use local subclasses of the yaml.SafeLoader class. But in order to dynamically subclass from CSafeLoader or SafeLoader we have to use the type() builtin to define a new class at runtime. Once we have new classes defined, we can safely isolate different YAML constructors and return unicode or not depending on the case. To be consistent we implement a new yaml_loader.py module to centralize all non-Heat template yaml loading to ensure correct uncode/str conversion Change-Id: Iebf9cf78fbda390977c390436b0869e7bbf503eb Signed-off-by: Ross Brattain <ross.b.brattain@intel.com> Signed-off-by: Deepak S <deepak.s@linux.intel.com> Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-08-14Add yardstick report for each taskchenjiankun1-0/+10
JIRA: YARDSTICK-784 Currently we have yardstick report in GUI. But if users do not use GUI, they can't see this report. So we need generate a report each task. After run each test case, we can see report: http://ip:port/report/report.html Change-Id: Ic76cf57f55aa6680b91272e210135136f0225373 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-08-13utils: create TASK_LOG_DIR if it doesn't existRoss Brattain1-0/+1
Change-Id: I10630599065e6c159b137bd4d6ee89353a1c411b Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-08-11Add real time log view in GUIchenjiankun1-2/+15
JIRA: YARDSTICK-775 We have GUI now, but we can't see real time log in GUI view. So I add real time log view in GUI. Change-Id: Ie83f327ef0a94302afa6b3def764fec6ef5818d1 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-08-10bugfix: tc037/38 multiprocessing failed sometimesrexlee87761-4/+4
set runner.join timeout JIRA: YARDSTICK-771 Change-Id: Iec7d04549a45c0fbd0eb48dcaad7770fd5ca1f4e Signed-off-by: rexlee8776 <limingjiang@huawei.com>
2017-08-09bugfix: tc025 and tc054 failrexlee87761-2/+3
also modify tc025 an tc054 to be able to pass paras JIRA: YARDSTICK-772 Change-Id: Ibeba931804cccfd74fc70fcf4fdb6af9d5c8ab77 Signed-off-by: rexlee8776 <limingjiang@huawei.com>
2017-08-08NSB updateDeepak S1-2/+6
Refactored main NSB VNF classes accroding to class diagram https://wiki.opnfv.org/display/yardstick/NSB+class+diagram All the SampleVNFs have been separated and placed under the SampleVNF class. Added AutoConnectSSH to automatically create SSH conneciton on demand. Added VnfdHelper class to wrap the VNFD dictionary in prepartion for class-based modeling. Extracted DpdkVnfSetupEnvHelper for DPDK based VNF setup. Extracted Stats and other client config to ResourceHelper Had to replace dict_key_flatten with deepgetitem due to Python 2.7 Jinja2 infinite recursion. Change-Id: Ia8840e9c44cdbdf39aab6b02e6d2176b31937dc9 Signed-off-by: Deepak S <deepak.s@linux.intel.com> Signed-off-by: Edward MacGillivray <edward.s.macgillivray@intel.com> Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-08-01replace yaml.load with yaml.safe_loadRoss Brattain3-4/+4
yaml.safe_load is safer, obviously. anteater will check for this template_format use specialized constructor based on yaml.SafeLoader JIRA: YARDSTICK-760 Change-Id: Ia3b0b3aa0765385a0ee472a4d83f49d424b5a77f Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-07-27Adding support for multi-dispatcherDeepak S1-3/+13
JIRA: YARDSTICK-684 This patch adds the support to enable multi-dispatcher of result. e.g. [DEFAULT] debug = False dispatcher = file, http Change-Id: Id0e1a1bba14edd899fcf5b275be1f0a091b6db77 Signed-off-by: Deepak S <deepak.s@linux.intel.com> Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-07-26Bugfix: plugin remove command use "JUMP_HOST_IP" if ip is set to localJingLu51-2/+2
JIRA: YARDSTICK-762 Change-Id: Id4a94ef8bbe578944e3664e5e7c24fdf7478805b Signed-off-by: JingLu5 <lvjing5@huawei.com>
2017-07-25Merge "yardstick env influxdb/grafana cmd support centos"Rex Lee1-1/+6
2017-07-24Merge "Open storperf testcase to huawei-pod2"Jing Lu1-2/+2
2017-07-20Add API(v2) to get all test suitechenjiankun1-0/+42
JIRA: YARDSTICK-747 API: /api/v2/yardstick/testsuites METHOD: GET Change-Id: I2375b6d2ea54704f5a2328cc8110058af038e777 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-07-19yardstick env influxdb/grafana cmd support centoschenjiankun1-1/+6
JIRA: YARDSTICK-714 Currently yardstick env influxdb/grafana command do not support centos. Because we use the gateway ip to get the service of influxdb and grafana. But in centos, we can not access influxdb/grafana service via gateway ip. In this patch, I use docker inspect to get the ip of influxdb and grafana. So these command can support centos. Change-Id: Ie4abcb8048b92d61bc62777dfc362bb29f354b2b Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-07-17Open storperf testcase to huawei-pod2JingLu51-2/+2
JIRA: YARDSTICK-712 Change-Id: If91c936a3a59580a987cb6762db5d063edf6c9d7 Signed-off-by: JingLu5 <lvjing5@huawei.com>
2017-07-13add network info to topologyRoss Brattain1-8/+22
we need to know which network each port is connected to so we can find VLAN or VXLAN ID. To do this we implement a new method for Contexts, Context.get_network(). This method is similar to Context.get_server(), it searches for a given network name in all the contexts. From this we generate a context_cfg["networks"] dict that stores all the network info for the nodes in the scenario. Then when we generate the topology for VNFD, we can lookup a given network by the vld_id and get the network_type, segmentation_id, etc. Then if we need to for example generated traffic on a given VLAN or VXLAN, we have this info available. Define default nd_route_tbl for ACL VNF we need default empty nd_route_tbl for IPv6 route. Change-Id: I9f9cfbd6acabeb4ae4675ca7354390efa57b29e7 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com> Signed-off-by: Edward MacGillivray <edward.s.macgillivray@intel.com>
2017-07-06Merge "Bugfix: Abnormally stopped background scenario causes test data lost"Rex Lee1-4/+0
2017-07-06Merge "Bugfix: test case constraint not work if "INSTALLER_TYPE" and ↵Kubi1-3/+3
"NODE_NAME" absent"
2017-07-05Merge "unify pod keywork so api can easily used"Kubi1-1/+1
2017-07-04Bugfix: test case constraint not work if "INSTALLER_TYPE" and "NODE_NAME" absentJingLu51-3/+3
Change-Id: Idffc07c28fabda590b58c9501f388099f84f06b2 Signed-off-by: JingLu5 <lvjing5@huawei.com>
2017-07-04Bugfix: Abnormally stopped background scenario causes test data lostJingLu51-4/+0
Change-Id: I19b8d1c1c6becf4335537d585046cbd59cab5935 Signed-off-by: JingLu5 <lvjing5@huawei.com>
2017-07-04Bugfix: background scenario can't not update resultchenjiankun1-2/+4
JIRA: YARDSTICK-700 In this patch: https://gerrit.opnfv.org/gerrit/#/c/35257/ In yardstick/benchmark/core/task.py We update background result only when it be stopped abnormally. We need move it out of 'if' statement. Change-Id: I25b76e160273a87da66098565a16e4a143cec11f Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-06-30unify pod keywork so api can easily usedrexlee87761-1/+1
JIRA: YARDSTICK-695 Change-Id: I826cb2ea07949b1eaae4df4cb77175ec8f5df89b Signed-off-by: rexlee8776 <limingjiang@huawei.com>
2017-06-29Call core code directly in the API of run test casechenjiankun3-3/+10
JIRA: YARDSTICK-688 We need to call core code directly in the API of runTestCase. It would be more stable. Change-Id: I431a85ded7cd3b20da0462f947c25d91bb99decd Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-06-24Merge "Yardstick output format unified"Jing Lu1-10/+102
2017-06-22Merge "Acquire NSB specific data from Heat."Ross Brattain1-0/+3
2017-06-22Yardstick output format unifiedchenjiankun1-10/+102
JIRA: YARDSTICK-658 Currently the yardstick have three dispatcher: file, influxdb, mongodb. (influxdb using API to get result and mongodb using testAPI to get result) But their output format is different. It is hard to use. In this patch, make all dispatchers using the same data source. And make the output format of file and influxdb unified. As for mongodb, since it is related to testAPI, so I make it push data every test case. The unified output format is: http://paste.openstack.org/show/610125/ Change-Id: I854ac4f03e6f904469b07b0c924c7d850545ae5b Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-06-20Acquire NSB specific data from Heat.Edward MacGillivray1-0/+3
First we add mac_address, subnet_cidr to Heat template outputs Then we convert those into a form for NSB and add vld_id. NSB also requires PCI Bus ID, kernel driver and dpdk_port_num. We get this by ssh-ing into instance and dumping sysfs We also need to fix allow for ssh key auth, and implement relative path file loading so NSB can find all its YAML files JIRA: YARDSTICK-580 Change history: don't hide heat create tracebacks we need tracebacks for debug vnf_generic: add task_path to scenario so we can load relative paths for vnf_generic we want to be able to load yaml relative to the task path For example: traffic_profile: ../../traffic_profiles/fixed.yaml topology: ping_tg_topology.yaml # TODO: look in relative path where the tc.yaml is found These need to be relative to samples/vnf_samples/nsut/ping/tc_ping_heat_context.yaml Add a scenario["task_path"] entry heat: log actual exception vnf_generic: replace list with set and iterate over values() some general refactors to remove redundact lookups and type conversions heat: provide mac_address, device_id and network_id from outputs We may need more information to dynamically determine test topology. Towards this end return more info in the heat template. We can return mac_address, device_id and network_id. Once we have this info we can add it to the context_cfg as an interfaces dict. add sample vnf ping multi-network test this test requires 3 network, one for mgmt and the other two for NSB traffic tests We have to make sure we don't use DPDK on mgmt interface because DPDK unbinds the driver heat: convert networks to OrderedDict so we can lookups networks as well as iterate over them in consisitent order heat: and vld_id to networks for vnf_generic vnf_generic uses vld_id Virtual Link Descriptor ID to identify interfaces Add the key to the networks dict and store in Networks object implement relative path file loading in vnf_generic in multiple places we need to load a file relative to the task path, so add open_relative_file_path and modify load_vnf_model to include the scenario_cfg parameter so we have access to task_path DRAFT: heat timeout support Heat stack in CI job failed due to some Nova issue. But then apparently yardstick kept running and took 180mins to timeout https://build.opnfv.org/ci/view/bottlenecks/job/bottlenecks-compass-posca_stress_ping-baremetal-daily-master/16/console We can add a Heat create timeout and fail faster if there is an error. The question is how long should we wait for a Heat stack to deploy. We can set a default and allow override in the heat context config, if users make complicated stacks heat: get netmask and gateway from heat outputs we have do some tricky business with finding the subnet cidr and converting it into netmask vnf_generic: get vpci, driver and dpdk_port_num use a big old find command to dump all the sysfs netdev info nicely. This was re-used from autotest FCoE tests. r"""find /sys/devices/pci* -type d -name net -exec sh -c '{ grep -sH ^ \ +$1/ifindex $1/address $1/operstate $1/device/vendor $1/device/device \ +$1/device/subsystem_vendor $1/device/subsystem_device ; \ +printf "%s/driver:" $1 ; basename $(readlink -s $1/device/driver); } \ +' sh \{\}/* \; This finds all PCI devices that are network devices, then dumps all the relevant info using /bin/sh. Then we parse this into a 'netdevs' dict inside the node_dict and also convert into VNF fields we need. vnf_generic: set node name for kpis node is a dict, so we have to use node_name vnfdgen: we CANNOT use TaskTemplate.render because it does not allow for missing variables, we need to allow password for key_filename to be undefined remove default ssh password hack, once rendering is fixed add new example tc_external_ping_heat_context Change-Id: If1fe0c1a2ab0a5be17e40790a66f28f706fa44d6 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com> Signed-off-by: Edward MacGillivray <edward.s.macgillivray@intel.com>
2017-05-26Pass parameters between scenarioschenjiankun1-0/+17
JIRA: YARDSTICK-641 Allowing parameters to pass between scenarios so that the one test case can be combination of several scenarios. Change-Id: I55a00855e77d5b719a27a069a3ea195d6bbd0ef8 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-04-11standardize ssh authRoss Brattain1-58/+14
we need to be following defautl paramiko rules, first use pkey, then key_filenames (autodetecting ~/.ssh/ keys), then password We have too much boilerplate redudant code everywhere, we need to standardize on a factory function that takes a node dict. Using Python3 ChainMap we can layer overrides and defaults. VNF descriptors have to default key_filename, password to Python None. The only way to do this is to omit key values if the variable is not defined, this way the dict will not have the value and it will default to Python None Add python2 chainmap backport Updated unittest mocking to use ssh.SSH.from_node Change-Id: I80b0cb606e593b33e317c9e5e8ed0b74da591514 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-04-01Yardstick virtualenv supportchenjiankun1-1/+1
JIRA: YARDSTICK-620 Currently we recommend using docker to run yardstick. And it is hard to use virtualenv to install yardstick. So I modify install.sh in yardstick root path. It will support using virtualenv to install yardstick(including API) in linux. In this patch, I make yardstick support read yardstick configuration have priority over constants. Change-Id: I9ea1241b228532a6497451e6c8f232173ddb783e Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-03-22Merge "Yardstick: User interface for Yardstick."Jing Lu3-0/+132
2017-03-20Bugfix: netperf_bottleneck execute failed because is_same_heat_context ↵chenjiankun1-1/+0
return True JIRA: YARDSTICK-600 Bugfix: netperf_bottleneck execute failed because is_same_heat_context return True So it exit without warning. I will remove the 'return True' from it. Change-Id: I48d62bd15ec359e2121f236c48d9d349174f2c10 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>