summaryrefslogtreecommitdiffstats
path: root/tests/unit/benchmark
AgeCommit message (Collapse)AuthorFilesLines
2017-07-17Merge "test_pktgen_dpdk_throughput: speedup unittest, mock time.sleep()"Ross Brattain1-8/+9
2017-07-17test_heat: ipaddress expects unicode inputsRoss Brattain1-14/+14
weird error E AddressValueError: '10.20.0.0/15' does not appear to be an IPv4 or IPv6 network. Did you pass in a bytes (str in Python 2) instead of a unicode object? I guess we need to convert the stack.outputs mock to unicode FAILED tests/unit/benchmark/contexts/test_heat.py:137 (HeatContextTestCase.test_add_server_port) self = <tests.unit.benchmark.contexts.test_heat.HeatContextTestCase testMethod=test_add_server_port> def test_add_server_port(self): network1 = mock.MagicMock() network1.vld_id = 'vld111' network2 = mock.MagicMock() network2.vld_id = 'vld777' self.test_context.name = 'foo' self.test_context.stack = mock.MagicMock() self.test_context.networks = { 'a': network1, 'c': network2, } self.test_context.stack.outputs = { 'b': '10.20.30.45', 'b-subnet_id': 1, 'foo-a-subnet-cidr': '10.20.0.0/15', 'foo-a-subnet-gateway_ip': '10.20.30.1', 'b-mac_address': '00:01', 'b-device_id': 'dev21', 'b-network_id': 'net789', 'd': '40.30.20.15', 'd-subnet_id': 2, 'foo-c-subnet-cidr': '40.30.0.0/18', 'foo-c-subnet-gateway_ip': '40.30.20.254', 'd-mac_address': '00:10', 'd-device_id': 'dev43', 'd-network_id': 'net987', } server = mock.MagicMock() server.ports = OrderedDict([ ('a', {'stack_name': 'b'}), ('c', {'stack_name': 'd'}), ]) expected = { "private_ip": '10.20.30.45', "subnet_id": 1, "subnet_cidr": '10.20.0.0/15', "network": '10.20.0.0', "netmask": '255.254.0.0', "gateway_ip": '10.20.30.1', "mac_address": '00:01', "device_id": 'dev21', "network_id": 'net789', "network_name": 'a', "local_mac": '00:01', "local_ip": '10.20.30.45', "vld_id": 'vld111', } > self.test_context.add_server_port(server) tests/unit/benchmark/contexts/test_heat.py:186: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ yardstick/benchmark/contexts/heat.py:307: in add_server_port network_name, port['stack_name'], self.stack.outputs) yardstick/benchmark/contexts/heat.py:315: in make_interface_dict subnet_ip = ipaddress.ip_network(subnet_cidr) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ address = '10.20.0.0/15', strict = True def ip_network(address, strict=True): """Take an IP string/int and return an object of the correct type. Args: address: A string or integer, the IP network. Either IPv4 or IPv6 networks may be supplied; integers less than 2**32 will be considered to be IPv4 by default. Returns: An IPv4Network or IPv6Network object. Raises: ValueError: if the string passed isn't either a v4 or a v6 address. Or if the network has host bits set. """ try: return IPv4Network(address, strict) except (AddressValueError, NetmaskValueError): pass try: return IPv6Network(address, strict) except (AddressValueError, NetmaskValueError): pass if isinstance(address, bytes): raise AddressValueError( '%r does not appear to be an IPv4 or IPv6 network. ' 'Did you pass in a bytes (str in Python 2) instead of' > ' a unicode object?' % address) E AddressValueError: '10.20.0.0/15' does not appear to be an IPv4 or IPv6 network. Did you pass in a bytes (str in Python 2) instead of a unicode object? ../../yardstick/yardstick_venv/lib/python2.7/site-packages/ipaddress.py:199: AddressValueError Change-Id: Ie3b087a26a054203573eaa9b13c3e90152bba6a9 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-07-17Merge "test_attacker_baremetal: don't run local commands"Ross Brattain1-14/+12
2017-07-17Merge "Kubernetes (k8s) support"Jing Lu1-0/+165
2017-07-17Merge "add network info to topology"Rex Lee5-194/+570
2017-07-17Kubernetes (k8s) supportchenjiankun1-0/+165
JIRA: YARDSTICK-682 We decide to support k8s in E release. We need to discuss with openretriver team and then rewrite the ping test case under k8s as the first step. Change-Id: I3f81ebca8de5c1f3a8b7d42581cd7342dc320239 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-07-13test_attacker_baremetal: don't run local commandsRoss Brattain1-14/+12
these tests were failing to mock subprocess.check_output and thus were trying to run sudo commands on the local system. This is dangerous. Add the subprocess mock. Also mock the LOG object so we don't print bogus Runtime error tracebacks in the unittest logs when we test assertRaises() Change-Id: I01535f9952fbd95ce2f5972b641c51ff836e7e8c Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-07-13test_monitor_command: mock LOG so we don't output errorRoss Brattain1-2/+6
Don't print fake tracebacks to the unittest logs Change-Id: I8a468b8c6566f02be88a9dd222567c14c66b0956 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-07-14Merge "test_monitor_multi: speedup unittests, sleep less"Ross Brattain1-1/+1
2017-07-13add network info to topologyRoss Brattain5-194/+570
we need to know which network each port is connected to so we can find VLAN or VXLAN ID. To do this we implement a new method for Contexts, Context.get_network(). This method is similar to Context.get_server(), it searches for a given network name in all the contexts. From this we generate a context_cfg["networks"] dict that stores all the network info for the nodes in the scenario. Then when we generate the topology for VNFD, we can lookup a given network by the vld_id and get the network_type, segmentation_id, etc. Then if we need to for example generated traffic on a given VLAN or VXLAN, we have this info available. Define default nd_route_tbl for ACL VNF we need default empty nd_route_tbl for IPv6 route. Change-Id: I9f9cfbd6acabeb4ae4675ca7354390efa57b29e7 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com> Signed-off-by: Edward MacGillivray <edward.s.macgillivray@intel.com>
2017-07-13test_monitor_multi: speedup unittests, sleep lessRoss Brattain1-1/+1
Change-Id: I306e8e5525c3722179a5321988cad76190b93951 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-07-13test_storperf: speedup unittest, use time.sleep(0)Ross Brattain1-2/+2
Change-Id: I3923d75c1ae7974c7803b0ad6997a5ad0fece11d Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-07-13test_pktgen_dpdk_throughput: speedup unittest, mock time.sleep()Ross Brattain1-8/+9
Change-Id: I74bbdee1a503ef9bad32db7d4444921f6730f465 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-07-12test_nstat: decrease sleep time for unittestRoss Brattain1-3/+3
there is no reason to sleep for a whole minute during unittest Change-Id: I206229bcaa7a50af1859f7b04e521955bb6217d4 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-07-06move flatten dict key to common utilsrexlee87764-13/+20
So it can easily be used by other testcase to unify result JIRA: YARDSTICK-702 Change-Id: Id4fde38a9a0c2a87a6c870bdb7b0c8f3a3b371ac Signed-off-by: rexlee8776 <limingjiang@huawei.com>
2017-06-24Merge "Yardstick output format unified"Jing Lu2-3/+2
2017-06-22Merge "Acquire NSB specific data from Heat."Ross Brattain2-5/+153
2017-06-22HA testcase improvementJingLu51-0/+2
This patch improve HA test case in the following aspects: 1. the "GeneralHA" type now will check if the target service process in the controller node. 2. support ignore server certificate 3. add debug log for recovering service failed 4. improve method to kill keystone process Change-Id: I9ae7ab54391fe41d5d7f3e4951a7ac2e3ba75968 Signed-off-by: JingLu5 <lvjing5@huawei.com>
2017-06-22Yardstick output format unifiedchenjiankun2-3/+2
JIRA: YARDSTICK-658 Currently the yardstick have three dispatcher: file, influxdb, mongodb. (influxdb using API to get result and mongodb using testAPI to get result) But their output format is different. It is hard to use. In this patch, make all dispatchers using the same data source. And make the output format of file and influxdb unified. As for mongodb, since it is related to testAPI, so I make it push data every test case. The unified output format is: http://paste.openstack.org/show/610125/ Change-Id: I854ac4f03e6f904469b07b0c924c7d850545ae5b Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-06-20Acquire NSB specific data from Heat.Edward MacGillivray2-5/+153
First we add mac_address, subnet_cidr to Heat template outputs Then we convert those into a form for NSB and add vld_id. NSB also requires PCI Bus ID, kernel driver and dpdk_port_num. We get this by ssh-ing into instance and dumping sysfs We also need to fix allow for ssh key auth, and implement relative path file loading so NSB can find all its YAML files JIRA: YARDSTICK-580 Change history: don't hide heat create tracebacks we need tracebacks for debug vnf_generic: add task_path to scenario so we can load relative paths for vnf_generic we want to be able to load yaml relative to the task path For example: traffic_profile: ../../traffic_profiles/fixed.yaml topology: ping_tg_topology.yaml # TODO: look in relative path where the tc.yaml is found These need to be relative to samples/vnf_samples/nsut/ping/tc_ping_heat_context.yaml Add a scenario["task_path"] entry heat: log actual exception vnf_generic: replace list with set and iterate over values() some general refactors to remove redundact lookups and type conversions heat: provide mac_address, device_id and network_id from outputs We may need more information to dynamically determine test topology. Towards this end return more info in the heat template. We can return mac_address, device_id and network_id. Once we have this info we can add it to the context_cfg as an interfaces dict. add sample vnf ping multi-network test this test requires 3 network, one for mgmt and the other two for NSB traffic tests We have to make sure we don't use DPDK on mgmt interface because DPDK unbinds the driver heat: convert networks to OrderedDict so we can lookups networks as well as iterate over them in consisitent order heat: and vld_id to networks for vnf_generic vnf_generic uses vld_id Virtual Link Descriptor ID to identify interfaces Add the key to the networks dict and store in Networks object implement relative path file loading in vnf_generic in multiple places we need to load a file relative to the task path, so add open_relative_file_path and modify load_vnf_model to include the scenario_cfg parameter so we have access to task_path DRAFT: heat timeout support Heat stack in CI job failed due to some Nova issue. But then apparently yardstick kept running and took 180mins to timeout https://build.opnfv.org/ci/view/bottlenecks/job/bottlenecks-compass-posca_stress_ping-baremetal-daily-master/16/console We can add a Heat create timeout and fail faster if there is an error. The question is how long should we wait for a Heat stack to deploy. We can set a default and allow override in the heat context config, if users make complicated stacks heat: get netmask and gateway from heat outputs we have do some tricky business with finding the subnet cidr and converting it into netmask vnf_generic: get vpci, driver and dpdk_port_num use a big old find command to dump all the sysfs netdev info nicely. This was re-used from autotest FCoE tests. r"""find /sys/devices/pci* -type d -name net -exec sh -c '{ grep -sH ^ \ +$1/ifindex $1/address $1/operstate $1/device/vendor $1/device/device \ +$1/device/subsystem_vendor $1/device/subsystem_device ; \ +printf "%s/driver:" $1 ; basename $(readlink -s $1/device/driver); } \ +' sh \{\}/* \; This finds all PCI devices that are network devices, then dumps all the relevant info using /bin/sh. Then we parse this into a 'netdevs' dict inside the node_dict and also convert into VNF fields we need. vnf_generic: set node name for kpis node is a dict, so we have to use node_name vnfdgen: we CANNOT use TaskTemplate.render because it does not allow for missing variables, we need to allow password for key_filename to be undefined remove default ssh password hack, once rendering is fixed add new example tc_external_ping_heat_context Change-Id: If1fe0c1a2ab0a5be17e40790a66f28f706fa44d6 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com> Signed-off-by: Edward MacGillivray <edward.s.macgillivray@intel.com>
2017-06-05Merge "Pass parameters between scenarios"Rex Lee3-0/+73
2017-05-27Imporve monitor_process pass criteriaJingLu56-17/+17
JIRA: YARDSTICK-660 The monitor func()'s criteria in the monitor_process.py now is whether at least one process of the specific controller node service is recovered. But in reality is more resonable to use whether processes have been recoverd to it's original amount. This patch is aiming at improving the isssue Change-Id: I950ce2a89555801b96092735b0d670e892049927 Signed-off-by: JingLu5 <lvjing5@huawei.com>
2017-05-26Pass parameters between scenarioschenjiankun3-0/+73
JIRA: YARDSTICK-641 Allowing parameters to pass between scenarios so that the one test case can be combination of several scenarios. Change-Id: I55a00855e77d5b719a27a069a3ea195d6bbd0ef8 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-05-16fix multi-monitor unitest hangRoss Brattain1-8/+8
need to mock SSH.from_node().execute.return_value because we switch most SSH calls to use from_node() factory function Change-Id: Ibea525c2fcc243d41de8ccbcd40bc98532d69006 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-05-16Merge "Add a new monitor type: MultiMonitor that can run any number of other ↵Ross Brattain1-0/+61
monitors at the same time."
2017-05-12Create flavor from heat contextDanielMartinBuckley2-3/+94
JIRA: YARDSTICK-582 Create a customizable flavor via heat context. All heat parameters are configurable including Core Affinity. The default flavor name is XXXX-flavor where XXXX is stackname. Flavor attributes are taken from the heat context file. If a flavor attribute is not used it takes default attribute value. If flavor name is not specified it uses the server name + "-flavor" or stack-name + "-flavor". Compute node specific attributes are configurable via "extra_specs" attribute. See https://docs.openstack.org/admin-guide/compute-flavors.html for details. Change-Id: If4015970b889b0b95bfa8eba9491ebf31e92f2c7 Signed-off-by: DanielMartinBuckley <daniel.m.buckley@intel.com>
2017-05-11Merge "Extend TC008 to run pktgen-dpdk inside VM Need a fast path inside VM ↵Rex Lee2-6/+192
to verify full throughput of SRIOV and OVS-dpdk."
2017-05-10Merge "pass user_data to heat template"Ross Brattain1-0/+21
2017-05-04Merge "Bugfix: Local Openstack Operation in HA test frameworks"Jing Lu1-0/+32
2017-05-04Bugfix: Local Openstack Operation in HA test frameworkstjuyinkanglin1-0/+32
JIRA: YARDSTICK-635 Change-Id: Ic27517714db9325e7a3b1ef623c49af61c36b2b5 Signed-off-by: tjuyinkanglin <14_ykl@tongji.edu.cn>
2017-05-03fix ha issue when run tc050~tc054 in cirexlee87761-3/+3
JIRA: YARDSTICK-634 Change-Id: I46681c7e8afe391eef9c5309470028167e911950 Signed-off-by: rexlee8776 <limingjiang@huawei.com>
2017-05-01Extend TC008 to run pktgen-dpdk inside VMJing Zhang2-6/+192
Need a fast path inside VM to verify full throughput of SRIOV and OVS-dpdk. Update 1: Change newly added file names to avoid conflict Update 2: Add more unit test cases Update 3: Fix default parameter typo for testpmd Update 4: Adapted to the pktgen-dpdk prompt change from "Pktgen>" to "Pktgen:/>", now just expect "Pktgen" Update 5: Per comment, merge common functions between latency and throughput tests to utils.py Update 6: Per comment, seperate the test case from TC008 to a new test case TC077 Change-Id: I1f7471d4ba77636a3a66c79c2652578321312185 JIRA: YARDSTICK-614 Signed-off-by: Jing Zhang <jing.c.zhang@nokia.com>
2017-04-24Merge "standardize ssh auth"Rex Lee33-289/+289
2017-04-23Merge "Add SRIOV support"Kubi2-2/+30
2017-04-18Add SRIOV supportJing Zhang2-2/+30
A generic provider network solution is introduced. To identify whether a network specified in the test case is a provider network new attributes are introduced in the test case network section: networks: test-net: cidr: '192.168.1.0/24' provider: "sriov" physical_network: 'physnet1' If the "provider" attribute is present, the network is an existing provider network. If the value is "sriov", binding:vnic_type=direct is added to the interface in the heat deployment template. In orchestrator/heat.py, the interface creating functions are given a new parameter that tells if the network in use is a provider network. The benchmark/contexts/model.py is changed to store the value of the provider attribute from the test case and function calls to port creation is updated with the provider parameter. The same change is made in contexts/heat.py as well. Also calls for creating a new tenant network is replaced for creating a new provider network if the provider attribute is present. Update-1: Change test_model.py Update-2: Per comment, change comment style to """" Update-3: Change test_heat.py Update-4: Add unit test cases to pass coverage test Update-5: Add SRIOV provider network example in opnfv_yardstick_tc008.yaml Update-6: Per comment, remove empty line in orchestrator/test_heat.py Update-7: Per comment, change comment lines in orchestrator/test_heat.py Update-8: Add more unit test cases to pass coverage test Update-9: Change to create SRIOV provider network on the fly so as to support co-current test runs Update-10: Per comment, init physical_network to 'physnet1' Change-Id: I76004c4fcc9bffcfd8ed021fd647e0cecb346ef4 JIRA: YARDSTICK-612 Signed-off-by: Jing Zhang <jing.c.zhang@nokia.com>
2017-04-12Add a new monitor type: MultiMonitor that can run any number of other ↵LiHuan1-0/+61
monitors at the same time. JIRA: YARDSTICK-397 Change-Id: Ic5cb79f0820029e306373abead1ea43fac9abee2 Signed-off-by: HuanLi <lihuansse@tongji.edu.cn>
2017-04-11standardize ssh authRoss Brattain33-289/+289
we need to be following defautl paramiko rules, first use pkey, then key_filenames (autodetecting ~/.ssh/ keys), then password We have too much boilerplate redudant code everywhere, we need to standardize on a factory function that takes a node dict. Using Python3 ChainMap we can layer overrides and defaults. VNF descriptors have to default key_filename, password to Python None. The only way to do this is to omit key values if the variable is not defined, this way the dict will not have the value and it will default to Python None Add python2 chainmap backport Updated unittest mocking to use ssh.SSH.from_node Change-Id: I80b0cb606e593b33e317c9e5e8ed0b74da591514 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-04-05Bugfix: fix query job status in TC074JingLu51-2/+2
JIRA: YARDSTICK-621 The storperf job status query in TC074 didn' work properly. The cause is the "type: status" is not passed in the URL. This patch also update storperf VM image to xenial. Change-Id: Idd9f501416b24612f6045a57ba2a95e2ed3a9572 Signed-off-by: JingLu5 <lvjing5@huawei.com>
2017-03-31Store packetsize in influxdbJing Zhang1-3/+3
This allows calculating throughput in Gbps. Add: test_pktgen.py Change-Id: I7e2132ec71985c14570ef18b952b2264ebf7abbc JIRA: YARKSTICK-611 Signed-off-by: Jing Zhang <jing.c.zhang@nokia.com>
2017-03-23model: pylint fixesRoss Brattain1-1/+1
remove redundnant parens replace type with isinstance fix import order Change-Id: I0407cbbf6993290f392f17c398827603b5bc1ebd Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-03-22Merge "Yardstick: User interface for Yardstick."Jing Lu1-0/+72
2017-03-17Yardstick: User interface for Yardstick.rajesh_4k1-0/+72
Currently Yardstick doesnt have any UI which gives detail analysis of the test-results. This commit generates a HTML page after the execution of a command "yardstick report generate <task-ID> <TC-name>" which intern can be executed after the execution of test-case. Used: Highcharts.js for the graphs. JIRA: YARDSTICK-280 Change-Id: Ic98cc348719f3922bff178f52e7944a4a931763a Signed-off-by: Rajesh K <4k.rajesh@gmail.com>
2017-03-15pass user_data to heat templateRoss Brattain1-0/+21
We had code to support adding user_data but the Server model was not connected. The string formatting of the user_data is left as an exercise for the reader. Change-Id: I298657b93de67e19436bca616b119df636dacce2 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-03-13Add CPU pinning support for node contextJingLu51-12/+8
JIRA: YARDSTICK-573 Since the yardstick framework now has supported an improved node type context, this patch adds support for VM vcpu pinning ability in the node type context. It provides several scripts that can be used to configurate the controller and compute nodes. Change-Id: If2c6e7b1b85ff78b9d2a5997bf03bdc6877aaf74 Signed-off-by: JingLu5 <lvjing5@huawei.com>
2017-03-09Bugfix: Failed executing command: 'free -s 1 -c 10'chenjiankun2-15/+36
JIRA: YARDSTICK-585 In CI when run tc070, there is a error: Failed executing command: 'free -s 1 -c 10' Here it is the log: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/yardstick/benchmark/runners/duration.py", line 69, in _worker_process method(data) File "/usr/local/lib/python2.7/dist-packages/yardstick/benchmark/scenarios/compute/memload.py", line 126, in run result.update(self._get_mem_usage()) File "/usr/local/lib/python2.7/dist-packages/yardstick/benchmark/scenarios/compute/memload.py", line 116, in _get_mem_usage result = self._execute_command(cmd) File "/usr/local/lib/python2.7/dist-packages/yardstick/benchmark/scenarios/compute/memload.py", line 70, in _execute_command cmd, stderr) RuntimeError: ('Failed executing command: ', 'free -s 1 -c 10',u"free: seconds argument `1' failed\n") And it is a bug of free. the -c option should in front of -s, so change the position will solve this problem. Also it has another bug: 'KeyError', there no 'cached' keyword, so I change it to 'buff/cache'. Change-Id: I0ca16e8d8cc11c6a3b2f364cadbdb3ea367eee53 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-02-26heat context support affinity and anti-affinityrexlee87762-1/+18
JIRA: YARDSTICK-566 Current Heat context support affinity and availability arguments but not support affinity and anti-affinity. Enhance Heat context to support affinity and anti-affinity: 1. can create heat server group with affinity/anti-affinity 2. each server could be specified which server group they are in Change-Id: I46e7376fd116c6e109cb5dcb1c168460918e6d43 Signed-off-by: rexlee8776 <limingjiang@huawei.com>
2017-02-17Update missing license headersDeepak S6-0/+48
Change-Id: I063fd37fe25754c94d164ae5a209d15b69322093 Signed-off-by: Deepak S <deepak.s@linux.intel.com>
2017-02-16Merge "Context improvement: add support for configing node environment"Rex Lee1-0/+96
2017-02-16Context improvement: add support for configing node environmentchenjiankun1-0/+96
JIRA: YARDSTICK-556 Currently we do can not config the node environment before heat create a stack. But in lots of scene, we need to config node environment before heat. So I add support for it. Change-Id: Iac1b74dc780eb40e6ab2c9cf04ed14e2b8f91ca8 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-02-16Merge "Bugfix: AttributeError: 'dict' object has no attribute 'encode'"Jing Lu1-0/+24