aboutsummaryrefslogtreecommitdiffstats
path: root/yardstick/benchmark
AgeCommit message (Collapse)AuthorFilesLines
2017-02-17Update missing license headersDeepak S15-0/+128
Change-Id: I063fd37fe25754c94d164ae5a209d15b69322093 Signed-off-by: Deepak S <deepak.s@linux.intel.com>
2017-02-16Merge "Context improvement: add support for configing node environment"Rex Lee2-8/+78
2017-02-16Context improvement: add support for configing node environmentchenjiankun2-8/+78
JIRA: YARDSTICK-556 Currently we do can not config the node environment before heat create a stack. But in lots of scene, we need to config node environment before heat. So I add support for it. Change-Id: Iac1b74dc780eb40e6ab2c9cf04ed14e2b8f91ca8 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-02-16Merge "Bugfix: AttributeError: 'dict' object has no attribute 'encode'"Jing Lu1-3/+19
2017-02-16Merge "New reliability/availability testcase - IP datagram error rate and etc."Rex Lee1-0/+130
2017-02-16Merge "pylint fixes: remove redundant parens, fix comparison order"Rex Lee3-4/+4
2017-02-15Bugfix: AttributeError: 'dict' object has no attribute 'encode'chenjiankun1-3/+19
JIRA: YARDSTICK-557 If we run task with sample/ping.yaml We will encounter below error, here is the log: Traceback (most recent call last): File "/usr/local/bin/yardstick", line 11, in <module> load_entry_point('yardstick==0.1.dev0', 'console_scripts', 'yardstick')() File "/usr/local/lib/python2.7/dist-packages/yardstick/main.py", line 49, in main YardstickCLI().main(sys.argv[1:]) File "/usr/local/lib/python2.7/dist-packages/yardstick/cmd/cli.py", line 167, in main self._dispath_func_notask() File "/usr/local/lib/python2.7/dist-packages/yardstick/cmd/cli.py", line 145, in _dispath_func_notask func(CONF.category) File "/usr/local/lib/python2.7/dist-packages/yardstick/cmd/commands/task.py", line 45, in do_start Task().start(param, **kwargs) File "/usr/local/lib/python2.7/dist-packages/yardstick/benchmark/core/task.py", line 83, in start self._run(scenarios, run_in_parallel, args.output_file) File "/usr/local/lib/python2.7/dist-packages/yardstick/benchmark/core/task.py", line 131, in _run runner = run_one_scenario(scenario, output_file) File "/usr/local/lib/python2.7/dist-packages/yardstick/benchmark/core/task.py", line 410, in run_one_scenario if is_ip_addr(scenario_cfg["target"]): File "/usr/local/lib/python2.7/dist-packages/yardstick/benchmark/core/task.py", line 358, in is_ip_addr ipaddress.ip_address(addr.encode('utf-8')) AttributeError: 'dict' object has no attribute 'encode' Change-Id: Iba1570416bd8614e38c9e847de730a31d9ddedc2 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-02-14Update Openstack CLI Command in yardstick ha test framework configuration ↵tjuyinkanglin2-2/+2
files and TC052 JIRA: YARDSTICK-547 Change-Id: Idde2dc56436c9cf4b8696bfd0056ab4e47c97c9a Signed-off-by: tjuyinkanglin <14_ykl@tongji.edu.cn>
2017-02-14Merge "heat: minor fixes, don't use len > 0 and use next instead of [0]"Kubi1-2/+2
2017-02-14Merge "fix super() calls, must use explicit class name"Rex Lee5-7/+7
2017-02-14Merge "more logging fixes"Rex Lee3-23/+21
2017-02-14Merge "heat: convert open to context manager"Rex Lee1-3/+4
2017-02-14Merge "vnf_generic: replace list comprehension with generator expression"Jing Lu1-3/+3
2017-02-13New reliability/availability testcase - IP datagram error rate and etc.JingLu51-0/+130
JIRA: YARDSTICK-534 This test case uses nstat to monitor network metrics provided by the kernel in a host and calculate IP datagram error rate, ICMP message error rate, TCP segment error rate and UDP datagram error rate. Change-Id: I2fe6457bb5c95d0446c1463991ae31cc664b09f8 Signed-off-by: JingLu5 <lvjing5@huawei.com>
2017-02-09fix super() calls, must use explicit class nameRoss Brattain5-7/+7
self.__class__ is not correct. For Python 2 we have to explicitly use the class name see http://stackoverflow.com/questions/4235078/how-to-avoid-infinite-recursion-with-super Change-Id: I584ca565707b7331c1742fb33c8b524f7b7c9bf9 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-02-07more logging fixesRoss Brattain3-23/+21
don't use .format() with logging, use regular %s logginer formatter Change-Id: I1ce0d81cc3f81c35003ef453e82c57faeb46c49f Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-02-08pylint fixes: remove redundant parens, fix comparison orderRoss Brattain3-4/+4
removed redundant parens in if and while clauses use var != constant, not constant != var. Python doesn't allow for assignment in if statements, so we don't have to use the old C workarounds remove unwanted commas use raw strings for regexps with backslashes, e.g. r'\s' instead of '\s' Change-Id: I7aad645dd3d7f4b4b62f4e4510a425611c9d28f2 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-02-08vnf_generic: replace list comprehension with generator expressionRoss Brattain1-3/+3
no need to build a list, just use next Change-Id: I8a899ac538849b765f12bebda4fb8c89c84f333e Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-02-08vnf_generic: convert sshmanager to classRoss Brattain1-27/+27
@contextmanager have an issue with respect to exceptions that makes them not suitable for real usage. @contextmanager uses yield to create a generator and then uses generator.throw() to raise any exceptions. Exceptions thrown from generators loose their call stack due to the way generators work, so any exception inside a context manager is harder to debug. For this reason we don't use @contextmanager and instead always define a new class with __enter__ and __exit__. There is sample code that demonstrates the issue with @contextmanager and generator.throw() here https://gist.github.com/rbbratta/e28b6e64a4551522c3ac9815ca7f25f0 Change-Id: I5383c01f40a63e33680112f39b5bd9c858e328f1 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-01-26Merge "Introducing Generic framework to do pre-deployment VNF & Network ↵Kubi1-0/+328
service testing"
2017-01-23Merge "Record task status if running via CLI"Rex Lee1-2/+0
2017-01-20Record task status if running via CLIchenjiankun1-2/+0
JIRA: YARDSTICK-542 Currently we do not record task status when using CLI to run task. So I add this function. If status=0, task is not done. if status=1, task is done. if status=2, there is an error. Change-Id: Ib9b3c8abd233909c04f792115199250419fa8d7a Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-01-20Create API to get a list of all test caseschenjiankun1-76/+67
JIRA: YARDSTICK-456 Currently we do not have a API to get a list of all test cases; Currently the test case info is from the comment; So I create a API to get a list of all test cases; And create a 'description' attribute to record info of a test case; And use the CLI call this API; Change-Id: Ife800600446683664097835c7b9f11899c85771d Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-01-20Merge "Bugfix:can not run a test suite if not under yardstick root path"Jing Lu1-0/+2
2017-01-19Introducing Generic framework to do pre-deployment VNF & Network service testingDeepak S1-0/+328
This patch introduces the framework which is aligned with ETSI-TST001 This patch adds: 1. NetworkServiceTestCase introduces following functions --> setup --> Verify if infrastructure mapping can meet topology --> Load VNF models --> Fill traffic profile with information from topology --> Provision VNFs --> Run experiment (traffic) --> run -> Yardstick calls run() at intervals defined in the yaml and produces timestamped samples --> teardown --> Stop VNFs 2. TrafficProfile is a generic class to get traffic profile for a given testcase and select the traffic generator for testcase. 3. QueueFileWrapper is a class to send/recive cmds to vnf 4. GenericVNF is a generic class to instantiate VNF 5. GenericTrafficGen is a generic class to run/listen/verify traffic. JIRA: YARDSTICK-483 Change-Id: Ic453c917d34dcb508a7f3afb459011da85f6402e Signed-off-by: Deepak S <deepak.s@linux.intel.com>
2017-01-19Introducing Standalone context for running test in non-managed environment.Deepak S1-0/+116
This patch introduces standalone virtualization context to deploy/undeploy NFVi infrastructure to run the VNF Supported NFVi Type: - vswitch - ovs - ovs-dpdk - sr-iov - testpmd - linuxbridge This patches inits the function stubs to enable the standalone context. Actual deploy/undeploy code will be added in later check-in v2: Added unit tests to keep test coverage :) JIRA: YARDSTICK-479 Change-Id: I6ab3ac3335f40eabc4efb0af7d5addc20c122d65 Signed-off-by: Deepak S <deepak.s@linux.intel.com>
2017-01-19Merge "Bugfix: context not found for server ''node1.LF''"Kubi1-1/+2
2017-01-19Bugfix: context not found for server ''node1.LF''chenjiankun1-1/+2
JIRA: YARDSTICK-540 We are adding dynamically generated suffix to server name in the previous patch. But in that patch we just change the 'target', 'host' in nodes, but not 'node1', 'node2' ... This patch will change all this. Change-Id: Ic7c82c323dea24a40f1fe30871603d30c2689f67 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-01-19Re-factor Node.py to use better python inbuilt functionsDeepak S1-21/+35
1. don't use sys.exit() to exit when there is an exception, it will hide the underlying error 2. use the Abstract Base Classes for type checking. 3. don't have to build list, can use next JIRA: YARDSTICK-541 Change-Id: Id4485acb21e7e02bbc22d3e689cbf0699363098a Signed-off-by: Deepak S <deepak.s@linux.intel.com>
2017-01-17heat: minor fixes, don't use len > 0 and use nextRoss Brattain1-2/+2
instead of [0] Change-Id: I1785b7ca3543a8d44eb68de61f4f29063a9c61c7 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-01-17heat: convert open to context managerRoss Brattain1-3/+4
Change-Id: I5ce6c77839f4104d305d66238bb2c6cbdc39a28c Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-01-16Use """ to replace ''' in docstringchenjiankun41-153/+153
JIRA: YARDSTICK-525 For consistency, we always use """triple double quotes""" around docstrings. Change-Id: I47a20bbd8b55bc544b4841ea4006929af0a044ac Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-01-14Bugfix:can not run a test suite if not under yardstick root pathchenjiankun1-0/+2
JIRA: YARDSTICK-530 Currently we can not run a test suite if we are not in yardstick root path. The reason is that the file in test suite config file use relative path. So I change it to absolute path when run. Change-Id: I62758bc67f466ac794d339b597562b3be05574fb Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-01-12Add support for Python 3Ross Brattain64-178/+345
Porting to Python3 using Openstack guidelines: https://wiki.openstack.org/wiki/Python3 This passes unittests on Python 3.5 and passes opnfv_smoke suite Updates: use six for urlparse and urlopen fix exception.message attribute removal run unittests on python3 use unitest.mock on python 3 fix open mock for vsperf fix float division by using delta/eplison comparison use unicode in StringIO use plugin/sample_config.yaml relative path from test case fixed apexlake unittests upgraded to mock 2.0.0 to match python3 unittest.mock features fixed flake8 issues implement safe JSON decode with oslo_serialization.jsonutils.dump_as_bytes() implement safe unicode encode/decode with oslo_utils.encodeutils heat: convert pub key file from bytes to unicode pkg_resources returns raw bytes, in python3 we have to decode this to utf-8 unicode so JSON can encode it for heat template JIRA: YARDSTICK-452 Change-Id: Ib80dd1d0c0eb0592acd832b82f6a7f8f7c20bfda Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-01-12heat: fix key_uuid formatRoss Brattain1-5/+6
The .format() won't work, since it can't slice the uuid We have to convert the uuid to string before we slice it. I thought .format() would implicitly call __str__() before applying width, but that is not the case. 'files/yardstick_key-{:.{width}}'.format(self.key_uuid, width=8)) We also need to define a constant short uuid lenght, we can't hardcode the length to 8 everywhere. Create a helper function to standardize the generation of the short key uuid and use that helper function everywhere Change-Id: I59e051bfe697587e967f93f5b8f209e0e7daa5c7 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-01-11Merge "associate an uuid to yardstick_key and yardstick_key.pub"Rex Lee1-7/+8
2017-01-10associate an uuid to yardstick_key and yardstick_key.pubJingLu51-7/+8
JIRA: YARDSTICK-527 This work is to support yardstick parallel tasks in the future. Currently, the RSA key we generated and used to access the VM is named 'yardstick_key'. If more than two tasks are running paralleled, the later 'yardstick_key' will cover the former. We want associate an uuid to identify differnets for each tasks. So the key files won't conflict. The first 8 digits will be used, as there is no need to used a full-length uuid. Change-Id: If8eaf47ae527cf9b3bd50f37ab3051fbdccf5f03 Signed-off-by: JingLu5 <lvjing5@huawei.com>
2017-01-10Merge "Yardstick framework concurrent support"Jing Lu1-1/+28
2017-01-10Merge "subprocess.call para stdout=PIPE is risky"Rex Lee1-2/+4
2017-01-09Yardstick framework concurrent supportchenjiankun1-1/+28
JIRA: YARDSTICK-528 Currently yardstick framework can not support run the same test case at the same time. But actually we need to support it. The reason why framework can't support it is that openstack do not allow to create stack with the same name. So I use the task_id to make the stack different. Change-Id: I9e853793650066dfc56606464f7826f330a1401c Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2016-12-30Split Yardstick CLI with Yardstick core logicchenjiankun6-0/+918
JIRA: YARDSTICK-511 We need to unify yardstick entry. Now the solution is using CLI call API as nova do. This is the first step: coupling the yardstick core logic from CLI. Moving the core logic to yardstick/benchmark/core and the CLI using a object to call yardstick core logic. Change-Id: I84f10d2134635880c281cc63212a8533f2dd7d4e Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2016-12-21subprocess.call para stdout=PIPE is riskyRyan.RCS1-2/+4
1.In 'def run' function of parser.py file, subprocess.call parameter stdout= subprocess.PIPE is risky, so I changed the function from 'call' to 'popen' 2.updated sample/tosca.ymal because the version of that file is old. JIRA: YARDSTICK-473 Change-Id: Ie242e77eed6fdc2849394a3f170e40a0dd2be632 Signed-off-by: Ryan.RCS <lihainong@huawei.com>
2016-12-12Merge "Increase Ping scenario ssh timeout limit to 600 seconds"Rex Lee1-1/+1
2016-12-12Merge "use context manager for stdin files and use _put_file_shell"Rex Lee25-121/+126
2016-12-12Merge "Add support for Storperf job status"Rex Lee1-28/+27
2016-12-05use context manager for stdin files and use _put_file_shellRoss Brattain25-121/+126
requires https://gerrit.opnfv.org/gerrit/#/c/25183/ use new ssh method _put_file_shell to upload files. We have to use _put_file_shell because we rely on ~/ path expansions. Eventually we should move to remote absolute paths so we can use sftp upload. For ssh.execute() replace open() with context manager context managers were invented partly to control freeing resources. Opening files without closing them will leak file descriptors. The old standard method for closing files: f = open('data.txt') try: data = f.read() finally: f.close() was replaced with a context manager with open('data.txt') as f: data = f.read() Reference: Raymond Hettinger's Pycon 2013 presentation: https://speakerdeck.com/pyconslides/transforming-code-into-beautiful-idiomatic-python-by-raymond-hettinger-1 Video: https://youtu.be/OSGv2VnC0go?t=2522 Always use context managers for files Update: rebased now that _put_file_shell was merged Change-Id: Iabfc0e43aa3b7766d7c658115e13d21c31efb2a9 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2016-12-01Merge "vsperf: Enhanced vswitchperf configuration"Rex Lee1-24/+43
2016-11-30switch logging to proper usageRoss Brattain28-87/+87
The logging methods do string interpolation themselves From the reference: https://docs.python.org/2/library/logging.html#logging.Logger.debug Logger.debug(msg, *args, **kwargs) Logs a message with level DEBUG on this logger. The msg is the message format string, and the args are the arguments which are merged into msg using the string formatting operator. (Note that this means that you can use keywords in the format string, together with a single dictionary argument.) There are two keyword arguments in kwargs which are inspected: exc_info which, if it does not evaluate as false, causes exception information to be added to the logging message. If an exception tuple (in the format returned by sys.exc_info()) is provided, it is used; otherwise, sys.exc_info() is called to get the exception informatio The reason logging does string interpolation itselfs is to implement deferred interpolation. String interpolation involves evaluating arguments, so it can introduce significant computation. The logging module tries to be smart about deferring interpolation until the last possible moment. The logging methods check isEnabledFor for the log level and won't interpolate if the level is not enabled. https://github.com/python/cpython/blob/2.7/Lib/logging/__init__.py#L1178 def warning(self, msg, *args, **kwargs): if self.isEnabledFor(WARNING): self._log(WARNING, msg, args, **kwargs) logging actually waits to interpolate the string in LogRecord.getMessage() https://github.com/python/cpython/blob/2.7/Lib/logging/__init__.py#L328 if self.args: msg = msg % self.args Change-Id: Ie09efe0a66881e19bd8119caa376075e605627a2 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2016-11-30vsperf: Enhanced vswitchperf configurationMartin Klozik1-24/+43
Original implementation of vsperf specific class was changed to relfect recent vsperf changes. It is now possible to modify any of vsperf's configuration parameters via --test-param CLI argument. It means, that it is possible to write a yardstick TC, which will define all required vsperf configuration inside the YAML TC definition. Vsperf documentation related to yardstick usage and sample TC files are located inside vsperf repository and they were updated by a separate patch. JIRA: VSPERF-422 Change-Id: I978d1c85ffeb3c90d9d47a20c6c0e0f68b437123 Signed-off-by: Martin Klozik <martinx.klozik@intel.com> Reviewed-by: <sunshine.wang@huawei.com> Reviewed-by: <lvjing5@huawei.com> Reviewed-by: <jean.gaoliang@huawei.com> Reviewed-by: <david.j.chou@intel.com>
2016-11-30Merge "netperf_node test case improvement."Rex Lee3-22/+58