aboutsummaryrefslogtreecommitdiffstats
path: root/yardstick/benchmark
AgeCommit message (Collapse)AuthorFilesLines
2016-01-15fix some bug in ipv6 to make it run in cikubi7-31/+167
JIRA:YARDSTICK-187 Change-Id: Ia15d17afdef145f7b230a8a4d25a61eed5cdfd76 Signed-off-by: kubi <jean.gaoliang@huawei.com>
2016-01-15HeatContext model update to match heat code updateQiLiang3-4/+6
In Heat Liberty release OS::Nova::Server will always use the user pre-configured in the image (e.g. "fedora" for stock Fedora cloud images, "ubuntu" for stock Ubuntu cloud images, "cloud-user" for stock CentOS cloud images etc) Change all ec2-user -> ubuntu Add admin-user in Heat model for backwards compatibility. Refer below links for detalis: https://etherpad.opnfv.org/p/yardstick_release_b_troubleshooting https://github.com/openstack/heat/commit/e423bec7f10b0f5d07f05d195b3b7860f6bceb00 http://blog.scottlowe.org/2015/04/23/ubuntu-openstack-heat-cloud-init/ JIRA: - Change-Id: I6b8b2b21daf113a3a86aee1126b0c3e74737ef4f Signed-off-by: QiLiang <liangqi1@huawei.com>
2016-01-14Update results for Test Cases 6, 7, 20, 21Vincenzo Riccobene4-8/+16
JIRA: YARDSTICK-219 Change-Id: I06143f074592e05cd17b740eb9751414a56c9fde Signed-off-by: Vincenzo Riccobene <vincenzo.m.riccobene@intel.com> Signed-off-by: Vincenzo Riccobene <vincenzox.m.riccobene@intel.com>
2016-01-13Fix flake8 errorsJo¶rgen Karlsson4-7/+8
Change-Id: I14f21092f5e97abf0629a92a27062846a6d3130b JIRA:- Signed-off-by: Jo¶rgen Karlsson <jorgen.w.karlsson@ericsson.com>
2016-01-12Workround for openstack nova bugQiLiang1-2/+21
Notes: if placement_groups policy is "availability" and > 2 servers in this group may still have this issue. JIRA: YARDSTICK-200 Signed-off-by: QiLiang <liangqi1@huawei.com>
2016-01-11Implementation of opnfv-yardstick-tc021 in YardstickVincenzo Riccobene1-0/+92
Includes - Yardstick Scenario for integration with ApexLake - Yardstick Task .yaml file - Documentation JIRA: YARDSTICK-219 Change-Id: I0554d0a211392902207ba9ceccf0b98dc3c2cdf1 Signed-off-by: Vincenzo Riccobene <vincenzox.m.riccobene@intel.com>
2016-01-11Implementation of opnfv-yardstick-tc020 in YardstickVincenzo Riccobene1-0/+85
Includes - Yardstick Scenario for integration with ApexLake - Yardstick Task .yaml file - Documentation JIRA: YARDSTICK-219 Change-Id: Ibbba1d9864e788fb6c81cc00c503e9f69e885651 Signed-off-by: Vincenzo Riccobene <vincenzox.m.riccobene@intel.com>
2016-01-11Implementation of yardstick-tc007 in YardstickVincenzo Riccobene1-0/+91
Includes - Yardstick Scenario for integration with ApexLake - Yardstick Task .yaml file - Documentation JIRA: YARDSTICK-219 Change-Id: Ifa2555336098e68d0fad8045e2f759aed587ad92 Signed-off-by: Vincenzo Riccobene <vincenzox.m.riccobene@intel.com>
2016-01-11Implementation of yardstick-tc006 in YardstickVincenzo Riccobene1-0/+85
Includes - Yardstick Scenario for integration with ApexLake - Yardstick Task .yaml file - Documentation JIRA: YARDSTICK-219 Change-Id: Iabde8fa63f346cf1e4a02691f22d1761de79a239 Signed-off-by: Vincenzo Riccobene <vincenzox.m.riccobene@intel.com>
2016-01-10Support run cyclictest on BareMetalQiLiang1-14/+91
JIRA: YARDSTICK-122 Change-Id: I8144215059a9abea08314a4c1e6a733dcdf0df53 Signed-off-by: QiLiang <liangqi1@huawei.com>
2016-01-10The secondi HA test case-shutdown controllerwym_libra16-69/+303
1) add "attacker_baremetal.py" for fault injection 2) modify the monitor to excute on remote node after ssh connection 3) move all shell scripts together JIRA: YARDSTICK-182 Change-Id: Ibb9dc908224ddb8b99a0140b75c1a046503f6dfb Signed-off-by: wym_libra <yimin.wang@huawei.com>
2016-01-10Rewrite the HA test case (2)wym_libra8-137/+344
idea: refact the Monitor class in old file "monitor.py" with the base class and sub-class. detail: 1) the BaseMonitor is the base class of other monitor 2) each monitor run in independent process 3) there are two monitor("openstack-cmd" and "process") for the first test case 4) MonitorMgr class used to manager monitor process JIRA: YARDSTICK-149 Change-Id: I2eede94481f740812212e6cb673d175b5f543c15 Signed-off-by: wym_libra <yimin.wang@huawei.com>
2016-01-09running Parser Yang-to-Tosca module as a toolkubi5-0/+160
with jnon and fatih's help, new docker image has been uploaded so this part is about parser verify validating output against expected outcome. Change-Id: If50d241a5338888f14fd11a752dc72678e0c569b JIRA:YARDSTICK-224 Signed-off-by: kubi <jean.gaoliang@huawei.com>
2016-01-08support for ipv6kubi7-57/+418
JIRA:YARDSTICK-187 Change-Id: I1cecd400b4449a09d22d43f4a42e889f00dd4fe7 Signed-off-by: kubi <jean.gaoliang@huawei.com>
2015-12-31Add Dummy context and scenario typeQiLiang3-0/+74
Dummy Context Usage: - if no context specified in the task file then automatically use Dummy Context - or specify the context with type Dummy in the task file, like context: type: Dummy Note: context without type name default use Heat Context. (e.g. samples/fio.yaml) JIRA: - Change-Id: I7f798a7260bdd6ac24902e2c835a3b121319fd8c Signed-off-by: QiLiang <liangqi1@huawei.com>
2015-12-26add scenario and sample file for Unixbench.kubi2-0/+202
JIRA:YARDSTICK-184 Change-Id: Iedd4a3708e08305b1c8fa7a8e1766ceef03ab8bb Signed-off-by: kubi <jean.gaoliang@huawei.com>
2015-12-21Fix lmbench memory read latency stop sizeKristian Hunt1-4/+4
Stop size parameter's type should be float to allow for performing measurements on only L1 and L2 cache if needed. JIRA: - Change-Id: Ibc4d052c343e9dcccf6101c6c949172bb5a0f9e1 Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-12-21To avoid breaking tests when monitor is setPer Holmgren2-2/+2
Two runners, Arithemic and Sequence, broke running tests even if SLA "monitor" was set. Changed these two runners to work in similar fashion as the other two, i.e. that only None can break. "assert" still asserts, and "monitor" just logs but then continues. Change-Id: Ic441761cad8747e5ecfcd0babe719c1aa23981e7 JIRA: -
2015-12-21Merge "Rewrite the HA test case (1)"qi liang9-123/+171
2015-12-18Changes Arithmetic runner to accept several step parametersPer Holmgren1-28/+63
The Arithmetic runner is modified to accept several named parameters to iterate over in each scenario. Possible to chose either to iterate through them in a nested-for-loop fashion (default) or in a tupled loop fashion (combine i:th element of each input). Still possible to only provide one parameter to step/iterate over, as before. All existing yaml files with Arithmetic types have been modified to run the tests as before, and to provide same results as before this change. The only change is in sample file pktgen.yaml to provide a valid example of how to use this modification. It is also possible to step backwards, i.e. start > stop, with a negative step parameter value. (Noticed that tests are terminated when there is SLA error problems in "monitor" mode. Should be fixed in another commit) Change-Id: Ib2ebd24b71dd55d6817cee8e67026a0dd13a9e17 JIRA: -
2015-12-18Rewrite the HA test case (1)wym_libra9-123/+171
refactor the attacker implement. 1) BaseAttacker is added 2) a simple attacker named "kill-process" inherit the BaseAttacker 3) serviceha.py selects an attacker through the BaseAttacker by attacker name JIRA: YARDSTICK-149 Change-Id: Ib718d5edc6b5e14bc3ea0592e0146468ff70b43e Signed-off-by: wym_libra <yimin.wang@huawei.com>
2015-12-15Add run_in_background attribute to scenariosJo¶rgen Karlsson5-14/+40
This change adds the possibility to run scenarios as "background tasks". Background scenarios/tasks: - are started before all "normal scenarios" - runs in parallel with "normal scenarios" - terminates when all "normal scenarios" have completed their tasks They are intended as a way to perform background tasks, e.g. collect data such as cpuload etc, in parallel with the execution of normal benchmarking scenarios. Note that we already have the 'run_in_parallel' attribute but this attribute has a couple of issues and do not solve all the uses cases. Change-Id: I9c5230bfdbbb66030f57b658ce1db87ff2c2d62b Signed-off-by: Jo¶rgen Karlsson <jorgen.w.karlsson@ericsson.com>
2015-12-08Add scenario for reading processor loadJo¶rgen Karlsson1-0/+239
This scenario reads processor and system load statistics and does not run any benchmark tests. The scenario is intended to be run in parallell with other scenarios in order to collect processor and system load statistics. System load is read from /proc/loadavg. Processor usage stats is read using the 'mpstat' utility if it has been installed on the host. If 'mpstat' is not installed on the host processor usage stats is read from /proc/stats. Change-Id: I7156e0c941100023571db750de7540786a4fedb8 JIRA: YARDSTICK-181 Signed-off-by: Jo¶rgen Karlsson <jorgen.w.karlsson@ericsson.com>
2015-11-13A initial HA test casewym_libra7-0/+376
1)stop an openstack service 2)then monitor the corresponding api and check the availability of it 3)recovery the openstack service JIRA: YARDSTICK-149 Change-Id: Id7b77d2f5c71844729c04f37442c8cfaa270ab12 Signed-off-by: wym_libra <yimin.wang@huawei.com>
2015-11-03Merge "Avoid the ValueError exception for a ping timeout."qi liang1-5/+8
2015-11-02Avoid the ValueError exception for a ping timeout.wu.zhihui1-5/+8
In ping.py: If ping a ip address timeout, the return value "stdout" is null. And the code "rtt=float(stdout)" raises a ValueError exception. The better handle is to avoid the exception and log a ping timeout. Change-Id: Ia2a3ff44c35dda1a700a5109f8e642a636e89bc6 Signed-off-by: wu.zhihui <wu.zhihui1@zte.com.cn>
2015-11-01Support NodeContext typeQiLiang1-0/+94
Initial NodeContext implementation to support BareMetal, Controller, Compute scenarios. Usage: 0) install yardstick 1) mkdir -p /etc/yardstick/nodes 2) cp <yardstick_repo>/etc/yardstick/nodes/pod.yaml.sample \ /etc/yardstick/nodes/pod.yaml 3) edit /etc/yardstick/nodes/pod.yaml (make show ip, username, ssh key are configured correctly) 4) yardstick -d task start \ <yardstick_repo>/samples/ping-node-context.yaml 5) cat /tmp/yardstick.out Design etherpad link: https://etherpad.opnfv.org/p/yardstick_framework JIRA: YARDSTICK-169 Change-Id: I3f6ade8243e68d88326f23ed213edb32c638ed32 Signed-off-by: QiLiang <liangqi1@huawei.com>
2015-10-28Extend lmbench scenario to measure memory bandwidthKristian Hunt3-21/+106
Lmbench scenario has now two scripts and will choose between them, based on whether the intention is to run memory latency or bandwidth tests. Added also unit test file for this scenario. JIRA: YARDSTICK-113 Change-Id: I2ba4dbef31f3cafbdb3c583ece5ed9512a906896 Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-10-27Heat context code refactor part 2QiLiang15-232/+348
Heat context code refactor to cater for the evolution of the Yardstick framework. Refactor runner_cfg host/target info handle, as specified at https://etherpad.opnfv.org/p/yardstick_framework step 4. Get general Context info (use Context.get). Before this refactor host and target vm must have the same user name and ssh key, that is not general enough for later extension. test_case.yaml do NOT need to change. JIRA: YARDSTICK-168 Change-Id: I5cfe868f3c6f633214ef550bc9676fe1de0709db Signed-off-by: QiLiang <liangqi1@huawei.com>
2015-10-22Update sla check for scenarioshoujingwen12-63/+58
This patch modify the question that SLA check result is not complete. JIRA: YARDSTICK-172 Change-Id: I10438390baee92caf00dbfcdbdb833823ff8ce31 Signed-off-by: houjingwen <houjingwen@huawei.com>
2015-10-20Heat context code refactorQiLiang6-469/+532
Heat context code refactor to cater for the evolution of the Yardstick framework. At test_case.yaml context segment add "type" to indicate the context type, see samples/ping-heat-context.yaml for an example. And the default context type is Heat, so the existing yaml file do not need to change. JIRA: YARDSTICK-168 Change-Id: Ida0ce12c17cd9b88d7acfb4c9eb1ac6986394b38 Signed-off-by: QiLiang <liangqi1@huawei.com>
2015-10-16Use result_collection_api to store test resultQiLiang1-0/+1
Execute a sample task file from Yardstick, push the test results to MongodB provided by Releng using the common result api provided by Functest. Usage: 0) install yardstick 1) config yardstick mkdir /etc/yardstick cat << EOF >> /etc/yardstick/yardstick.conf [DEFAULT] debug = True dispatcher = http [dispatcher_http] timeout = 5 target = http://213.77.62.197/results EOF 2) run test yardstick task start sample/fio.yaml 3) fetch result from remote result_collection_api curl "http://213.77.62.197/results?case=Fio&installer=compass" JIRA: YARDSTICK-132 Change-Id: I0996c6487c1900704380feb895555057a3f184e9 Signed-off-by: QiLiang <liangqi1@huawei.com>
2015-10-16add error handling when given failing script callswym_libra1-23/+70
JIRA:YARDSTICK-100 Change-Id: I1986b5e443abf186ef9c0235b66ada26c8574af1 Signed-off-by: wym_libra <yimin.wang@huawei.com>
2015-10-15Add Cyclictest scenario and sampleQiLiang2-0/+205
Support measuring operating system's high resolution by using Cyclictest. JIRA: YARDSTICK-122 Change-Id: I2e00ce117e263deaaf52cd2d663b845bd5b65432 Signed-off-by: QiLiang <liangqi1@huawei.com>
2015-10-15Merge "Support general configuration file"QiLiang1-1/+5
2015-10-15Support general configuration fileQiLiang1-1/+5
Use openstack library oslo_config for parsing configuration options from the command line and configuration files. Refer http://docs.openstack.org/developer/oslo.config/ or rally source code for more info on oslo_config library usage. This patch is initially for test result dispatcher configuration, but it is very general to use. Usage: 0) install yardstick 1) mkdir /etc/yardstick 2) cp <repo_root_dir>/etc/yardstick/yardstick.conf.sample \ /etc/yardstick/yardstick.conf 3) edit /etc/yardstick/yardstick.conf 4) run `yardstick task start xxx` cmd JIRA: YARDSTICK-61 Change-Id: I01677ef6e9ab7c1975aa193799195e850da73478 Signed-off-by: QiLiang <liangqi1@huawei.com>
2015-10-15Support for netperfkubi2-0/+210
As agreed in the IRC meeting, use iperf for YARDSTICK-112; support for netperf shall also be included in the framework (Yardstick-121). bulk data test and req/rsp test are supportted by netperf JIRA:YARDSTICK-121 Change-Id: I0e934dc067802c2792d751a19a187cd00ad8741b Signed-off-by: kubi <jean.gaoliang@huawei.com>
2015-10-13Fio scenario support slahoujingwen1-8/+24
JIRA: YARDSTICK-34 Change-Id: I782ba5845f8bd54a19bad078fe7be546400f7524 Signed-off-by: houjingwen <houjingwen@huawei.com>
2015-10-08Add testing packet delay variation between VMswym_libra1-8/+22
1)create test case("iperfs-jitter") for measuring packet delay variation 2)and modify "iperf3.py" to support it JIRA:YARDSTICK-131 Change-Id: Ic3416a713b69dc7687f6f85ecc0b79da8dfb7ed9 Signed-off-by: wym_libra <yimin.wang@huawei.com>
2015-09-24Fix bug in fio scenariohoujingwen2-53/+29
1) when duration>3600s in fio.yaml, ssh time out 2) sometimes the latency value in test result is null 3) update plot.py, fit for code update in fio.py 4) small bug in file.py (result output dump) Add --output-format=json in default args, so fio command can return json format data. JIRA: YARDSTICK-143 Change-Id: Ie02977b8c9f11986a1eed66896b84d18db3d2211 Signed-off-by: houjingwen <houjingwen@huawei.com>
2015-09-22Structure output and make it less redundantJo¶rgen Karlsson5-74/+90
Note: this commit replaces: https://gerrit.opnfv.org/gerrit/#/c/976/8 Adjusts the JSON output of the runners to follow a different structure, laid out below. It is based upon the patch above but is not using the output manager. The purpose is to provide a unified basic layout (as already existed), while making long data series much less repetitive and more space efficient. OUTPUT FORMAT: ------------------------------------------------------------------------ RUNNER PREP - printed exactly once per runner per scenario. Runner MUST print this before sending any RUNNER DATA output { runner_id: <int> scenario_cfg: { <scenario and runner config> } } where runner_id: ID of the runner sending this block scenario_cfg: scenario and runner configuration ------------------------------------------------------------------------ RUNNER DATA runner may print any number of these AFTER having printed a RUNNER PREP { runner_id: <int> benchmark: { <measurements> } } ------------------------------------------------------------------------ The runner_id currently is not unique across runners as it is assigned by noting the runner process id in the underlying operating system. A possible improvement would be to assign runner_id an UUID value according to RFC 4122 (e.g. uuid.uuid4() in python). ------------------------------------------------------------------------ Other changes/cleanups in this patch: - Removed the context argument from _worker_process as it was redundant. It contained a dictionary with the runner configuration but the same dictionary was already in included in the scenario_args argument. - For clarity renamed scenario_args to scenario_cfg. scenario_cfg was the original name used in task.py and it changed name across function calls. Change-Id: I17d96f37c7d3e24b0747d23fcad7509fa949d662 JIRA: YARDSTICK-59 Signed-off-by: Jo¶rgen Karlsson <jorgen.w.karlsson@ericsson.com>
2015-09-16Remove setting logger level to debug in scenariosKristian Hunt6-13/+8
Logging level should be specified using a command line flag -v or -d when running yardstick, rather than hardcoded into source code. If the message is to be displayed whenever yardstick is executed regardless of the verbosity level, then the message should be logged to warning or error levels, instead of debug. JIRA: YARDSTICK-95 Change-Id: Idc9b81b583f4999bfbc57893f0ab3c3675c70f71 Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-09-15Merge "Add test result dispatcher"Hou Jingwen1-11/+13
2015-08-26FIO support variable iodepthhoujingwen1-3/+10
FIO storage performance varification support variable iodepth. JIRA: YARDSTICK-119 Change-Id: Ib6af55c5fbcaa7727a7541b13053b5e4ce8d3203 Signed-off-by: houjingwen <houjingwen@huawei.com>
2015-08-24Add test result dispatcherQiLiang1-11/+13
This patch is the initial implementation of DB result storage. Having implemented a dispathcer which enable user select different ways to store test results according to different requirements. This patch can support not only local file storage, but also remote storage by using http request(it will call common-db-api when available). Later, local DB sotrage will be supported. This patch is raw and simple, which is implemented with reference to openstack ceilometer. Any comment is welcome. JIRA: YARDSTICK-61 Change-Id: Icaf8369edfab5d05f0819eb02d5b05dc8a04d69d Signed-off-by: QiLiang <liangqi1@huawei.com>
2015-08-16Add iteration type for Runnerkubi1-0/+111
A Iteration runner is roughly the same as a Duration runner but runs for a configurable number of times/iterations instead of time. Basically just the termination condition is different. modify the unit and default value of iteration. rename iteration to iterations JIRA:YARDSTICK-49 Change-Id: I67f4014dc3cf923cd31cc2e990e2f7219bce40fe Signed-off-by: kubi <jean.gaoliang@huawei.com>
2015-08-13Add perf scenario and sampleKristian Hunt2-0/+208
Supports using perf software event counters. No hardware events are currently supported. It is possible to add some load to the system using the load parameter, otherwise the system will sleep while the measurements are collected. It is possible to configure SLA for any event measured. Change-Id: Ic413d940093aadd10dc32888ea416aa94316a6fe JIRA: YARDSTICK-51 Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-08-13Merge "Add lmbench scenario and sample"Jörgen Karlsson3-0/+153
2015-08-13Add lmbench scenario and sampleKristian Hunt3-0/+153
Supports measuring memory read latency using Lmbench. SLA can be verified for maximum latency time in nanoseconds compared against results from every measurement taken. Change-Id: Ia187e68eaa03341ba14b6e9eb986afb1d00dd0f6 JIRA: YARDSTICK-89 JIRA: YARDSTICK-92 Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-08-06Fix typo in runner outputraindirve2-2/+2
Sequence and Arithmetic runners have an extra colon in their 'sargs:' output. This commit removes that unnecessary and incorrect character. Change-Id: I9bb1af5397812318e94e3bd71c680605c4917c1a JIRA: YARDSTICK-98 Signed-off-by: raindirve <seanw@kth.se>