summaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)AuthorFilesLines
2015-10-13Add test suite functionality to task commandsKristian Hunt4-22/+165
This patch proposes a more permanent solution to replace the already existing one (YARDSTICK-106). A test suite is a .yaml file which consists of multiple task files which are meant to run as a one set. Each test case in the test suite has to specify the location of the test .yaml task file and can specify additional task arguments or a task arguments file. It is possible to specify the directory where the task files are located in the suite file. If it is left unspecified then it defaults to tests/opnfv/test_cases according to https://etherpad.opnfv.org/p/yardstick_folders A Command line argument --suite is added to the task command. This argument indicates that the intention is to run a test suite. JIRA: YARDSTICK-107 Change-Id: I69508b3b536dd592b6240a2a32dd5d665e0ac4b4 Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-10-09add a key check to fix bugjean.gaoliang@huawei.com1-3/+4
JIRA:YARDSTICK-160 Change-Id: I2cfe91b30d0c906417234d5bf7787fb5dc568921 Signed-off-by: jean.gaoliang@huawei.com <jean.gaoliang@huawei.com>
2015-10-08Add testing packet delay variation between VMswym_libra4-13/+109
1)create test case("iperfs-jitter") for measuring packet delay variation 2)and modify "iperf3.py" to support it JIRA:YARDSTICK-131 Change-Id: Ic3416a713b69dc7687f6f85ecc0b79da8dfb7ed9 Signed-off-by: wym_libra <yimin.wang@huawei.com>
2015-09-29Add QiLiang as committerAna C1-0/+1
QiLiang email added to INFO file. JIRA:- Change-Id: I1703df8fb02f2570ddb47c9aaa5f4db362ed43ff Signed-off-by: Ana C <ana.cunha@ericsson.com>
2015-09-25Fix the bug described in YARDSTICK-142Julien-zte2-2/+1
Delete two line codes in setup.py. And add "easy_install -U setuptools" in README.rst. JIRA: YARDSTICK-142 Change-Id: I254589b776a209e3ca9a18e76d910d7f49e2a01a
2015-09-25Add missing YAML document start boundariesKristian Hunt2-0/+2
This patch adds YAML document start boundaries (YAML spec 4.3.2) to samples, where the boundaries are missing. JIRA: - Change-Id: I6e67d4d73a449ad901392c017220a61f642716ac Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-09-25Add functional tests in verify and mergekubi6-2/+187
As Ana said ,"The first functional test should be as simple as a "Hello world", it shall be possible to run the "Hello world" test without using OpenStack." so i just finish functional test framework and do functional test for subcommand "runner"and"scenario" without using Openstack. JIRA:YARDSTICK-103 Change-Id: I673ae61f9922536a685d32ae62e5ad5165472f9d Signed-off-by: kubi <jean.gaoliang@huawei.com>
2015-09-24Fix bug in fio scenariohoujingwen7-108/+60
1) when duration>3600s in fio.yaml, ssh time out 2) sometimes the latency value in test result is null 3) update plot.py, fit for code update in fio.py 4) small bug in file.py (result output dump) Add --output-format=json in default args, so fio command can return json format data. JIRA: YARDSTICK-143 Change-Id: Ie02977b8c9f11986a1eed66896b84d18db3d2211 Signed-off-by: houjingwen <houjingwen@huawei.com>
2015-09-23Modify iperf sample to use yardstick testing imageKristian Hunt1-3/+3
This patch will make the iperf sample test case consistent with other test cases by using the yardstick-trusty-server image and making it possible to run this test case "out of the box" without requiring the modification of the sample file. Change-Id: I9b0da0d94f9c33f07bca79067a1eb5d615e2c15a JIRA: YARDSTICK-148 Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-09-22Structure output and make it less redundantJo¶rgen Karlsson5-74/+90
Note: this commit replaces: https://gerrit.opnfv.org/gerrit/#/c/976/8 Adjusts the JSON output of the runners to follow a different structure, laid out below. It is based upon the patch above but is not using the output manager. The purpose is to provide a unified basic layout (as already existed), while making long data series much less repetitive and more space efficient. OUTPUT FORMAT: ------------------------------------------------------------------------ RUNNER PREP - printed exactly once per runner per scenario. Runner MUST print this before sending any RUNNER DATA output { runner_id: <int> scenario_cfg: { <scenario and runner config> } } where runner_id: ID of the runner sending this block scenario_cfg: scenario and runner configuration ------------------------------------------------------------------------ RUNNER DATA runner may print any number of these AFTER having printed a RUNNER PREP { runner_id: <int> benchmark: { <measurements> } } ------------------------------------------------------------------------ The runner_id currently is not unique across runners as it is assigned by noting the runner process id in the underlying operating system. A possible improvement would be to assign runner_id an UUID value according to RFC 4122 (e.g. uuid.uuid4() in python). ------------------------------------------------------------------------ Other changes/cleanups in this patch: - Removed the context argument from _worker_process as it was redundant. It contained a dictionary with the runner configuration but the same dictionary was already in included in the scenario_args argument. - For clarity renamed scenario_args to scenario_cfg. scenario_cfg was the original name used in task.py and it changed name across function calls. Change-Id: I17d96f37c7d3e24b0747d23fcad7509fa949d662 JIRA: YARDSTICK-59 Signed-off-by: Jo¶rgen Karlsson <jorgen.w.karlsson@ericsson.com>
2015-09-18Merge "Remove setting logger level to debug in scenarios"Hou Jingwen6-13/+8
2015-09-16Move documentation to new sphinx buildAric Gardner5-0/+65
TODO: Reorganize documentation into properly named subfolders TODO: Add logos to documentation (see index.rst for example) Note you can see the results of this change via a link in the gerrit comments below NOTE: The merge job for your project has not been enabled, this code will not go to the artifacts page upon a merge. you can merge it and work on it in your own time, please contact me to enable the merge (the merge job will make this documentation avaliable to the public on artifacts.opnfv.org) JIRA:RELENG-15 Change-Id: I8fd519393416e2ac2b3ada6fe0670aa4d038fc87 Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
2015-09-16Remove setting logger level to debug in scenariosKristian Hunt6-13/+8
Logging level should be specified using a command line flag -v or -d when running yardstick, rather than hardcoded into source code. If the message is to be displayed whenever yardstick is executed regardless of the verbosity level, then the message should be logged to warning or error levels, instead of debug. JIRA: YARDSTICK-95 Change-Id: Idc9b81b583f4999bfbc57893f0ab3c3675c70f71 Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-09-15Merge "Add test result dispatcher"Hou Jingwen5-11/+176
2015-09-14Merge "add coverage report in Jenkins for Yardstick."Hou Jingwen3-0/+11
2015-09-10Force apt-get to use ipv4 addressesJo¶rgen Karlsson1-0/+3
Workaround to prevent apt-get from using ipv6 addresses on LF POD. JIRA: - Change-Id: I58e31a01f03271cb716a76e5abed18d67d87a0a7 Signed-off-by: Jo¶rgen Karlsson <jorgen.w.karlsson@ericsson.com>
2015-09-10add coverage report in Jenkins for Yardstick.kubi3-0/+11
since coverage report based on master, coverage configure file should be merged to master at first then coverage shell script will be committed after coverage conf As first part, this patch only include two configure files and one init file : .testr.conf , .coveragerc, __init__.py Change-Id: I073e65b641144e4878595f073cd37081e4a4940b JIRA:YARDSTICK-108 Signed-off-by: kubi <jean.gaoliang@huawei.com>
2015-09-09Add support for building images on CentOSJo¶rgen Karlsson2-7/+18
This is a workaround for building on CentOS. CentOS don't have nbd so guestfstool is used to mount the ubuntu image. The http sources has been changed to ftp as the apt-get tool inside the Ubuntu image dont work well with http sources when running on CentOS. Change-Id: I23679ff034ea76782dcc73d3283cb70ebace5ee8 JIRA: YARDSTICK-136 Signed-off-by: Jo¶rgen Karlsson <jorgen.w.karlsson@ericsson.com>
2015-09-06Support to config external network by shell envQiLiang16-16/+4
Example invocation: $ export EXTERNAL_NETWORK=net04_ext $ yardstick task start samples/ping.yaml The default external network is "net04_ext" external_net should not be possible set in yaml anymore. JIRA: YARDSTICK-54 Change-Id: I8c7b2850801f312c6878e9a363b06cdc8ade09bf Signed-off-by: QiLiang <liangqi1@huawei.com>
2015-09-03Add unit test for iperf3Kristian Hunt2-0/+134
Sample output from the iperf3 is read from a pre-generated json file. Change-Id: I6e04c6bd2e61b841c7e5673cfa78aeb17220a5f9 JIRA:- Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-09-01add support for Jinja2 in task filekubi6-6/+340
Add support in task file for the template syntax based on Jinja2. JIRA:YARDSTICK-101 Change-Id: I24be133ba590510612d97a1fce6c024e6edb57e4 Signed-off-by: kubi <jean.gaoliang@huawei.com>
2015-08-26Add unit test for logical modelKristian Hunt2-0/+345
Add unit tests for logical model (model.py). This test suite does not include tests for unfinished functionality - some parts of methods or in some cases whole methods with unresolved TODOs are not tested. This test suite is not complete, for most methods only the default case is tested. JIRA:- Change-Id: I0984fb3655513f5ad67dcc7436461990da9296e7 Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-08-26Add script for running yardstick tasks back-to-backKristian Hunt1-0/+43
This script enables to run multiple yardstick tasks in the yardstick daily build without having to change the Jenkins job in the releng repository. This script should be executed from the main yardstick directory just as run_tests.sh. If at least one of the tasks listed returns a non-zero value then the script continues executing, but in the end will exit with value 1. JIRA: YARDSTICK-106 Change-Id: I8a6bb7e6c03ec551709ff66a45aad7257fb36e92 Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-08-26FIO support variable iodepthhoujingwen2-7/+16
FIO storage performance varification support variable iodepth. JIRA: YARDSTICK-119 Change-Id: Ib6af55c5fbcaa7727a7541b13053b5e4ce8d3203 Signed-off-by: houjingwen <houjingwen@huawei.com>
2015-08-26Merge "Add unit test for fio"Hou Jingwen2-0/+88
2015-08-25Add unit test for utilsKristian Hunt1-0/+90
Unit test case for utility function itersubclasses is from rally. A separate test case is created for each utility function. Running of unittest from run_test.sh is NOT enabled. Change-Id: I47a8020a05b19eb3f1edcebde1042c732bff5561 Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-08-24Add test result dispatcherQiLiang5-11/+176
This patch is the initial implementation of DB result storage. Having implemented a dispathcer which enable user select different ways to store test results according to different requirements. This patch can support not only local file storage, but also remote storage by using http request(it will call common-db-api when available). Later, local DB sotrage will be supported. This patch is raw and simple, which is implemented with reference to openstack ceilometer. Any comment is welcome. JIRA: YARDSTICK-61 Change-Id: Icaf8369edfab5d05f0819eb02d5b05dc8a04d69d Signed-off-by: QiLiang <liangqi1@huawei.com>
2015-08-20Add unit test for template_format.pyKristian Hunt2-0/+50
Unit test file for template_format.py originates from python-heatclient. Dependency for testscenarios module is removed and testtools module is replaced with python native unittest module. Running of unittest from run_test.sh is NOT enabled. JIRA: - Change-Id: I14e77ce04763b503cc757fe7098d2c704a7ae22c Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-08-20Add unit test for sshKristian Hunt1-0/+285
Unit test file for ssh.py originates from rally. It is modified to be consistent with the ssh module used in yardstick - some tests for missing functionality are removed and naming convention is changed (ssh in yardstick vs sshutils in rally). Running of unittest from run_test.sh is NOT enabled. JIRA:- Change-Id: I3ea554e3fbe6f1f906778f88ddd859424331c8df Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-08-19Merge "Add unit test for pktgen"Jörgen Karlsson1-0/+195
2015-08-17Merge "Add iteration type for Runner"Hans Feldt2-0/+156
2015-08-17Add dependencies to dev-env section in README.rstKristian Hunt1-1/+3
Add paragraph for installing required dependencies to README.rst to make setting up a new development environment easier and documented. Moved existing command to install virtualenv to the new paragraph to maintain integrity. JIRA:- Change-Id: I356611fbb92a5d2a556675e779a11308c22a8ceb Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-08-16Add iteration type for Runnerkubi2-0/+156
A Iteration runner is roughly the same as a Duration runner but runs for a configurable number of times/iterations instead of time. Basically just the termination condition is different. modify the unit and default value of iteration. rename iteration to iterations JIRA:YARDSTICK-49 Change-Id: I67f4014dc3cf923cd31cc2e990e2f7219bce40fe Signed-off-by: kubi <jean.gaoliang@huawei.com>
2015-08-14Add support for generating graphs from outputKristian Hunt3-1/+316
Command line tool yardstick-plot is to be used to visualize results gathered from yardstick framework's output file. Currently supports plotting graphs from ping, pktgen, iperf3 and fio tests. Yardstick-plot takes two arguments - input file and output folder and both of them have defaults to fall to if left unspecified. Supports having multiple different scenario types in an input file, while assuming that all results from the same scenario type belong to one graph. Thus, results plotted from a single scenario type with different parameters are currently non-informative. yardstick-plot is declared as an extra for yardstick in setup.py as it is not required for all use cases of the yardstick framework. It can be installed for example using command: $ pip install -e .[plot] from the folder where setup.py is located. Example invocation: yardstick-plot -i /tmp/yardstick.out -o /tmp/plots/ JIRA: YARDSTICK-65 Change-Id: Ic436ca360ba2496aa829ca817b1d9d5f3c944c6c Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-08-13Add perf scenario and sampleKristian Hunt4-0/+253
Supports using perf software event counters. No hardware events are currently supported. It is possible to add some load to the system using the load parameter, otherwise the system will sleep while the measurements are collected. It is possible to configure SLA for any event measured. Change-Id: Ic413d940093aadd10dc32888ea416aa94316a6fe JIRA: YARDSTICK-51 Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-08-13Merge "Add lmbench scenario and sample"Jörgen Karlsson5-0/+200
2015-08-13Add lmbench scenario and sampleKristian Hunt5-0/+200
Supports measuring memory read latency using Lmbench. SLA can be verified for maximum latency time in nanoseconds compared against results from every measurement taken. Change-Id: Ia187e68eaa03341ba14b6e9eb986afb1d00dd0f6 JIRA: YARDSTICK-89 JIRA: YARDSTICK-92 Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-08-13Make run_tests.sh verbose and enable unit testsKristian Hunt1-10/+29
By default output from run_tests is verbose. This enables seeing the error messages in Jenkins' console output when a job fails. Output can alternatively be captured to a log file using an -f flag. Example invocation for logging output to a file: $ ./run_tests.sh -f Running of unittest from run_test.sh is enabled. JIRA: YARDSTICK-102 YARDSTICK-104 Change-Id: I3be7d7873cf75247980d97c3fbdf0bfdbabe53ad Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-08-06Fix typo in runner outputraindirve2-2/+2
Sequence and Arithmetic runners have an extra colon in their 'sargs:' output. This commit removes that unnecessary and incorrect character. Change-Id: I9bb1af5397812318e94e3bd71c680605c4917c1a JIRA: YARDSTICK-98 Signed-off-by: raindirve <seanw@kth.se>
2015-08-05Merge "Update pbr requirement version in setup.py"Hou Jingwen1-1/+1
2015-08-04Update pbr requirement version in setup.pyQiLiang1-1/+1
JIRA: YARDSTICK-96 Change-Id: Ic7e918b6d287725e2451da93aa8a36ec2b09ca0a Signed-off-by: QiLiang <liangqi1@huawei.com>
2015-08-04Merge "Update example invocation of yardstick in main.py"Hou Jingwen1-3/+3
2015-07-28Fix a bug for algorithm of arithmetic runner.kubi1-1/+2
since value will overstep the stop value under some condition JIRA:YARDSTICK-97 Change-Id: I11656206fc9b576283d243a3be249276dba03759 Signed-off-by: kubi <jean.gaoliang@huawei.com>
2015-07-28Add unit test for fioKristian Hunt2-0/+88
Sample output from fio_benchmark.bash is read from a string. Running of unittest from run_test.sh is NOT enabled. JIRA:- Change-Id: If4f4d00cf215fa1d332260817a8a3e702e72fd7e Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-07-27Add unit test for pktgenKristian Hunt1-0/+195
Sample output from pktgen_benchmark.bash is read from a string. Running of unittest from run_test.sh is NOT enabled. JIRA:- Change-Id: I51391df0726ed9e7486775d2cdd5583b305ea8e0 Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-07-22Update example invocation of yardstick in main.pyKristian Hunt1-3/+3
Example invocation in main.py is in accordance with change 40f49a4 JIRA:- Change-Id: Id3b07e83db768c6d43f6a5bddc954448c9f1e6c4 Signed-off-by: Kristian Hunt <kristian.hunt@gmail.com>
2015-07-16Merge "Fix yardstick.out overwritten bug"Hou Jingwen2-1/+5
2015-07-15Add test case description templateAna C1-0/+88
Template states mandatory information for each test case to be delivered by Yardstick Project. JIRA: YARDSTICK-2 Change-Id: I16b98acc6689e24f8e6a85c356ee1b8991bd5276 Signed-off-by: Ana C <ana.cunha@ericsson.com>
2015-07-15Fix yardstick.out overwritten bugpanghao12-1/+5
The output of scenarios will be appended to the yardstick.out file. JIRA: YARDSTICK-45 Change-Id: I185c6d2a8a534c8bb2b731bb84c47bdf4bad4427 Signed-off-by: panghao1 <shamrock.pang@huawei.com>
2015-07-14Fix bug in cli.pykubi1-1/+1
Make sure log level set correctly. JIRA: YARDSTICK-57 Change-Id: Icc0e100432ce773f70ee0ac9b18471f89849d990 Signed-off-by: kubi <jean.gaoliang@huawei.com>