aboutsummaryrefslogtreecommitdiffstats
path: root/yardstick/tests/unit/benchmark/core
AgeCommit message (Collapse)AuthorFilesLines
2019-04-26Merge changes from topic 'feat/baro_nfvi_metrics'Volodymyr Mytnyk1-5/+328
* changes: Use baro and yardstick metrics in dynamic HTML report benchmark.core.report: Add _combine_times benchmark.core.report: Add _get_baro_metrics Add ability to get data from different DBs in influx Refactor: add _format_datasets
2019-04-09Merge "Suppress log outputs in test_task.py"Volodymyr Mytnyk1-0/+8
2019-04-04Update to support using external heat templatetreyad1-0/+37
Modify the _get_server() function when using external heat template. The server attrs should be read from stack.outputs. JIRA: YARDSTICK-1453 Change-Id: I00eda7476e77fc0a5c15ef8db7d057fa1cf267e6 Signed-off-by: treyad <treyad@viosoft.com>
2019-04-03Use baro and yardstick metrics in dynamic HTML reportEmma Foley1-1/+0
_combine_metrics combines metrics from different sources. This is for use with the ``yardstick report generate-nsb`` command, which will combine yardstick and barometer metrics in the dynamic HTML report. JIRA: YARDSTICK-1593 Change-Id: I87002948ebb4cc88fb0932380bcb9920eb53db58 Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2019-02-11benchmark.core.report: Add _combine_timesEmma Foley1-0/+142
_combine_times() combines and interleves separate lists of timestamps. This will allow the Yardstick and Barometer timestamps to be combined for the dynamic HTML report created with ``yardstick report generate-nsb`` command. JIRA: YARDSTICK-1593 Change-Id: I0f5ea4d001775495fb9b9b4de2d2360c9c61cc51 Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2019-02-11benchmark.core.report: Add _get_baro_metricsEmma Foley1-1/+131
Retrieve and parse Barometer metrics relevant to a Yardstick testcase. _get_barometer_metrics() retrieves NFVi metrics from collectd for metrics collected during a Yardstick test case. JIRA: YARDSTICK-1593 Change-Id: I1ef1ccd999e1cbee546db39fb8ad6de0d97e6576 Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2019-02-11Refactor: add _format_datasetsEmma Foley1-4/+56
JIRA: YARDSTICK-1593 Change-Id: I8d2c8665d767c92da20db8f97690f20da4a68908 Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2019-01-30Refactor: add _get_timestamps()Emma Foley1-1/+9
JIRA: YARDSTICK-1593 Change-Id: I3ec352dd577c6030fa86a2817e264ba7c80773f1 Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2019-01-28Suppress log outputs in test_task.pyEmma Foley1-0/+8
Change-Id: Ic7b4ebe5fff9a0a31e395e67b8a3ea52d99442a9 JIRA: YARDSTICK-1541 Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2018-12-21Additional rework of NSB reportPatrice Buriez1-17/+9
- Make format_for_jstree expect a list of metric names - Avoid displaying timestamps twice in initial data table - Sort metrics in initial data table - Display testcase name - Fix styling - Make better use of JS and jQuery features - Move event handler to JS file - Avoid adding multiple tbody elements into data table - Adjust unit tests and functional tests accordingly JIRA: YARDSTICK-1367 Topic: report/html_table (12 of 12) Change-Id: I85d853f8e392953cace67e94fa0af2e2492a2b86 Signed-off-by: Patrice Buriez <patrice.buriez@intel.com>
2018-12-21Fix conversion to JS for HTML reportsPatrice Buriez1-17/+96
Some new fields in InfluxDB have plain text contents that AST cannot parse as valid expressions, resulting in exceptions being raised by yardstick CLI commands "report generate" and "report generate-nsb". Reworked _generate_common() to properly handle and convert: - unicode, str and missing keys - None, unicode, str, float, long and int values - float, long and int stored as unicode or str values Added test__generate_common() unit test, to verify all possible conversion flavors. Also renamed incorrect "tasks" to "metrics". Also fixed warning in Python3 for some unit tests, caused by deprecation of unittest.TestCase.assertRaisesRegexp(). JIRA: YARDSTICK-1367 Topic: report/html_table (10 of 12) Change-Id: Iff75bfd2c1dcaf3806f67f52f0ea594f10aceb5b Signed-off-by: Patrice Buriez <patrice.buriez@intel.com>
2018-12-19Add report generate-nsb commandPatrice Buriez1-20/+32
JIRA: YARDSTICK-1367 Topic: report/html_table (8 of 12) Change-Id: I8a2f96224a745334b67dd71875bebb3b69b9adc3 Signed-off-by: Emma Foley <emma.l.foley@intel.com> Signed-off-by: Patrice Buriez <patrice.buriez@intel.com>
2018-11-22Add parser to support jsTree in reportPatrice Buriez1-0/+70
Allow the user to select what data to show in the report using jsTree to navigate a hierarchical metrics list. JIRA: YARDSTICK-1367 Topic: report/html_table (4 of 11) Change-Id: I86d782a0a70b80a1cdfaab2f41afb7668066cbf7 Signed-off-by: Emma Foley <emma.l.foley@intel.com> Signed-off-by: Patrice Buriez <patrice.buriez@intel.com>
2018-11-21Update unit tests for yardstick/benchmark/core/report.pyPatrice Buriez1-44/+70
* Remove redundant imports * Rename some constants * Add additional unit tests * Reword KeyError message JIRA: YARDSTICK-1367 Topic: report/html_table (1 of 11) Change-Id: I3d89c1ed9acaeb7421a0734e33075ebae252c035 Signed-off-by: Emma Foley <emma.l.foley@intel.com> Signed-off-by: Patrice Buriez <patrice.buriez@intel.com>
2018-07-23Cleanup Testcase unit testsRodolfo Alonso Hernandez1-7/+7
Removed unneeded output. JIRA: YARDSTICK-1342 Change-Id: I45ab14f854cd059625c1a96959b1cbd0a19f52a8 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-07-23Cleanup TaskParseTestCase unit testsRodolfo Alonso Hernandez1-3/+6
Removed unneeded output. JIRA: YARDSTICK-1340 Change-Id: I66c0902edfea5fad9282bcb36e57b67cd6c340ab Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-07-05Make context types constantsRodolfo Alonso Hernandez1-4/+4
Make context types constants. These are the current context types: - Dummy - Heat - Kubernetes - Node - StandaloneOvsDpdk - StandaloneSriov Any reference to a context type should use the referred constant. JIRA: YARDSTICK-1292 Change-Id: I1ff0085161f88304d71e3ddad7975f9020e0fc20 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-06-29Cleanup Plugin unit testsRodolfo Alonso Hernandez1-2/+8
Removed unneeded output (LOG.info and print functions). JIRA: YARDSTICK-1274 Change-Id: Ida3b62ce28e27058a8b4a3a9e6fa16f041a1f99c Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-06-26Addition of Configurable Sampling and Configurable Confirmation RetryDanielMartinBuckley1-0/+25
JIRA: YARDSTICK-1221 In order to increase accuracy of result the following are done :- - Improve Measurement Accuracy (YARDSTICK-1212) - Improve Sampling Interval (YARDSTICK-1219) - Allow 4 PROX ports to be read simultaneously (YARDSTICK-1220) This change does the following :- - Stores LINE Rate statistics of Sample - Requires a confirmation retry before deciding to increase or decrease this is configurable. - Allows the user to disable Sampling or specify a sample interval - Added Code Coverage of ProxDurationRunner based on YARDSTICK-1199 Change-Id: I27242ac1849c9a2712866385b5fbc05977c71516 Signed-off-by: Daniel Martin Buckley <daniel.m.buckley@intel.com>
2018-06-13assert[Greater,Equal] -> assert_{,not_}calledEmma Foley1-5/+5
assertEqual(mock_xxx.call_count, 1) -> mock_xxx.assert_called_once assertEqual(mock_xxx.call_count, 0) -> mock_xxx.assert_not_called assertGreater(mock.call_count, 0) -> mock.assert_called() assertGreaterEqual(mock.call_count, 1) -> mock.assert_called() JIRA: YARDSTICK-1069 Change-Id: I890084d120c8e78304e169e2a0e5d30011a41525 Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2018-06-11Move tests: unit/network_services/vnf_genericEmma Foley1-6/+8
* Fix pylint errors * Add TODOs Some errors are ignored locally, as they were a symptom of other problems. These issues have been flagged with a TODO, and should be fixed later. JIRA: YARDSTICK-837 Signed-off-by: Emma Foley <emma.l.foley@intel.com> Change-Id: I3d95537a64bde923703421510d27bac0d6dd92bd
2018-05-31Remove instantiated contexts in "test_task"Rodolfo Alonso Hernandez1-0/+9
During unit testing, all contexts instantiated must be deleted at the end of the test to avoid clashes with other tests. JIRA: YARDSTICK-1209 Change-Id: I1994b915001b4a518e096833a0c5d33febbd8de6 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-03-14Fix retrieving "options" section in "scenario"Rodolfo Alonso Hernandez1-0/+28
In [1] a new method to rename the "scenario" host names was implemented. This method parses all sections with host names and applies the "qualified" name. Some malformed test cases define the "options" section without content. This causes that [2] retrieves "None" value instead of an empty dictionary. This possibility must be handled in the new method. [1]I44da30dac562c1a4166e084645ae91c17798651d [2]https://github.com/opnfv/yardstick/blob/4b8b674b65830a24230faed71e8d9a1048139c89/yardstick/benchmark/core/task.py#L630 JIRA: YARDSTICK-1073 Change-Id: I8864b428734ead8c5aa39de5091d3a2a691be060 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-03-05assertTrue(mock.called) -> mock.assert_calledEmma Foley1-1/+1
When checking whether a mocked method has been called, mock.assert_called or mock.assert_called_once should be used, and not unittest.assertTrue(mock.called) This change does this update. It can be verified with: grep -irn "self.assertTrue(.*called)" --exclude-dir=.tox And the replacement was done with: sed -i 's/self.assertTrue(\(.*\).called)/\1.assert_called_once()/g' <list_of_files> Change-Id: I4f26e0c736bf33e0b2413c8e8c33dbdb91f090e2 JIRA: YARDSTICK-865 Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2018-03-02Remove main() and __main__ from tests.Emma Foley2-12/+0
Removes the unnecessary main() functions from tests. Also removes shebang (#!) where it appears at the top of files. JIRA: YARDSTICK-861 Change-Id: I79180d1eb9c5bce640142dd62ce28c7437c1b210 Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2018-03-02Replace assertEqual(None, xx) with assertIsNone(xx)Emma Foley1-1/+1
Change-Id: Ie956d8064a8fbcbd3d56c5a79c4c613d35184af4 JIRA: YARDSTICK-942 Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2018-03-02Merge "Add "render-only" option to "task" command"Rodolfo Alonso Hernandez1-1/+137
2018-03-01Merge "Add ability to reuse existing OpenStack set-up"Ross Brattain1-0/+32
2018-03-01Add "render-only" option to "task" commandRodolfo Alonso Hernandez1-1/+137
This new option provides to the user the ability to output the rendered input files. This option could be useful in case the input files are Jinja2 templates, depending on input arguments. The user can preview the rendered input tasks files without executing them. JIRA: YARDSTICK-1020 Change-Id: Ib15ade7e1adcb29beae5e635fb5d02045c4432bb Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-03-01Add ability to reuse existing OpenStack set-upEmma Foley1-0/+32
This change adds two optional flags to the context configuration for heat: * no_setup -- to re-use an existing OpenStack deployment * no_teardown -- to skip undeploying the context, so it is available in other testcases In order to achieve this, resource naming had to be made more predictable. When no_setup or no_teardown are set: * Do not add a suffix to context name * When searching, try with and without a suffix to find a server by name All behaviors remain the same unless these values are set. NOTE: This feature doesn't work with the NSPerf scenario, because the interfaces in the VNFs are bound to DPDK after the first test, so the look-up for missing topology information fails in subsequent test runs, as the interfaces are not visable to the kernel. This can be addressed either by either unbinding the interfaces from DPDK at the end of each test, or modifying the look-up for the missing topology information to query DPDK as well. The change will be made in a future patch. JIRA: YARDSTICK-886 Change-Id: I6df5e86e419c283a2bee41917e4f62179aa9c31a Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2018-03-01Merge changes from topics 'feat/keep_vnf', 'YARDSTICK-886'Ross Brattain1-29/+86
* changes: Add _create_new_stack method Update TaskParser to deal with qualified name in Context Add qualified name to Context
2018-03-01Update TaskParser to deal with qualified name in ContextEmma Foley1-29/+86
The context name depends on the defined name in the testcase input file, the task ID and the flags of the context. If the context is going to be undeployed at the end of the test, the task ID is suffixed to the name to avoid interferences with previous deployments. If the context needs to be deployed at the end of the test, the name assigned is kept. This patch makes this process transparent to the developer. This patch modifies how TaskParser determines the correct context name, taking into account that the name might change based on context flags. JIRA: YARDSTICK-886 Change-Id: I44da30dac562c1a4166e084645ae91c17798651d Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com> Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2018-02-23Yardstick real-time influxdb KPI reporting regressionschenjiankun1-2/+9
JIRA: YARDSTICK-989 We used to have real-time influxdb reporting of test KPIs. The user could monitor using grafana and see the real-time output. The record format was changed to now only report KPIs at the end of the test. This is a problem for test cases which run for a long duration, we need to wait until the end of the test execution to get any results from influxdb. If the test fails in between or doesn't exit cleanly for some reason, we are left with no results stored in influxdb which gives the user no information. This is also a regression from the previous behaviour. Change-Id: I0f476dff9162a359f0286fb421f2e9c4befaa5cc Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2018-02-22Merge "Make files pep8 compliant"Emma Foley1-75/+57
2018-02-20Make files pep8 compliantEmma Foley1-75/+57
This change makes files pep8 compliant before adding the reuse context feature JIRA: YARDSTICK-886 Change-Id: Iae7daaa159a9ddbb5809a9a7ac74f8a53683089a Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2018-02-16Refactor benchmark.core.test_plugin.PluginTestCaseRodolfo Alonso Hernandez1-82/+122
Tests in benchmark.core.test_plugin.PluginTestCase are not covering correctly the functionality of the tested class. YARDSTICK-851 Change-Id: I0b4cda7dbf109776a202167abfde8c9eb268db12 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-02-12Move tests: unit/benchmarkEmma Foley9-0/+635
* Fix pylint errors * Add TODOs Some errors are ignored locally, as they were a symptom of other problems. These issues have been flagged with a TODO, and should be fixed later. Change-Id: I30eb4b0aafe0575d0cddbc946108291f21a98ed8 Jira: YARDSTICK-837 Signed-off-by: Emma Foley <emma.l.foley@intel.com> Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>