aboutsummaryrefslogtreecommitdiffstats
path: root/yardstick/benchmark/core/task.py
AgeCommit message (Collapse)AuthorFilesLines
2019-04-04Update to support using external heat templatetreyad1-1/+19
Modify the _get_server() function when using external heat template. The server attrs should be read from stack.outputs. JIRA: YARDSTICK-1453 Change-Id: I00eda7476e77fc0a5c15ef8db7d057fa1cf267e6 Signed-off-by: treyad <treyad@viosoft.com>
2019-02-21Unify TG and VNF names in infrastructure filesOrest Voznyy1-1/+1
Update naming in related files, i.e. pod, docs, docstrings. Rename TGs and VNFs to follow a single naming convention, for all VNFs: name: trafficgen_0 role: tg__0 name: vnf_0 role: vnf__0 JIRA: YARDSTICK-1578 Change-Id: I7596b915939f242349a99721b6088ae738dd277f Signed-off-by: Orest Voznyy <orestx.voznyy@intel.com>
2018-12-07Compatible with python2 and 3 string checkcope.li1-1/+2
JIRA:YARDSTICK-1557 Change-Id: I54ee4d6ef835e8e6ecd55cfa305151e842f88b03 Signed-off-by: cope.li <cope.li@huawei.com>
2018-07-05Make context types constantsRodolfo Alonso Hernandez1-16/+20
Make context types constants. These are the current context types: - Dummy - Heat - Kubernetes - Node - StandaloneOvsDpdk - StandaloneSriov Any reference to a context type should use the referred constant. JIRA: YARDSTICK-1292 Change-Id: I1ff0085161f88304d71e3ddad7975f9020e0fc20 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-05-21Add "host_name_separator" variable to Context classRodolfo Alonso Hernandez1-19/+14
This feature will provide Kubernetes context the ability to handle the context name inside the class itself, providing to the developer an abstraction of the possible naming limitations in Kubernetes. E.g.: "dot" character in Pod names is no allowed [1]. [1] https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/issues/58 JIRA: YARDSTICK-1188 Change-Id: I82121f970b550170357a443b6340be7900602a57 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-04-16Get HA test case results on failureMiikka Koistinen1-11/+18
Now, yardstick HA test cases (which have 'sla action: monitor') do not output results if the test case SLA validation fails. This patch modifies the task runner and the serviceHA scenario so that during an SLA validation failure the test result data is captured correctly. Do similar changes in the general scenario. JIRA: YARDSTICK-1115 Change-Id: I648a8b229600c0ad089320ac3f803698f73aa800 Signed-off-by: Miikka Koistinen <miikka.koistinen@nokia.com>
2018-03-22Merge "Running Test Case in Latest Yardstick Docker Image shows Error"Jack Chan1-3/+13
2018-03-20Running Test Case in Latest Yardstick Docker Image shows Errorchenjiankun1-3/+13
JIRA: YARDSTICK-1080 This bug will only occur in kubernetes test case. In openstack, we use 'stack_name.context', but some kubernetes resources(pod, etc) do not support this format. So we use 'name-context'. Change-Id: I87d61c027d55449914bc283672eb98be3d3fe22f Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2018-03-14Fix retrieving "options" section in "scenario"Rodolfo Alonso Hernandez1-2/+4
In [1] a new method to rename the "scenario" host names was implemented. This method parses all sections with host names and applies the "qualified" name. Some malformed test cases define the "options" section without content. This causes that [2] retrieves "None" value instead of an empty dictionary. This possibility must be handled in the new method. [1]I44da30dac562c1a4166e084645ae91c17798651d [2]https://github.com/opnfv/yardstick/blob/4b8b674b65830a24230faed71e8d9a1048139c89/yardstick/benchmark/core/task.py#L630 JIRA: YARDSTICK-1073 Change-Id: I8864b428734ead8c5aa39de5091d3a2a691be060 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-03-08[bugfix] report generate command print errorrexlee87761-1/+1
JIRA: YARDSTICK-1056 Change-Id: I32ab9ae7482f6b81e00d6d3e7e858d980a2dd95f Signed-off-by: rexlee8776 <limingjiang@huawei.com>
2018-03-02Merge "Add "render-only" option to "task" command"Rodolfo Alonso Hernandez1-47/+82
2018-03-01Merge "Add ability to reuse existing OpenStack set-up"Ross Brattain1-1/+3
2018-03-01Add "render-only" option to "task" commandRodolfo Alonso Hernandez1-47/+82
This new option provides to the user the ability to output the rendered input files. This option could be useful in case the input files are Jinja2 templates, depending on input arguments. The user can preview the rendered input tasks files without executing them. JIRA: YARDSTICK-1020 Change-Id: Ib15ade7e1adcb29beae5e635fb5d02045c4432bb Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
2018-03-01Add ability to reuse existing OpenStack set-upEmma Foley1-1/+3
This change adds two optional flags to the context configuration for heat: * no_setup -- to re-use an existing OpenStack deployment * no_teardown -- to skip undeploying the context, so it is available in other testcases In order to achieve this, resource naming had to be made more predictable. When no_setup or no_teardown are set: * Do not add a suffix to context name * When searching, try with and without a suffix to find a server by name All behaviors remain the same unless these values are set. NOTE: This feature doesn't work with the NSPerf scenario, because the interfaces in the VNFs are bound to DPDK after the first test, so the look-up for missing topology information fails in subsequent test runs, as the interfaces are not visable to the kernel. This can be addressed either by either unbinding the interfaces from DPDK at the end of each test, or modifying the look-up for the missing topology information to query DPDK as well. The change will be made in a future patch. JIRA: YARDSTICK-886 Change-Id: I6df5e86e419c283a2bee41917e4f62179aa9c31a Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2018-03-01Merge changes from topics 'feat/keep_vnf', 'YARDSTICK-886'Ross Brattain1-43/+64
* changes: Add _create_new_stack method Update TaskParser to deal with qualified name in Context Add qualified name to Context
2018-03-01Update TaskParser to deal with qualified name in ContextEmma Foley1-43/+64
The context name depends on the defined name in the testcase input file, the task ID and the flags of the context. If the context is going to be undeployed at the end of the test, the task ID is suffixed to the name to avoid interferences with previous deployments. If the context needs to be deployed at the end of the test, the name assigned is kept. This patch makes this process transparent to the developer. This patch modifies how TaskParser determines the correct context name, taking into account that the name might change based on context flags. JIRA: YARDSTICK-886 Change-Id: I44da30dac562c1a4166e084645ae91c17798651d Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com> Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2018-02-23Yardstick real-time influxdb KPI reporting regressionschenjiankun1-9/+10
JIRA: YARDSTICK-989 We used to have real-time influxdb reporting of test KPIs. The user could monitor using grafana and see the real-time output. The record format was changed to now only report KPIs at the end of the test. This is a problem for test cases which run for a long duration, we need to wait until the end of the test execution to get any results from influxdb. If the test fails in between or doesn't exit cleanly for some reason, we are left with no results stored in influxdb which gives the user no information. This is also a regression from the previous behaviour. Change-Id: I0f476dff9162a359f0286fb421f2e9c4befaa5cc Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2018-02-20Make files pep8 compliantEmma Foley1-3/+3
This change makes files pep8 compliant before adding the reuse context feature JIRA: YARDSTICK-886 Change-Id: Iae7daaa159a9ddbb5809a9a7ac74f8a53683089a Signed-off-by: Emma Foley <emma.l.foley@intel.com>
2017-10-13task: use CONF_FILE from constantsRoss Brattain1-2/+2
I'm not sure why we hardcoded config_file when the CONF_FILE constant will work. Change-Id: I45a1dc17ac2b185c659725f55462214acb74838d Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-10-11task: drain background runner queuesRoss Brattain1-6/+14
We were not draining the queues in the background runners. Modify the main runner_join loop to loop over all the runners and drain them. The runner join method does extra work for periodic actions so we can't change its behavior. Instead create a new poll() method and use that to check runner status Change-Id: I9466ba40a6a4c45c82cedff279cbb4817c6b66ad Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-10-02drain runner queue and undo cancel_join_threadRoss Brattain1-15/+16
Sometimes the runners can hang. Initially debugging lead to the queue join thread, so I thought we could cancel all the join threads and everything would be okay. But it turns out canceling the queue join threads can lead to corruption of the queues, so when we go to drain the queues the task hangs. But it also turns out that we were not properly draining the queues in the task process. We were waiting for all the runners to exit, then draining the queues. This is bad and will cause the queues to fill up and hang and/or drop data or corrupt the queues. The proper fix seems to be to draining the queues in a loop before calling join with a timeout. Also modified the queue drain loops to no block on queue.get() Revert "cancel all queue join threads" This reverts commit 75c0e3a54b8f6e8fd77c7d9d95decab830159929. Revert "duration runner: add teardown and cancel all queue join threads" This reverts commit 7eb6abb6931b24e085b139cc3500f4497cdde57d. Change-Id: Ic4f8e814cf23615621c1250535967716b425ac18 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-09-25task: fixup s/exe_info/exc_infoRoss Brattain1-1/+1
JIRA: YARDSTICK-785 Change-Id: Ib37498e8df6a520f1d03256b73346fcedab3a177 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-09-25Merge "Log each test case status in a task"Ross Brattain1-2/+5
2017-09-21Add service in kubernetes contextchenjiankun1-5/+5
JIRA: YARDSTICK-803 Currently kubernetes test case can only run in master node. We need to support it run in jump server. So I add service and use nodePort type. Then we can login the pod using nodePort. Change-Id: Ia7900d263f1c5323f132435addec27ad10547ef9 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-09-21Merge "Add host&targer in scenario['options']['server_name'] support"Jing Lu1-31/+37
2017-09-15Generate pod.yaml from current contextRoss Brattain1-0/+1
We want to generate pod.yaml from Heat contexts so we can re-use the context without destroying it. But we don't have node role information and it doesn't make sense in this case, so make the role optional. Since we changed Heat to use pkey instead of key_filename, we can embed the pkey into the pod.yaml, but we have to make sure to convert the pkey to string, in case it is a RSAKey object Change-Id: Ibefcfbd8236e68013a704c39964cb870da825da8 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-09-15Merge "improve logging, clear using print"Rex Lee1-17/+15
2017-09-15Add host&targer in scenario['options']['server_name'] supportchenjiankun1-31/+37
JIRA: YARDSTICK-810 Currently host, target is in scenario, but as a input, we prefer it in scenario['options']. So I add support for under scenario['options']['server_name'] If we write host in scenario['options']['server_name'], the host ip info will be written in context. Change-Id: I90df20467ef5da772d22e9f272a2cac250f822e0 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-09-13Heat: support non-mesh network toplogyRoss Brattain1-6/+4
Previsouly we added all servers to every network in Heat in a full mesh. To more closely replicate test topology and to limit then number of ports we need to all each server to specify which ports should be connected to each network. This should also allow for some kind of multiport setup. Add optional network_ports dict to each server with network to port_list mapping match inteface based on port name or vld_id replace vld_id matching with network name matching, since network_name == vld_id Change-Id: I5de46b8f673949e3c17d8df6fa96f055c43886ce Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-09-14improve logging, clear using printrexlee87761-17/+15
Change-Id: I744353f631cf1771d75f750543e8612f81be71ee Signed-off-by: rexlee8776 <limingjiang@huawei.com>
2017-09-11Log each test case status in a taskchenjiankun1-2/+5
JIRA: YARDSTICK-785 Currently if one test case failed, we will log error. But if one case success, we do not have any tips. We need log success when one case success. Change-Id: I0f41ac55f2569f44b787133e3f2594a5c5547f4a Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-08-28Kubernetes context adoption when run in CIchenjiankun1-15/+0
JIRA: YARDSTICK-801 We need kubernetes test case run in CI. We have one scenario k8-nosdb-lb-noha for kubernetes. But for now, we need to do some adoption in CI script. Also I move check OS_AUTH_URL in heat.py. Change-Id: Iff86caf77fd74e70469ad2dc43f2238097ce0001 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-08-17Merge "utils: create TASK_LOG_DIR if it doesn't exist"Ross Brattain1-0/+1
2017-08-15YAML fixesRoss Brattain1-3/+4
There are multiple issues wiht YAML loading. 1. Jinja2 renders None values as a string 'None'. This is not valid YAML we need to render None values to '~' or 'null' which is the native YAML None value. 2. Jinja2 renders dict and lists that contain unicode with u'foo' values. This is not value YAML syntax. Because we are serializing dict and lists into YAML, we need to encode them as valid YAML. We can override Jinja2 finalize to use yaml.dump to dump inline YAML. We use yaml.safe_dump(elem, default_flow_style=True).replace('\n', '') to generate valid single-line YAML dict and list values. But this problem highlights the general difficulties with templating and loading files. We could avoid this Python->Jinja2->YAML->Python issue by directly injecting the list or dict after the YAML is loaded. I'm not sure of the real utility of these templates. 3. On Python 2 YAML loader is rendering all strings as unicode. This does not work for Trex because Trex is broken and badly coded. Trex does type checking against str() which is different for Python 2 and Python 3. The default YAML loader will return native string types, str() or unicode() for Python 2 and Python 3 respectively. The bad Trex codes is in convert_val: https://github.com/cisco-system-traffic-generator/trex-core/blob/master/scripts/automation/trex_control_plane/stl/trex_stl_lib/trex_stl_packet_builder_scapy.py#L674 def convert_val (val): if is_integer(val): return val if type(val) == str: return ipv4_str_to_num (is_valid_ipv4(val)) raise CTRexPacketBuildException(-11,("init val invalid %s ") % val ); This code is doing type(val) == str. This is bad and broken. We can't fix Trex, so we have to render all strings as native str() types The bug here was that the Heat template loader template_format.py was overriding the global YAML loader to always return unicode. We don't want this global override. To fix this we have to use local subclasses of the yaml.SafeLoader class. But in order to dynamically subclass from CSafeLoader or SafeLoader we have to use the type() builtin to define a new class at runtime. Once we have new classes defined, we can safely isolate different YAML constructors and return unicode or not depending on the case. To be consistent we implement a new yaml_loader.py module to centralize all non-Heat template yaml loading to ensure correct uncode/str conversion Change-Id: Iebf9cf78fbda390977c390436b0869e7bbf503eb Signed-off-by: Ross Brattain <ross.b.brattain@intel.com> Signed-off-by: Deepak S <deepak.s@linux.intel.com> Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-08-14Add yardstick report for each taskchenjiankun1-0/+10
JIRA: YARDSTICK-784 Currently we have yardstick report in GUI. But if users do not use GUI, they can't see this report. So we need generate a report each task. After run each test case, we can see report: http://ip:port/report/report.html Change-Id: Ic76cf57f55aa6680b91272e210135136f0225373 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-08-13utils: create TASK_LOG_DIR if it doesn't existRoss Brattain1-0/+1
Change-Id: I10630599065e6c159b137bd4d6ee89353a1c411b Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-08-11Add real time log view in GUIchenjiankun1-2/+15
JIRA: YARDSTICK-775 We have GUI now, but we can't see real time log in GUI view. So I add real time log view in GUI. Change-Id: Ie83f327ef0a94302afa6b3def764fec6ef5818d1 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-08-10bugfix: tc037/38 multiprocessing failed sometimesrexlee87761-4/+4
set runner.join timeout JIRA: YARDSTICK-771 Change-Id: Iec7d04549a45c0fbd0eb48dcaad7770fd5ca1f4e Signed-off-by: rexlee8776 <limingjiang@huawei.com>
2017-08-09bugfix: tc025 and tc054 failrexlee87761-2/+3
also modify tc025 an tc054 to be able to pass paras JIRA: YARDSTICK-772 Change-Id: Ibeba931804cccfd74fc70fcf4fdb6af9d5c8ab77 Signed-off-by: rexlee8776 <limingjiang@huawei.com>
2017-08-08NSB updateDeepak S1-2/+6
Refactored main NSB VNF classes accroding to class diagram https://wiki.opnfv.org/display/yardstick/NSB+class+diagram All the SampleVNFs have been separated and placed under the SampleVNF class. Added AutoConnectSSH to automatically create SSH conneciton on demand. Added VnfdHelper class to wrap the VNFD dictionary in prepartion for class-based modeling. Extracted DpdkVnfSetupEnvHelper for DPDK based VNF setup. Extracted Stats and other client config to ResourceHelper Had to replace dict_key_flatten with deepgetitem due to Python 2.7 Jinja2 infinite recursion. Change-Id: Ia8840e9c44cdbdf39aab6b02e6d2176b31937dc9 Signed-off-by: Deepak S <deepak.s@linux.intel.com> Signed-off-by: Edward MacGillivray <edward.s.macgillivray@intel.com> Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-08-01replace yaml.load with yaml.safe_loadRoss Brattain1-2/+2
yaml.safe_load is safer, obviously. anteater will check for this template_format use specialized constructor based on yaml.SafeLoader JIRA: YARDSTICK-760 Change-Id: Ia3b0b3aa0765385a0ee472a4d83f49d424b5a77f Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-07-27Adding support for multi-dispatcherDeepak S1-3/+13
JIRA: YARDSTICK-684 This patch adds the support to enable multi-dispatcher of result. e.g. [DEFAULT] debug = False dispatcher = file, http Change-Id: Id0e1a1bba14edd899fcf5b275be1f0a091b6db77 Signed-off-by: Deepak S <deepak.s@linux.intel.com> Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-07-25Merge "yardstick env influxdb/grafana cmd support centos"Rex Lee1-1/+6
2017-07-19yardstick env influxdb/grafana cmd support centoschenjiankun1-1/+6
JIRA: YARDSTICK-714 Currently yardstick env influxdb/grafana command do not support centos. Because we use the gateway ip to get the service of influxdb and grafana. But in centos, we can not access influxdb/grafana service via gateway ip. In this patch, I use docker inspect to get the ip of influxdb and grafana. So these command can support centos. Change-Id: Ie4abcb8048b92d61bc62777dfc362bb29f354b2b Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2017-07-13add network info to topologyRoss Brattain1-8/+22
we need to know which network each port is connected to so we can find VLAN or VXLAN ID. To do this we implement a new method for Contexts, Context.get_network(). This method is similar to Context.get_server(), it searches for a given network name in all the contexts. From this we generate a context_cfg["networks"] dict that stores all the network info for the nodes in the scenario. Then when we generate the topology for VNFD, we can lookup a given network by the vld_id and get the network_type, segmentation_id, etc. Then if we need to for example generated traffic on a given VLAN or VXLAN, we have this info available. Define default nd_route_tbl for ACL VNF we need default empty nd_route_tbl for IPv6 route. Change-Id: I9f9cfbd6acabeb4ae4675ca7354390efa57b29e7 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com> Signed-off-by: Edward MacGillivray <edward.s.macgillivray@intel.com>
2017-07-06Merge "Bugfix: Abnormally stopped background scenario causes test data lost"Rex Lee1-4/+0
2017-07-06Merge "Bugfix: test case constraint not work if "INSTALLER_TYPE" and ↵Kubi1-3/+3
"NODE_NAME" absent"
2017-07-05Merge "unify pod keywork so api can easily used"Kubi1-1/+1
2017-07-04Bugfix: test case constraint not work if "INSTALLER_TYPE" and "NODE_NAME" absentJingLu51-3/+3
Change-Id: Idffc07c28fabda590b58c9501f388099f84f06b2 Signed-off-by: JingLu5 <lvjing5@huawei.com>
2017-07-04Bugfix: Abnormally stopped background scenario causes test data lostJingLu51-4/+0
Change-Id: I19b8d1c1c6becf4335537d585046cbd59cab5935 Signed-off-by: JingLu5 <lvjing5@huawei.com>