Age | Commit message (Collapse) | Author | Files | Lines |
|
JIRA: YARDSTICK-855
1. using prettytable to reformat cli below:
yardstick runner list
yardstick scenario list
yardstick testcase list
2. remove redundant function print_hbar
3. fix pep8 problems, the framework using do_list dynamicly,
so disable unused-argument check for now.
Change-Id: I79e1226e21fca4624bf4436414917bd07ecabca0
Signed-off-by: rexlee8776 <limingjiang@huawei.com>
|
|
Created two new class functions for Scenario class:
* "get_description" will return the first line of the docstring,
if exists.
* "get_scenario_type" will return the value of "__scenario_type__"
variable.
Change-Id: I8e3b47e33e0bae101b874c3975bb9c383397d188
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>
|
|
I'm not sure why we hardcoded config_file when
the CONF_FILE constant will work.
Change-Id: I45a1dc17ac2b185c659725f55462214acb74838d
Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
|
|
We were not draining the queues in the background
runners.
Modify the main runner_join loop to loop over
all the runners and drain them.
The runner join method does extra work for periodic actions
so we can't change its behavior.
Instead create a new poll() method and use that
to check runner status
Change-Id: I9466ba40a6a4c45c82cedff279cbb4817c6b66ad
Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
|
|
Sometimes the runners can hang. Initially
debugging lead to the queue join thread, so I thought
we could cancel all the join threads and everything would be okay.
But it turns out canceling the queue join threads can lead
to corruption of the queues, so when we go to drain the queues
the task hangs.
But it also turns out that we were not properly draining
the queues in the task process. We were waiting for all
the runners to exit, then draining the queues.
This is bad and will cause the queues to fill up and hang
and/or drop data or corrupt the queues.
The proper fix seems to be to draining the queues in a
loop before calling join with a timeout.
Also modified the queue drain loops to no block on queue.get()
Revert "cancel all queue join threads"
This reverts commit 75c0e3a54b8f6e8fd77c7d9d95decab830159929.
Revert "duration runner: add teardown and cancel all queue join threads"
This reverts commit 7eb6abb6931b24e085b139cc3500f4497cdde57d.
Change-Id: Ic4f8e814cf23615621c1250535967716b425ac18
Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
|
|
JIRA: YARDSTICK-785
Change-Id: Ib37498e8df6a520f1d03256b73346fcedab3a177
Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
|
|
|
|
JIRA: YARDSTICK-803
Currently kubernetes test case can only run in master node.
We need to support it run in jump server.
So I add service and use nodePort type.
Then we can login the pod using nodePort.
Change-Id: Ia7900d263f1c5323f132435addec27ad10547ef9
Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
|
|
|
|
We want to generate pod.yaml from Heat contexts so we can
re-use the context without destroying it.
But we don't have node role information and it doesn't
make sense in this case, so make the role optional.
Since we changed Heat to use pkey instead of key_filename,
we can embed the pkey into the pod.yaml, but we have
to make sure to convert the pkey to string, in case
it is a RSAKey object
Change-Id: Ibefcfbd8236e68013a704c39964cb870da825da8
Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
|
|
|
|
JIRA: YARDSTICK-810
Currently host, target is in scenario, but as a input, we prefer it in
scenario['options'].
So I add support for under scenario['options']['server_name']
If we write host in scenario['options']['server_name'], the host ip info
will be written in context.
Change-Id: I90df20467ef5da772d22e9f272a2cac250f822e0
Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
|
|
Previsouly we added all servers to every network
in Heat in a full mesh.
To more closely replicate test topology and to limit
then number of ports we need to all each server
to specify which ports should be connected to each network.
This should also allow for some kind of multiport setup.
Add optional network_ports dict to each server with network to port_list
mapping
match inteface based on port name or vld_id
replace vld_id matching with network name matching, since network_name == vld_id
Change-Id: I5de46b8f673949e3c17d8df6fa96f055c43886ce
Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
|
|
Change-Id: I744353f631cf1771d75f750543e8612f81be71ee
Signed-off-by: rexlee8776 <limingjiang@huawei.com>
|
|
JIRA: YARDSTICK-785
Currently if one test case failed, we will log error.
But if one case success, we do not have any tips.
We need log success when one case success.
Change-Id: I0f41ac55f2569f44b787133e3f2594a5c5547f4a
Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
|
|
JIRA: YARDSTICK-801
We need kubernetes test case run in CI.
We have one scenario k8-nosdb-lb-noha for kubernetes.
But for now, we need to do some adoption in CI script.
Also I move check OS_AUTH_URL in heat.py.
Change-Id: Iff86caf77fd74e70469ad2dc43f2238097ce0001
Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
|
|
|
|
There are multiple issues wiht YAML loading.
1. Jinja2 renders None values as a string 'None'. This is not valid YAML
we need to render None values to '~' or 'null' which is the native YAML
None value.
2. Jinja2 renders dict and lists that contain unicode with
u'foo' values. This is not value YAML syntax.
Because we are serializing dict and lists into YAML, we
need to encode them as valid YAML. We can override Jinja2 finalize to
use yaml.dump to dump inline YAML.
We use yaml.safe_dump(elem, default_flow_style=True).replace('\n', '')
to generate valid single-line YAML dict and list values.
But this problem highlights the general difficulties with templating and
loading files.
We could avoid this Python->Jinja2->YAML->Python issue by directly
injecting the list or dict after the YAML is loaded.
I'm not sure of the real utility of these templates.
3. On Python 2 YAML loader is rendering all strings
as unicode. This does not work for Trex because Trex is broken
and badly coded. Trex does type checking against str() which
is different for Python 2 and Python 3.
The default YAML loader will return native string types, str() or unicode()
for Python 2 and Python 3 respectively.
The bad Trex codes is in convert_val:
https://github.com/cisco-system-traffic-generator/trex-core/blob/master/scripts/automation/trex_control_plane/stl/trex_stl_lib/trex_stl_packet_builder_scapy.py#L674
def convert_val (val):
if is_integer(val):
return val
if type(val) == str:
return ipv4_str_to_num (is_valid_ipv4(val))
raise CTRexPacketBuildException(-11,("init val invalid %s ") % val );
This code is doing type(val) == str. This is bad and broken.
We can't fix Trex, so we have to render all strings as native str() types
The bug here was that the Heat template loader template_format.py
was overriding the global YAML loader to always return unicode.
We don't want this global override.
To fix this we have to use local subclasses of the yaml.SafeLoader
class.
But in order to dynamically subclass from CSafeLoader or SafeLoader
we have to use the type() builtin to define a new class at runtime.
Once we have new classes defined, we can safely isolate different
YAML constructors and return unicode or not depending on the case.
To be consistent we implement a new yaml_loader.py module to centralize
all non-Heat template yaml loading to ensure correct uncode/str
conversion
Change-Id: Iebf9cf78fbda390977c390436b0869e7bbf503eb
Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
Signed-off-by: Deepak S <deepak.s@linux.intel.com>
Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
|
|
JIRA: YARDSTICK-784
Currently we have yardstick report in GUI.
But if users do not use GUI, they can't see this report.
So we need generate a report each task.
After run each test case, we can see report:
http://ip:port/report/report.html
Change-Id: Ic76cf57f55aa6680b91272e210135136f0225373
Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
|
|
Change-Id: I10630599065e6c159b137bd4d6ee89353a1c411b
Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
|
|
JIRA: YARDSTICK-775
We have GUI now, but we can't see real time log in GUI view.
So I add real time log view in GUI.
Change-Id: Ie83f327ef0a94302afa6b3def764fec6ef5818d1
Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
|
|
set runner.join timeout
JIRA: YARDSTICK-771
Change-Id: Iec7d04549a45c0fbd0eb48dcaad7770fd5ca1f4e
Signed-off-by: rexlee8776 <limingjiang@huawei.com>
|
|
also modify tc025 an tc054 to be able to pass paras
JIRA: YARDSTICK-772
Change-Id: Ibeba931804cccfd74fc70fcf4fdb6af9d5c8ab77
Signed-off-by: rexlee8776 <limingjiang@huawei.com>
|
|
Refactored main NSB VNF classes accroding to class diagram
https://wiki.opnfv.org/display/yardstick/NSB+class+diagram
All the SampleVNFs have been separated and placed under
the SampleVNF class.
Added AutoConnectSSH to automatically create SSH conneciton on demand.
Added VnfdHelper class to wrap the VNFD dictionary in prepartion for
class-based modeling.
Extracted DpdkVnfSetupEnvHelper for DPDK based VNF setup.
Extracted Stats and other client config to ResourceHelper
Had to replace dict_key_flatten with deepgetitem due to Python 2.7
Jinja2 infinite recursion.
Change-Id: Ia8840e9c44cdbdf39aab6b02e6d2176b31937dc9
Signed-off-by: Deepak S <deepak.s@linux.intel.com>
Signed-off-by: Edward MacGillivray <edward.s.macgillivray@intel.com>
Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
|
|
yaml.safe_load is safer, obviously.
anteater will check for this
template_format use specialized constructor based on yaml.SafeLoader
JIRA: YARDSTICK-760
Change-Id: Ia3b0b3aa0765385a0ee472a4d83f49d424b5a77f
Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
|
|
JIRA: YARDSTICK-684
This patch adds the support to enable multi-dispatcher of result.
e.g.
[DEFAULT]
debug = False
dispatcher = file, http
Change-Id: Id0e1a1bba14edd899fcf5b275be1f0a091b6db77
Signed-off-by: Deepak S <deepak.s@linux.intel.com>
Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
|
|
JIRA: YARDSTICK-762
Change-Id: Id4a94ef8bbe578944e3664e5e7c24fdf7478805b
Signed-off-by: JingLu5 <lvjing5@huawei.com>
|
|
|
|
|
|
JIRA: YARDSTICK-747
API: /api/v2/yardstick/testsuites
METHOD: GET
Change-Id: I2375b6d2ea54704f5a2328cc8110058af038e777
Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
|
|
JIRA: YARDSTICK-714
Currently yardstick env influxdb/grafana command do not support centos.
Because we use the gateway ip to get the service of influxdb and grafana.
But in centos, we can not access influxdb/grafana service via gateway ip.
In this patch, I use docker inspect to get the ip of influxdb and grafana.
So these command can support centos.
Change-Id: Ie4abcb8048b92d61bc62777dfc362bb29f354b2b
Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
|
|
JIRA: YARDSTICK-712
Change-Id: If91c936a3a59580a987cb6762db5d063edf6c9d7
Signed-off-by: JingLu5 <lvjing5@huawei.com>
|
|
we need to know which network each port is connected to
so we can find VLAN or VXLAN ID.
To do this we implement a new method for Contexts,
Context.get_network(). This method is similar to
Context.get_server(), it searches for a given
network name in all the contexts.
From this we generate a context_cfg["networks"]
dict that stores all the network info for the nodes
in the scenario.
Then when we generate the topology for VNFD, we can
lookup a given network by the vld_id and get the
network_type, segmentation_id, etc.
Then if we need to for example generated
traffic on a given VLAN or VXLAN, we have this
info available.
Define default nd_route_tbl for ACL VNF
we need default empty nd_route_tbl for IPv6 route.
Change-Id: I9f9cfbd6acabeb4ae4675ca7354390efa57b29e7
Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
Signed-off-by: Edward MacGillivray <edward.s.macgillivray@intel.com>
|
|
|
|
"NODE_NAME" absent"
|
|
|
|
Change-Id: Idffc07c28fabda590b58c9501f388099f84f06b2
Signed-off-by: JingLu5 <lvjing5@huawei.com>
|
|
Change-Id: I19b8d1c1c6becf4335537d585046cbd59cab5935
Signed-off-by: JingLu5 <lvjing5@huawei.com>
|
|
JIRA: YARDSTICK-700
In this patch:
https://gerrit.opnfv.org/gerrit/#/c/35257/
In yardstick/benchmark/core/task.py
We update background result only when it be stopped abnormally.
We need move it out of 'if' statement.
Change-Id: I25b76e160273a87da66098565a16e4a143cec11f
Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
|
|
JIRA: YARDSTICK-695
Change-Id: I826cb2ea07949b1eaae4df4cb77175ec8f5df89b
Signed-off-by: rexlee8776 <limingjiang@huawei.com>
|
|
JIRA: YARDSTICK-688
We need to call core code directly in the API of runTestCase.
It would be more stable.
Change-Id: I431a85ded7cd3b20da0462f947c25d91bb99decd
Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
|
|
|
|
|
|
JIRA: YARDSTICK-658
Currently the yardstick have three dispatcher: file, influxdb, mongodb.
(influxdb using API to get result and mongodb using testAPI to get result)
But their output format is different. It is hard to use.
In this patch, make all dispatchers using the same data source.
And make the output format of file and influxdb unified.
As for mongodb, since it is related to testAPI, so I make it push data
every test case.
The unified output format is:
http://paste.openstack.org/show/610125/
Change-Id: I854ac4f03e6f904469b07b0c924c7d850545ae5b
Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
|
|
First we add mac_address, subnet_cidr to Heat template outputs
Then we convert those into a form for NSB and add vld_id.
NSB also requires PCI Bus ID, kernel driver
and dpdk_port_num. We get this by ssh-ing
into instance and dumping sysfs
We also need to fix allow for ssh key auth,
and implement relative path file loading
so NSB can find all its YAML files
JIRA: YARDSTICK-580
Change history:
don't hide heat create tracebacks we need tracebacks for debug
vnf_generic: add task_path to scenario so we can load relative paths
for vnf_generic we want to be able to load yaml relative to the
task path
For example:
traffic_profile: ../../traffic_profiles/fixed.yaml
topology: ping_tg_topology.yaml # TODO: look in relative path where the tc.yaml is found
These need to be relative to samples/vnf_samples/nsut/ping/tc_ping_heat_context.yaml
Add a scenario["task_path"] entry
heat: log actual exception
vnf_generic: replace list with set and iterate over values()
some general refactors to remove redundact lookups and
type conversions
heat: provide mac_address, device_id and network_id from outputs
We may need more information to dynamically
determine test topology.
Towards this end return more info in the heat template.
We can return mac_address, device_id and network_id.
Once we have this info we can add it to the context_cfg
as an interfaces dict.
add sample vnf ping multi-network test
this test requires 3 network, one for mgmt
and the other two for NSB traffic tests
We have to make sure we don't use DPDK
on mgmt interface because DPDK unbinds
the driver
heat: convert networks to OrderedDict
so we can lookups networks as well as
iterate over them in consisitent order
heat: and vld_id to networks for vnf_generic
vnf_generic uses vld_id Virtual Link Descriptor ID
to identify interfaces
Add the key to the networks dict
and store in Networks object
implement relative path file loading in vnf_generic
in multiple places we need to load a file
relative to the task path, so add
open_relative_file_path
and modify load_vnf_model to include the scenario_cfg
parameter so we have access to task_path
DRAFT: heat timeout support
Heat stack in CI job failed due to some Nova issue.
But then apparently yardstick kept running and took 180mins to timeout
https://build.opnfv.org/ci/view/bottlenecks/job/bottlenecks-compass-posca_stress_ping-baremetal-daily-master/16/console
We can add a Heat create timeout and fail faster if there is an error.
The question is how long should we wait for a Heat stack to deploy. We
can set a default and allow override in the heat context config, if
users make complicated stacks
heat: get netmask and gateway from heat outputs
we have do some tricky business with finding
the subnet cidr and converting it into netmask
vnf_generic: get vpci, driver and dpdk_port_num
use a big old find command to dump all the sysfs
netdev info nicely. This was re-used from autotest FCoE tests.
r"""find /sys/devices/pci* -type d -name net -exec sh -c '{ grep -sH ^ \
+$1/ifindex $1/address $1/operstate $1/device/vendor $1/device/device \
+$1/device/subsystem_vendor $1/device/subsystem_device ; \
+printf "%s/driver:" $1 ; basename $(readlink -s $1/device/driver); } \
+' sh \{\}/* \;
This finds all PCI devices that are network devices, then
dumps all the relevant info using /bin/sh.
Then we parse this into a 'netdevs' dict inside the node_dict
and also convert into VNF fields we need.
vnf_generic: set node name for kpis
node is a dict, so we have to use node_name
vnfdgen: we CANNOT use TaskTemplate.render
because it does not allow for missing variables, we need to allow
password for key_filename to be undefined
remove default ssh password hack, once rendering is fixed
add new example tc_external_ping_heat_context
Change-Id: If1fe0c1a2ab0a5be17e40790a66f28f706fa44d6
Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
Signed-off-by: Edward MacGillivray <edward.s.macgillivray@intel.com>
|
|
JIRA: YARDSTICK-641
Allowing parameters to pass between scenarios so that the one test case
can be combination of several scenarios.
Change-Id: I55a00855e77d5b719a27a069a3ea195d6bbd0ef8
Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
|
|
we need to be following defautl paramiko rules,
first use pkey, then key_filenames (autodetecting ~/.ssh/ keys),
then password
We have too much boilerplate redudant code everywhere, we need
to standardize on a factory function that takes a node dict.
Using Python3 ChainMap we can layer overrides and defaults.
VNF descriptors have to default key_filename, password to Python None.
The only way to do this is to omit key values if the variable is not
defined, this way the dict will not have the value and it will
default to Python None
Add python2 chainmap backport
Updated unittest mocking to use ssh.SSH.from_node
Change-Id: I80b0cb606e593b33e317c9e5e8ed0b74da591514
Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
|
|
JIRA: YARDSTICK-620
Currently we recommend using docker to run yardstick.
And it is hard to use virtualenv to install yardstick.
So I modify install.sh in yardstick root path.
It will support using virtualenv to install yardstick(including API) in
linux.
In this patch, I make yardstick support read yardstick configuration
have priority over constants.
Change-Id: I9ea1241b228532a6497451e6c8f232173ddb783e
Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
|
|
|
|
return True
JIRA: YARDSTICK-600
Bugfix: netperf_bottleneck execute failed because is_same_heat_context return True
So it exit without warning.
I will remove the 'return True' from it.
Change-Id: I48d62bd15ec359e2121f236c48d9d349174f2c10
Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
|