aboutsummaryrefslogtreecommitdiffstats
path: root/yardstick/network_services/vnf_generic/vnfdgen.py
AgeCommit message (Collapse)AuthorFilesLines
2017-08-15YAML fixesRoss Brattain1-3/+6
There are multiple issues wiht YAML loading. 1. Jinja2 renders None values as a string 'None'. This is not valid YAML we need to render None values to '~' or 'null' which is the native YAML None value. 2. Jinja2 renders dict and lists that contain unicode with u'foo' values. This is not value YAML syntax. Because we are serializing dict and lists into YAML, we need to encode them as valid YAML. We can override Jinja2 finalize to use yaml.dump to dump inline YAML. We use yaml.safe_dump(elem, default_flow_style=True).replace('\n', '') to generate valid single-line YAML dict and list values. But this problem highlights the general difficulties with templating and loading files. We could avoid this Python->Jinja2->YAML->Python issue by directly injecting the list or dict after the YAML is loaded. I'm not sure of the real utility of these templates. 3. On Python 2 YAML loader is rendering all strings as unicode. This does not work for Trex because Trex is broken and badly coded. Trex does type checking against str() which is different for Python 2 and Python 3. The default YAML loader will return native string types, str() or unicode() for Python 2 and Python 3 respectively. The bad Trex codes is in convert_val: https://github.com/cisco-system-traffic-generator/trex-core/blob/master/scripts/automation/trex_control_plane/stl/trex_stl_lib/trex_stl_packet_builder_scapy.py#L674 def convert_val (val): if is_integer(val): return val if type(val) == str: return ipv4_str_to_num (is_valid_ipv4(val)) raise CTRexPacketBuildException(-11,("init val invalid %s ") % val ); This code is doing type(val) == str. This is bad and broken. We can't fix Trex, so we have to render all strings as native str() types The bug here was that the Heat template loader template_format.py was overriding the global YAML loader to always return unicode. We don't want this global override. To fix this we have to use local subclasses of the yaml.SafeLoader class. But in order to dynamically subclass from CSafeLoader or SafeLoader we have to use the type() builtin to define a new class at runtime. Once we have new classes defined, we can safely isolate different YAML constructors and return unicode or not depending on the case. To be consistent we implement a new yaml_loader.py module to centralize all non-Heat template yaml loading to ensure correct uncode/str conversion Change-Id: Iebf9cf78fbda390977c390436b0869e7bbf503eb Signed-off-by: Ross Brattain <ross.b.brattain@intel.com> Signed-off-by: Deepak S <deepak.s@linux.intel.com> Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-08-15deepgetitem: try string index before intRoss Brattain1-4/+9
Change-Id: I3b353c5887db0ebbe33e37db505e72b85167b54e Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-08-08NSB updateDeepak S1-30/+34
Refactored main NSB VNF classes accroding to class diagram https://wiki.opnfv.org/display/yardstick/NSB+class+diagram All the SampleVNFs have been separated and placed under the SampleVNF class. Added AutoConnectSSH to automatically create SSH conneciton on demand. Added VnfdHelper class to wrap the VNFD dictionary in prepartion for class-based modeling. Extracted DpdkVnfSetupEnvHelper for DPDK based VNF setup. Extracted Stats and other client config to ResourceHelper Had to replace dict_key_flatten with deepgetitem due to Python 2.7 Jinja2 infinite recursion. Change-Id: Ia8840e9c44cdbdf39aab6b02e6d2176b31937dc9 Signed-off-by: Deepak S <deepak.s@linux.intel.com> Signed-off-by: Edward MacGillivray <edward.s.macgillivray@intel.com> Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-07-24vnfdgen: replace yaml.load with yaml.safe_loadRoss Brattain1-1/+1
JIRA: YARDSTICK-760 Change-Id: I145c948c1f08562e12877f5d0fe7e797e9d989f9 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-06-20Acquire NSB specific data from Heat.Edward MacGillivray1-2/+16
First we add mac_address, subnet_cidr to Heat template outputs Then we convert those into a form for NSB and add vld_id. NSB also requires PCI Bus ID, kernel driver and dpdk_port_num. We get this by ssh-ing into instance and dumping sysfs We also need to fix allow for ssh key auth, and implement relative path file loading so NSB can find all its YAML files JIRA: YARDSTICK-580 Change history: don't hide heat create tracebacks we need tracebacks for debug vnf_generic: add task_path to scenario so we can load relative paths for vnf_generic we want to be able to load yaml relative to the task path For example: traffic_profile: ../../traffic_profiles/fixed.yaml topology: ping_tg_topology.yaml # TODO: look in relative path where the tc.yaml is found These need to be relative to samples/vnf_samples/nsut/ping/tc_ping_heat_context.yaml Add a scenario["task_path"] entry heat: log actual exception vnf_generic: replace list with set and iterate over values() some general refactors to remove redundact lookups and type conversions heat: provide mac_address, device_id and network_id from outputs We may need more information to dynamically determine test topology. Towards this end return more info in the heat template. We can return mac_address, device_id and network_id. Once we have this info we can add it to the context_cfg as an interfaces dict. add sample vnf ping multi-network test this test requires 3 network, one for mgmt and the other two for NSB traffic tests We have to make sure we don't use DPDK on mgmt interface because DPDK unbinds the driver heat: convert networks to OrderedDict so we can lookups networks as well as iterate over them in consisitent order heat: and vld_id to networks for vnf_generic vnf_generic uses vld_id Virtual Link Descriptor ID to identify interfaces Add the key to the networks dict and store in Networks object implement relative path file loading in vnf_generic in multiple places we need to load a file relative to the task path, so add open_relative_file_path and modify load_vnf_model to include the scenario_cfg parameter so we have access to task_path DRAFT: heat timeout support Heat stack in CI job failed due to some Nova issue. But then apparently yardstick kept running and took 180mins to timeout https://build.opnfv.org/ci/view/bottlenecks/job/bottlenecks-compass-posca_stress_ping-baremetal-daily-master/16/console We can add a Heat create timeout and fail faster if there is an error. The question is how long should we wait for a Heat stack to deploy. We can set a default and allow override in the heat context config, if users make complicated stacks heat: get netmask and gateway from heat outputs we have do some tricky business with finding the subnet cidr and converting it into netmask vnf_generic: get vpci, driver and dpdk_port_num use a big old find command to dump all the sysfs netdev info nicely. This was re-used from autotest FCoE tests. r"""find /sys/devices/pci* -type d -name net -exec sh -c '{ grep -sH ^ \ +$1/ifindex $1/address $1/operstate $1/device/vendor $1/device/device \ +$1/device/subsystem_vendor $1/device/subsystem_device ; \ +printf "%s/driver:" $1 ; basename $(readlink -s $1/device/driver); } \ +' sh \{\}/* \; This finds all PCI devices that are network devices, then dumps all the relevant info using /bin/sh. Then we parse this into a 'netdevs' dict inside the node_dict and also convert into VNF fields we need. vnf_generic: set node name for kpis node is a dict, so we have to use node_name vnfdgen: we CANNOT use TaskTemplate.render because it does not allow for missing variables, we need to allow password for key_filename to be undefined remove default ssh password hack, once rendering is fixed add new example tc_external_ping_heat_context Change-Id: If1fe0c1a2ab0a5be17e40790a66f28f706fa44d6 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com> Signed-off-by: Edward MacGillivray <edward.s.macgillivray@intel.com>
2017-04-11standardize ssh authRoss Brattain1-0/+2
we need to be following defautl paramiko rules, first use pkey, then key_filenames (autodetecting ~/.ssh/ keys), then password We have too much boilerplate redudant code everywhere, we need to standardize on a factory function that takes a node dict. Using Python3 ChainMap we can layer overrides and defaults. VNF descriptors have to default key_filename, password to Python None. The only way to do this is to omit key values if the variable is not defined, this way the dict will not have the value and it will default to Python None Add python2 chainmap backport Updated unittest mocking to use ssh.SSH.from_node Change-Id: I80b0cb606e593b33e317c9e5e8ed0b74da591514 Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-02-08vnfdgen: replace six.iteritems() with .items()Ross Brattain1-3/+3
Openstack python3 style guide prefers to not use six.iteritems() but instead use data.items() for all cases. We should follow the Openstack guides since OPNFV is aligned with Openstack https://wiki.openstack.org/wiki/Python3#Port_Python_2_code_to_Python_3 Common patterns: * Replace "for key in dict.iterkeys()" with "for key in dict" * Replace dict.iteritems() with dict.items() * Replace dict.itervalues() with dict.values() Change-Id: I99b0505199a568771341658caf8b71c1896db9cd Signed-off-by: Ross Brattain <ross.b.brattain@intel.com>
2017-01-19Introducing Generic framework to do pre-deployment VNF & Network service testingDeepak S1-0/+71
This patch introduces the framework which is aligned with ETSI-TST001 This patch adds: 1. NetworkServiceTestCase introduces following functions --> setup --> Verify if infrastructure mapping can meet topology --> Load VNF models --> Fill traffic profile with information from topology --> Provision VNFs --> Run experiment (traffic) --> run -> Yardstick calls run() at intervals defined in the yaml and produces timestamped samples --> teardown --> Stop VNFs 2. TrafficProfile is a generic class to get traffic profile for a given testcase and select the traffic generator for testcase. 3. QueueFileWrapper is a class to send/recive cmds to vnf 4. GenericVNF is a generic class to instantiate VNF 5. GenericTrafficGen is a generic class to run/listen/verify traffic. JIRA: YARDSTICK-483 Change-Id: Ic453c917d34dcb508a7f3afb459011da85f6402e Signed-off-by: Deepak S <deepak.s@linux.intel.com>