diff options
Diffstat (limited to 'docs/release/userguide/collectd.ves.userguide.rst')
| -rw-r--r-- | docs/release/userguide/collectd.ves.userguide.rst | 834 |
1 files changed, 680 insertions, 154 deletions
diff --git a/docs/release/userguide/collectd.ves.userguide.rst b/docs/release/userguide/collectd.ves.userguide.rst index 7077cecd..1f69b6ad 100644 --- a/docs/release/userguide/collectd.ves.userguide.rst +++ b/docs/release/userguide/collectd.ves.userguide.rst @@ -5,57 +5,19 @@ ========================== VES Application User Guide ========================== -The Barometer repository contains a python based application for VES. + +The Barometer repository contains a python based application for VES (VNF Event +Stream) which receives the `collectd`_ specific metrics via `Kafka`_ bus, +normalizes the metric data into the VES message and sends it into the VES +collector. The application currently supports pushing platform relevant metrics through the additional measurements field for VES. -Collectd has a write_kafka plugin that will send collectd metrics and values to -a Kafka Broker. -The VES application uses Kafka Consumer to receive metrics from the Kafka -Broker. - -Installation Instructions: --------------------------- -1. Clone this repo: - - .. code:: bash - - git clone https://gerrit.opnfv.org/gerrit/barometer - -2. Install collectd - - .. code:: bash - - $ sudo apt-get install collectd - - CentOS 7.x use: - - .. code:: bash - - $ sudo yum install collectd +Collectd has a ``write_kafka`` plugin that sends collectd metrics and values to +a Kafka Broker. The VES application uses Kafka Consumer to receive metrics +from the Kafka Broker. - .. note:: You may need to add epel repository if the above does not work. - - .. code:: bash - - $ sudo yun install epel-release - -3. Modify the collectd configuration script: `/etc/collectd/collectd.conf` - - .. code:: bash - - <Plugin write_kafka> - Property "metadata.broker.list" "localhost:9092" - <Topic "collectd"> - Format JSON - </Topic> - </Plugin> - - .. note:: - - The above configuration is for a single host setup. Simply change localhost to remote - server IP addess or hostname. Install Kafka Broker -------------------- @@ -86,7 +48,9 @@ Install Kafka Broker $ sudo yum install zookeeper - .. note:: You may need to add the the repository that contains zookeeper + .. note:: You may need to add the repository that contains zookeeper. + To do so, follow the step below and try to install `zookeeper` again + using steps above. Otherwise, skip next step. .. code:: bash @@ -99,11 +63,20 @@ Install Kafka Broker $ sudo zookeeper-server start + if you get the error message like ``ZooKeeper data directory is missing at /var/lib/zookeeper`` + during the start of zookeeper, initialize zookeeper data directory using + the command below and start zookeeper again, otherwise skip the next step. + + .. code:: bash + + $ sudo /usr/lib/zookeeper/bin/zkServer-initialize.sh + No myid provided, be sure to specify it in /var/lib/zookeeper/myid if using non-standalone + To test if Zookeeper is running as a daemon. .. code:: bash - $ sudo telnet localhost 2181 + $ telnet localhost 2181 Type 'ruok' & hit enter. Expected response is 'imok'. Zookeeper is running fine. @@ -115,25 +88,26 @@ Install Kafka Broker .. code:: bash + $ sudo yum install python-pip $ sudo pip install kafka-python 2. Download Kafka: .. code:: bash - $ sudo wget "http://www-eu.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz" + $ wget "http://www-eu.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz" 3. Extract the archive: .. code:: bash - $ sudo tar -xvzf kafka_2.11-0.11.0.0.tgz + $ tar -xvzf kafka_2.11-0.11.0.0.tgz 4. Configure Kafka Server: .. code:: bash - $ sudo vi kafka_2.11-0.11.0.0/config/server.properties + $ vi kafka_2.11-0.11.0.0/config/server.properties By default Kafka does not allow you to delete topics. Please uncomment: @@ -170,180 +144,732 @@ Install Kafka Broker localhost:2181 --topic TopicTest --from-beginning -VES application configuration description: ------------------------------------------- +Install collectd +---------------- + +Install development tools: -Within the VES directory there is a configuration file called 'ves_app.conf'. +.. code:: bash + + $ sudo yum group install 'Development Tools' -.. note:: Details of the Vendor Event Listener REST service +.. The libkafka installed via yum pkg manager is 0.11.0 which doesn't work with + collectd (compilation issue). Thus, we have to use the library installed + from sources using latest stable version which works with collectd. -REST resources are defined with respect to a ServerRoot: +Install Apache Kafka C/C++ client library: .. code:: bash - ServerRoot = https://{Domain}:{Port}/{optionalRoutingPath} + $ git clone https://github.com/edenhill/librdkafka.git ~/librdkafka + $ cd ~/librdkafka + $ git checkout -b v0.9.5 v0.9.5 + $ ./configure --prefix=/usr + $ make + $ sudo make install + +Build collectd with Kafka support: + +.. code:: bash + + $ git clone https://github.com/collectd/collectd.git ~/collectd + $ cd ~/collectd + $ ./build.sh + $ ./configure --with-librdkafka=/usr --without-perl-bindings --enable-perl=no + $ make && sudo make install + +Configure and start collectd. Create ``/opt/collectd/etc/collectd.conf`` +collectd configuration file as following: + +.. note:: The following collectd configuration file allows user to run VES + application in the guest mode. To run the VES in host mode, please follow + the `Configure VES in host mode`_ steps. + +.. include:: collectd-ves-guest.conf + :code: bash + +Start collectd process as a service as described in :ref:`install-collectd-as-a-service`. + + +Setup VES Test Collector +------------------------ + +.. note:: Test Collector setup is required only for VES application testing + purposes. + +Install dependencies: -REST resources are of the form: +.. code:: bash + + $ sudo pip install jsonschema + +Clone VES Test Collector: + +.. code:: bash + + $ git clone https://github.com/att/evel-test-collector.git ~/evel-test-collector + +Modify VES Test Collector config file to point to existing log directory and +schema file: .. code:: bash + $ sed -i.back 's/^\(log_file[ ]*=[ ]*\).*/\1collector.log/' ~/evel-test-collector/config/collector.conf + $ sed -i.back 's/^\(schema_file[ ]*=.*\)event_format_updated.json$/\1CommonEventFormat.json/' ~/evel-test-collector/config/collector.conf + +Start VES Test Collector: + +.. code:: bash + + $ cd ~/evel-test-collector/code/collector + $ nohup python ./collector.py --config ../../config/collector.conf > collector.stdout.log & + + +Setup VES application (guest mode) +---------------------------------- + +Install dependencies: + +.. code:: bash + + $ sudo pip install pyyaml + +Clone Barometer repo and start the VES application: + +.. code:: bash + + $ git clone https://gerrit.opnfv.org/gerrit/barometer ~/barometer + $ cd ~/barometer/3rd_party/collectd-ves-app/ves_app + $ nohup python ves_app.py --events-schema=guest.yaml --config=ves_app_config.conf > ves_app.stdout.log & + +.. note:: + + The above configuration is used for a localhost. The VES application can be + configured to use remote real VES collector and remote Kafka server. To do + so, the IP addresses/host names needs to be changed in ``collector.conf`` + and ``ves_app_config.conf`` files accordingly. + + +Configure VES in host mode +-------------------------- + +Running the VES in host mode looks like steps described in +`Setup VES application (guest mode)`_ but with the following exceptions: + +- The ``host.yaml`` configuration file should be used instead of ``guest.yaml`` + file when VES application is running. + +- Collectd should be running on host machine only. + +- Addition ``libvirtd`` dependencies needs to be installed on a host where + collectd daemon is running. To install those dependencies, see :ref:`virt-plugin` + section of Barometer user guide. + +- At least one VM instance should be up and running by hypervisor on the host. + +- The next (minimum) configuration needs to be provided to collectd to be able + to generate the VES message to VES collector. + + .. include:: collectd-ves-host.conf + :code: bash + + to apply this configuration, the ``/opt/collectd/etc/collectd.conf`` file + needs to be modified based on example above and collectd daemon needs to + be restarted using the command below: + + .. code:: bash + + $ sudo systemctl restart collectd + +.. note:: The list of the plugins can be extented depends on your needs. + + +VES application configuration description +----------------------------------------- + +**Details of the Vendor Event Listener REST service** + +REST resources are defined with respect to a ``ServerRoot``:: + + ServerRoot = https://{Domain}:{Port}/{optionalRoutingPath} + +REST resources are of the form:: + {ServerRoot}/eventListener/v{apiVersion}` {ServerRoot}/eventListener/v{apiVersion}/{topicName}` {ServerRoot}/eventListener/v{apiVersion}/eventBatch` +Within the VES directory (``3rd_party/collectd-ves-app/ves_app``) there is a +configuration file called ``ves_app.conf``. The description of the +configuration options are described below: + **Domain** *"host"* VES domain name. It can be IP address or hostname of VES collector - (default: `127.0.0.1`) + (default: ``127.0.0.1``) **Port** *port* - VES port (default: `30000`) + VES port (default: ``30000``) **Path** *"path"* - Used as the "optionalRoutingPath" element in the REST path (default: `empty`) + Used as the "optionalRoutingPath" element in the REST path (default: empty) **Topic** *"path"* - Used as the "topicName" element in the REST path (default: `empty`) + Used as the "topicName" element in the REST path (default: empty) **UseHttps** *true|false* - Allow application to use HTTPS instead of HTTP (default: `false`) + Allow application to use HTTPS instead of HTTP (default: ``false``) **Username** *"username"* - VES collector user name (default: `empty`) + VES collector user name (default: empty) **Password** *"passwd"* - VES collector password (default: `empty`) - -**FunctionalRole** *"role"* - Used as the 'functionalRole' field of 'commonEventHeader' event (default: - `Collectd VES Agent`) + VES collector password (default: empty) **SendEventInterval** *interval* This configuration option controls how often (sec) collectd data is sent to - Vendor Event Listener (default: `20`) + Vendor Event Listener (default: ``20``) **ApiVersion** *version* - Used as the "apiVersion" element in the REST path (default: `5.1`) + Used as the "apiVersion" element in the REST path (default: ``5.1``) **KafkaPort** *port* Kafka Port (Default ``9092``) **KafkaBroker** *host* Kafka Broker domain name. It can be an IP address or hostname of local or remote server - (default: localhost) + (default: ``localhost``) -Other collectd.conf configurations ----------------------------------- -Please ensure that FQDNLookup is set to false -.. code:: bash +VES YAML configuration format +----------------------------- - FQDNLookup false +The format of the VES message is generated by the VES application based on the +YAML schema configuration file provided by user via ``--events-schema`` +command-line option of the application. -Please ensure that the virt plugin is enabled and configured as follows. +.. note:: Use ``--help`` option of VES application to see the + description of all available options -.. code:: bash +.. note:: The detailed installation guide of the VES application is described in + the `VES Application User Guide`_. - LoadPlugin virt +The message defined by YAML schema should correspond to format defined in +`VES shema definition`_. - <Plugin virt> - Connection "qemu:///system" - RefreshInterval 60 - HostnameFormat uuid - PluginInstanceFormat name - ExtraStats "cpu_util perf" - </Plugin> +.. warning:: This section doesn't explain the YAML language itself, so the + knowledge of the YAML language is required before continue reading it! +Since, the VES message output is a JSON format, it's recommended to understand +how YAML document is converted to JSON before starting to create a new YAML +definition for the VES. E.g.: -.. note:: For more detailed information on the `virt` plugin configuration, - requirements etc., please see the userguide of the collectd virt plugin. +The following YAML document: -Please ensure that the cpu plugin is enabled and configured as follows +.. code:: yaml -.. code:: bash + --- + additionalMeasurements: + plugin: somename + instance: someinstance - LoadPlugin cpu +will be converted to JSON like this: - <Plugin cpu> - ReportByCpu false - ValuesPercentage true - </Plugin> +.. code:: json -.. note:: + { + "additionalMeasurements": { + "instance": "someinstance", + "plugin": "somename" + } + } - The ``ReportByCpu`` option should be set to `true` (default) - if VES application is running on guest machine ('GuestRunning' = true). +.. note:: The `YAML syntax` section of `PyYAML documentation`_ can be used + as a reference for this. -Please ensure that the aggregation plugin is enabled and configured as follows -(required if 'GuestRunning' = true) -.. code:: bash +VES message types +----------------- - LoadPlugin aggregation +The VES agent can generate two type of messages which are sent to the VES +collector. Each message type must be specified in the YAML configuration file +using a specific YAML tag. - <Plugin aggregation> - <Aggregation> - Plugin "cpu" - Type "percent" - GroupBy "Host" - GroupBy "TypeInstance" - SetPlugin "cpu-aggregation" - CalculateAverage true - </Aggregation> - </Plugin> +Measurements + This type is used to send a message defined in YAML configuration to the VES + collector with a specified interval (default is 20 sec and it's configurable + via command line option of the application). This type can be specified in + the configuration using ``!Measurements`` tag. For instance: -If application is running on a guest side, it is important to enable uuid plugin -too. In this case the hostname in event message will be represented as UUID -instead of system host name. + .. code:: yaml -.. code:: bash + --- + # My comments + My Host Measurements: !Measurements + ... # message definition - LoadPlugin uuid +Events + This type is used to send a message defined in YAML configuration to the VES + collector when collectd notification is received from Kafka bus (collectd + ``write_kafka`` plugin). This type can be specified in the configuration + using ``!Events`` tag. For instance: -If a custom UUID needs to be provided, the following configuration is required in collectd.conf -file: + .. code:: yaml -.. code:: bash + --- + # My comments + My Events: !Events + ... # event definition - <Plugin uuid> - UUIDFile "/etc/uuid" - </Plugin> -Where "/etc/uuid" is a file containing custom UUID. +Collectd metrics in VES +----------------------- -Please also ensure that the following plugins are enabled: +The VES application caches collectd metrics received via Kafka bus. The data +is stored in a table format. It's important to know it before mapping collectd +metric values to message defined in YAML configuration file. -.. code:: bash +VES collectd metric cache example: - LoadPlugin disk - LoadPlugin interface - LoadPlugin memory ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| host | plugin | plugin_instance | type | type_instace | time | value | ds_name | interval | ++===========+===========+====================+===========+================+===================+========+=========+==========+ +| localhost | cpu | | percent | user | 1509535512.30567 | 16 | value | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| localhost | memory | | memory | free | 1509535573.448014 | 798456 | value | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| localhost | interface | | eth0 | if_packets | 1509544183.956496 | 253 | rx | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| 7ec333e7 | virt | Ubuntu-12.04.5-LTS | percent | virt_cpu_total | 1509544402.378035 | 0.2 | value | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| 7ec333e7 | virt | Ubuntu-12.04.5-LTS | memory | rss | 1509544638.55119 | 123405 | value | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| 7ec333e7 | virt | Ubuntu-12.04.5-LTS | if_octets | vnet1 | 1509544646.27108 | 67 | tx | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| cc659a52 | virt | Ubuntu-16.04 | percent | virt_cpu_total | 1509544745.617207 | 0.3 | value | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| cc659a52 | virt | Ubuntu-16.04 | memory | rss | 1509544754.329411 | 4567 | value | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| cc659a52 | virt | Ubuntu-16.04 | if_octets | vnet0 | 1509544760.720381 | 0 | rx | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ -VES application with collectd notifications example ---------------------------------------------------- +It's possible from YAML configuration file to refer to any field of any row of +the table via special YAML tags like ``ValueItem`` or ``ArrayItem``. See the +`Collectd metric reference`_ reference for more details. -A good example of collectD notification is monitoring of the total CPU usage on a VM -using the 'threshold' plugin. The following configuration will setup VES plugin to send 'Fault' -event every time a total VM CPU value is out of range (e.g.: WARNING: VM CPU TOTAL > 50%, -CRITICAL: VM CPU TOTAL > 96%) and send 'Fault' NORMAL event if the CPU value is back -to normal. In the example below, there is one VM with two CPUs configured which is running -on the host with a total of 48 cores. Thus, the threshold value 2.08 (100/48) means that -one CPU of the VM is fully loaded (e.g.: 50% of total CPU usage of the VM) and 4.0 means -96% of total CPU usage of the VM. Those values can also be obtained by virt-top -command line tool. +.. note:: The `collectd data types file`_ contains map of ``type`` to + ``ds_name`` field. It can be used to get possible value for ``ds_name`` + field. See the `collectd data types description`_ for more details on + collectd data types. -.. code:: bash - LoadPlugin threshold +Aging of collectd metric +~~~~~~~~~~~~~~~~~~~~~~~~ + +If the metric will not be updated by the collectd during the double metric +interval time, it will be removed (aged) from VES internal cache. + + +VES YAML references +------------------- + +There are four type of references supported by the YAML format: System, +Config, Collectd metric and Collectd notification. The format of the reference +is the following: + +.. code:: yaml + + "{<ref type>.<attribute name>}" + +.. note:: Possible values for ``<ref type>`` and ``<attribute name>`` are + described in farther sections. + + +System reference +~~~~~~~~~~~~~~~~ + +This reference is used to get system statistics like time, date etc. The system +reference (``<ref type>`` = ``system``) can be used in any place of the YAML +configuration file. This type of reference provides the following attributes: + +``hostname`` + The name of the host where VES application is running. + +``id`` + Unique ID during VES application runtime. + +``time`` + Current time in seconds since the Epoch. For example ``1509641411.631951``. + +``date`` + Current date in ISO 8601 format, ``YYYY-MM-DD``. For instance ``2017-11-02``. + + +For example: + +.. code:: yaml + + Date: "{system.date}" + + +Config reference +~~~~~~~~~~~~~~~~ + +This reference is used to get VES configuration described in `VES application +configuration description`_ sectoin. The reference (``<ref type>`` = ``config``) +can be used in any place of the YAML configuration file. This type of reference +provides the following attributes: + +``interval`` + Measurements dispatch interval. It referenses to ``SendEventInterval`` + configuration of the VES application. + + +For example: + +.. code:: yaml + + Interval: "{config.interval}" + + +Collectd metric reference +~~~~~~~~~~~~~~~~~~~~~~~~~ + +This reference is used to get the attribute value of collectd metric from the +VES cache. The reference (``<ref type>`` = ``vl``) can be used ONLY inside +``Measurements``, ``ValueItem`` and ``ArrayItem`` tags. Using the reference +inside a helper tag is also allowed if the helper tag is located inside the +tag where the reference is allowed (e.g.: ``ArrayItem``). The +``<attribute name>`` name corresponds to the table field name described in +`Collectd metrics in VES`_ section. For example: + +.. code:: yaml + + name: "{vl.type}-{vl.type_instance}" + + +Collectd notification reference +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This reference is used to get the attribute value of received collectd +notification. The reference (``<ref type>`` = ``n``) can be used ONLY inside +``Events`` tag. Using the reference inside a helper tag is also allowed if +the helper tag is located inside the ``Events`` tag. This type of reference +provides the following attributes: + +``host`` + The hostname of received collectd notification. + +``plugin`` + The plugin name of received collectd notification. + +``plugin_instance`` + The plugin instance of the received collectd notification. + +``type`` + The type name of the received collectd notification. + +``type_instance`` + The type instance of the received collectd notification. + +``severity`` + The severity of the received collectd notification. + +``message`` + The message of the received collectd notification. + + +.. note:: The exact value for each attribute depends on the collectd plugin + which may generate the notification. Please refer to the + `collectd plugin description`_ document to get more details on the specific + collectd plugin. + +YAML config example: + +.. code:: yaml + + sourceId: "{n.plugin_instance}" + + +Collectd metric mapping YAML tags +--------------------------------- + +This section describes the YAML tags used to map collectd metric values in the +YAML configuration file. + + +Measurements tag +~~~~~~~~~~~~~~~~ + +This tag is a YAML map which is used to define the VES measurement message. It's +allowed to be used multiple times in the document (e.g.: you can define multiple +VES messages in one YAML document). This tag works in the same way as `ArrayItem +tag`_ does and all keys have the same description/rules. + + +ValueItem tag +~~~~~~~~~~~~~ + +This tag is used to select a collectd metric and get its attribute value using +`Collectd metric reference`_. The type of this tag is a YAML array of maps with +the possible keys described below. + +``SELECT`` (required) + Is a YAML map which describes the select metric criteria. Each key name of the + map must correspond to the table field name described in `Collectd metrics in VES`_ + section. + +``VALUE`` (optional) + Describes the value to be assigned. If not provided, the default + ``!Number "{vl.value}"`` expression is used. + +``DEFAULT`` (optional) + Describes the default value which will be assigned if no metric is selected + by ``SELECT`` criteria. + + +ValueItem tag description example: + +.. code:: yaml + + memoryFree: !ValueItem + - SELECT: + plugin: memory + type: memory + type_instance: rss + - VALUE: !Bytes2Kibibytes "{vl.value}" + - DEFAULT: 0 + +The tag process workflow is described on the figure below. + +.. figure:: value-item-parse-workflow.png + + YAML ValueItem tag process workflow + + +ArrayItem tag +~~~~~~~~~~~~~ + +This tag is used to select a list of collectd metrics and generate a YAML array +of YAML items described by ``ITEM-DESC`` key. If no collectd metrics are +selected by the given criteria, the empty array will be returned. + +``SELECT`` (optional) + Is a YAML map which describes the select metrics criteria. Each key name of + the map must correspond to the table field name described in `Collectd + metrics in VES`_ section. The value of the key may be regular expression. To + enable regular expression in the value, the YAML string containing ``/`` char + at the beginning and at the end should be used. For example: + + .. code:: yaml + + plugin: "/^(?!virt).*$/" # selected all metrics except ``virt`` plugin + + The VES application uses the python RE library to work with regular + expression specified in the YAML configuration. Please refer to `python + regular expression syntax`_ documentation for more details on a syntax + used by the VES. + + Multiple ``SELECT`` keys are allowed by the tag. If nor ``SELECT`` or + ``INDEX-KEY`` key is specified, the VES error is generated. + +``INDEX-KEY`` (optional) + Is a YAML array which describes the unique fields to be selected by the tag. + Each element of array is a YAML string which should be one of the table field + name described in `Collectd metrics in VES`_ section. Please note, if this + key is used, only fields specified by the key can be used as a collectd + reference in the ``ITEM-DESC`` key. + +``ITEM-DESC`` (required) + Is a YAML map which describes each element of the YAML array generated by + the tag. Using ``ArrayItem`` tags and other `Helper YAML tags`_ are also + allowed in the definition of the key. + +In the example below, the ArrayItem tag is used to generate an array of +``ITEM-DESC`` items for each collectd metrics except ``virt`` plugin with +unique ``plugin``, ``plugin_instance`` attribute values. + +.. code:: yaml + + Measurements: !ArrayItem + - SELECT: + plugin: "/^(?!virt).*$/" + - INDEX-KEY: + - plugin + - plugin_instance + - ITEM-DESC: + name: !StripExtraDash "{vl.plugin}-{vl.plugin_instance}" + +The tag process workflow is described on the figure below. + +.. figure:: array-item-parse-workflow.png + + YAML ArrayItem tag process workflow + + +Collectd event mapping YAML tags +-------------------------------- + +This section describes the YAML tags used to map the collectd notification to +the VES event message in the YAML configuration file. + + +Events tag +~~~~~~~~~~ + +This tag is a YAML map which is used to define the VES event message. It's +allowed to be used multiple times in the document (e.g.: you can map multiple +collectd notification into VES message in one YAML document). The possible +keys of the tag are described below. + +``CONDITION`` (optional) + Is a YAML map which describes the select metric criteria. Each key name of + the map must correspond to the name of attribute provided by `Collectd + notification reference`_. If no such key provided, any collectd notification + will map the defined YAML message. + +``ITEM-DESC`` (required) + Is a YAML map which describes the message generated by this tag. Only `Helper + YAML tags`_ are allowed in the definition of the key. + +The example of the VES event message which will be generate by the VES +application when collectd notification of the ``virt`` plugin is triggered +is described below. + +.. code:: yaml + + --- + Virt Event: !Events + - ITEM-DESC: + event: + commonEventHeader: + domain: fault + eventType: Notification + sourceId: &event_sourceId "{n.plugin_instance}" + sourceName: *event_sourceId + lastEpochMicrosec: !Number "{n.time}" + startEpochMicrosec: !Number "{n.time}" + faultFields: + alarmInterfaceA: !StripExtraDash "{n.plugin}-{n.plugin_instance}" + alarmCondition: "{n.severity}" + faultFieldsVersion: 1.1 + - CONDITION: + plugin: virt + + +Helper YAML tags +---------------- + +This section describes the YAML tags used as utilities for formatting the output +message. The YAML configuration process workflow is described on the figure +below. + +.. figure:: parse-work-flow.png + + YAML configuration process workflow + + +Convert string to number tag +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``!Number`` tag is used in YAML configuration file to convert string value into +the number type. For instance: + +.. code:: yaml + + lastEpochMicrosec: !Number "3456" + +The output of the tag will be the JSON number. + +.. code:: json + + { + lastEpochMicrosec: 3456 + } + + +Convert bytes to Kibibytes tag +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``!Bytes2Kibibytes`` tag is used in YAML configuration file to convert +bytes into kibibytes (1 kibibyte = 1024 bytes). For instance: + +.. code:: yaml + + memoryConfigured: !Bytes2Kibibytes 4098 + memoryConfigured: !Bytes2Kibibytes "1024" + +The output of the tag will be the JSON number. + +.. code:: json + + { + memoryConfigured: 4 + memoryConfigured: 1 + } + + +Convert one value to another tag +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``!MapValue`` tag is used in YAML configuration file to map one value +into another value defined in the configuration. For instance: + +.. code:: yaml + + Severity: !MapValue + VALUE: Failure + TO: + Failure: Critical + Error: Warnign + +The output of the tag will be the mapped value. + +.. code:: json + + { + Severity: Critical + } + + +Strip extra dash tag +~~~~~~~~~~~~~~~~~~~~ + +The ``!StripExtraDash`` tag is used in YAML configuration file to strip extra +dashes in the string (dashes at the beginning, at the end and double dashes). +For example: + +.. code:: yaml + + name: !StripExtraDash string-with--extra-dashes- + +The output of the tag will be the JSON string with extra dashes removed. + +.. code:: json + + { + name: string-with-extra-dashes + } + + +Limitations +----------- + +#. Only one document can be defined in the same YAML configuration file. + +#. The collectd notification is not supported by `Kafka collectd plugin`_. Due to + this limitation, the collectd notifications cannot be received by the VES + application and the defined YAML event will not be generated and sent to the + VES collector. Please note, that VES YAML format already supports the events + definition and the format is descibed in the document. - <Plugin "threshold"> - <Plugin "virt"> - <Type "percent"> - WarningMax 2.08 - FailureMax 4.0 - Instance "virt_cpu_total" - </Type> - </Plugin> - </Plugin> -More detailed information on how to configure collectD thresholds can be found at -https://collectd.org/documentation/manpages/collectd-threshold.5.shtml +.. _collectd: http://collectd.org/ +.. _Kafka: https://kafka.apache.org/ +.. _`VES`: https://wiki.opnfv.org/display/fastpath/VES+plugin+updates +.. _`VES shema definition`: https://gerrit.onap.org/r/gitweb?p=demo.git;a=tree;f=vnfs/VES5.0/evel/evel-test-collector/docs/att_interface_definition;hb=refs/heads/master +.. _`PyYAML documentation`: https://pyyaml.org/wiki/PyYAMLDocumentation +.. _`collectd plugin description`: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod +.. _`collectd data types file`: https://github.com/collectd/collectd/blob/master/src/types.db +.. _`collectd data types description`: https://github.com/collectd/collectd/blob/master/src/types.db.pod +.. _`python regular expression syntax`: https://docs.python.org/2/library/re.html#regular-expression-syntax +.. _`Kafka collectd plugin`: https://collectd.org/wiki/index.php/Plugin:Write_Kafka |