summaryrefslogtreecommitdiffstats
path: root/docs/release/userguide/collectd.ves.userguide.rst
diff options
context:
space:
mode:
Diffstat (limited to 'docs/release/userguide/collectd.ves.userguide.rst')
-rw-r--r--docs/release/userguide/collectd.ves.userguide.rst834
1 files changed, 680 insertions, 154 deletions
diff --git a/docs/release/userguide/collectd.ves.userguide.rst b/docs/release/userguide/collectd.ves.userguide.rst
index 7077cecd..1f69b6ad 100644
--- a/docs/release/userguide/collectd.ves.userguide.rst
+++ b/docs/release/userguide/collectd.ves.userguide.rst
@@ -5,57 +5,19 @@
==========================
VES Application User Guide
==========================
-The Barometer repository contains a python based application for VES.
+
+The Barometer repository contains a python based application for VES (VNF Event
+Stream) which receives the `collectd`_ specific metrics via `Kafka`_ bus,
+normalizes the metric data into the VES message and sends it into the VES
+collector.
The application currently supports pushing platform relevant metrics through the
additional measurements field for VES.
-Collectd has a write_kafka plugin that will send collectd metrics and values to
-a Kafka Broker.
-The VES application uses Kafka Consumer to receive metrics from the Kafka
-Broker.
-
-Installation Instructions:
---------------------------
-1. Clone this repo:
-
- .. code:: bash
-
- git clone https://gerrit.opnfv.org/gerrit/barometer
-
-2. Install collectd
-
- .. code:: bash
-
- $ sudo apt-get install collectd
-
- CentOS 7.x use:
-
- .. code:: bash
-
- $ sudo yum install collectd
+Collectd has a ``write_kafka`` plugin that sends collectd metrics and values to
+a Kafka Broker. The VES application uses Kafka Consumer to receive metrics
+from the Kafka Broker.
- .. note:: You may need to add epel repository if the above does not work.
-
- .. code:: bash
-
- $ sudo yun install epel-release
-
-3. Modify the collectd configuration script: `/etc/collectd/collectd.conf`
-
- .. code:: bash
-
- <Plugin write_kafka>
- Property "metadata.broker.list" "localhost:9092"
- <Topic "collectd">
- Format JSON
- </Topic>
- </Plugin>
-
- .. note::
-
- The above configuration is for a single host setup. Simply change localhost to remote
- server IP addess or hostname.
Install Kafka Broker
--------------------
@@ -86,7 +48,9 @@ Install Kafka Broker
$ sudo yum install zookeeper
- .. note:: You may need to add the the repository that contains zookeeper
+ .. note:: You may need to add the repository that contains zookeeper.
+ To do so, follow the step below and try to install `zookeeper` again
+ using steps above. Otherwise, skip next step.
.. code:: bash
@@ -99,11 +63,20 @@ Install Kafka Broker
$ sudo zookeeper-server start
+ if you get the error message like ``ZooKeeper data directory is missing at /var/lib/zookeeper``
+ during the start of zookeeper, initialize zookeeper data directory using
+ the command below and start zookeeper again, otherwise skip the next step.
+
+ .. code:: bash
+
+ $ sudo /usr/lib/zookeeper/bin/zkServer-initialize.sh
+ No myid provided, be sure to specify it in /var/lib/zookeeper/myid if using non-standalone
+
To test if Zookeeper is running as a daemon.
.. code:: bash
- $ sudo telnet localhost 2181
+ $ telnet localhost 2181
Type 'ruok' & hit enter.
Expected response is 'imok'. Zookeeper is running fine.
@@ -115,25 +88,26 @@ Install Kafka Broker
.. code:: bash
+ $ sudo yum install python-pip
$ sudo pip install kafka-python
2. Download Kafka:
.. code:: bash
- $ sudo wget "http://www-eu.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz"
+ $ wget "http://www-eu.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz"
3. Extract the archive:
.. code:: bash
- $ sudo tar -xvzf kafka_2.11-0.11.0.0.tgz
+ $ tar -xvzf kafka_2.11-0.11.0.0.tgz
4. Configure Kafka Server:
.. code:: bash
- $ sudo vi kafka_2.11-0.11.0.0/config/server.properties
+ $ vi kafka_2.11-0.11.0.0/config/server.properties
By default Kafka does not allow you to delete topics. Please uncomment:
@@ -170,180 +144,732 @@ Install Kafka Broker
localhost:2181 --topic TopicTest --from-beginning
-VES application configuration description:
-------------------------------------------
+Install collectd
+----------------
+
+Install development tools:
-Within the VES directory there is a configuration file called 'ves_app.conf'.
+.. code:: bash
+
+ $ sudo yum group install 'Development Tools'
-.. note:: Details of the Vendor Event Listener REST service
+.. The libkafka installed via yum pkg manager is 0.11.0 which doesn't work with
+ collectd (compilation issue). Thus, we have to use the library installed
+ from sources using latest stable version which works with collectd.
-REST resources are defined with respect to a ServerRoot:
+Install Apache Kafka C/C++ client library:
.. code:: bash
- ServerRoot = https://{Domain}:{Port}/{optionalRoutingPath}
+ $ git clone https://github.com/edenhill/librdkafka.git ~/librdkafka
+ $ cd ~/librdkafka
+ $ git checkout -b v0.9.5 v0.9.5
+ $ ./configure --prefix=/usr
+ $ make
+ $ sudo make install
+
+Build collectd with Kafka support:
+
+.. code:: bash
+
+ $ git clone https://github.com/collectd/collectd.git ~/collectd
+ $ cd ~/collectd
+ $ ./build.sh
+ $ ./configure --with-librdkafka=/usr --without-perl-bindings --enable-perl=no
+ $ make && sudo make install
+
+Configure and start collectd. Create ``/opt/collectd/etc/collectd.conf``
+collectd configuration file as following:
+
+.. note:: The following collectd configuration file allows user to run VES
+ application in the guest mode. To run the VES in host mode, please follow
+ the `Configure VES in host mode`_ steps.
+
+.. include:: collectd-ves-guest.conf
+ :code: bash
+
+Start collectd process as a service as described in :ref:`install-collectd-as-a-service`.
+
+
+Setup VES Test Collector
+------------------------
+
+.. note:: Test Collector setup is required only for VES application testing
+ purposes.
+
+Install dependencies:
-REST resources are of the form:
+.. code:: bash
+
+ $ sudo pip install jsonschema
+
+Clone VES Test Collector:
+
+.. code:: bash
+
+ $ git clone https://github.com/att/evel-test-collector.git ~/evel-test-collector
+
+Modify VES Test Collector config file to point to existing log directory and
+schema file:
.. code:: bash
+ $ sed -i.back 's/^\(log_file[ ]*=[ ]*\).*/\1collector.log/' ~/evel-test-collector/config/collector.conf
+ $ sed -i.back 's/^\(schema_file[ ]*=.*\)event_format_updated.json$/\1CommonEventFormat.json/' ~/evel-test-collector/config/collector.conf
+
+Start VES Test Collector:
+
+.. code:: bash
+
+ $ cd ~/evel-test-collector/code/collector
+ $ nohup python ./collector.py --config ../../config/collector.conf > collector.stdout.log &
+
+
+Setup VES application (guest mode)
+----------------------------------
+
+Install dependencies:
+
+.. code:: bash
+
+ $ sudo pip install pyyaml
+
+Clone Barometer repo and start the VES application:
+
+.. code:: bash
+
+ $ git clone https://gerrit.opnfv.org/gerrit/barometer ~/barometer
+ $ cd ~/barometer/3rd_party/collectd-ves-app/ves_app
+ $ nohup python ves_app.py --events-schema=guest.yaml --config=ves_app_config.conf > ves_app.stdout.log &
+
+.. note::
+
+ The above configuration is used for a localhost. The VES application can be
+ configured to use remote real VES collector and remote Kafka server. To do
+ so, the IP addresses/host names needs to be changed in ``collector.conf``
+ and ``ves_app_config.conf`` files accordingly.
+
+
+Configure VES in host mode
+--------------------------
+
+Running the VES in host mode looks like steps described in
+`Setup VES application (guest mode)`_ but with the following exceptions:
+
+- The ``host.yaml`` configuration file should be used instead of ``guest.yaml``
+ file when VES application is running.
+
+- Collectd should be running on host machine only.
+
+- Addition ``libvirtd`` dependencies needs to be installed on a host where
+ collectd daemon is running. To install those dependencies, see :ref:`virt-plugin`
+ section of Barometer user guide.
+
+- At least one VM instance should be up and running by hypervisor on the host.
+
+- The next (minimum) configuration needs to be provided to collectd to be able
+ to generate the VES message to VES collector.
+
+ .. include:: collectd-ves-host.conf
+ :code: bash
+
+ to apply this configuration, the ``/opt/collectd/etc/collectd.conf`` file
+ needs to be modified based on example above and collectd daemon needs to
+ be restarted using the command below:
+
+ .. code:: bash
+
+ $ sudo systemctl restart collectd
+
+.. note:: The list of the plugins can be extented depends on your needs.
+
+
+VES application configuration description
+-----------------------------------------
+
+**Details of the Vendor Event Listener REST service**
+
+REST resources are defined with respect to a ``ServerRoot``::
+
+ ServerRoot = https://{Domain}:{Port}/{optionalRoutingPath}
+
+REST resources are of the form::
+
{ServerRoot}/eventListener/v{apiVersion}`
{ServerRoot}/eventListener/v{apiVersion}/{topicName}`
{ServerRoot}/eventListener/v{apiVersion}/eventBatch`
+Within the VES directory (``3rd_party/collectd-ves-app/ves_app``) there is a
+configuration file called ``ves_app.conf``. The description of the
+configuration options are described below:
+
**Domain** *"host"*
VES domain name. It can be IP address or hostname of VES collector
- (default: `127.0.0.1`)
+ (default: ``127.0.0.1``)
**Port** *port*
- VES port (default: `30000`)
+ VES port (default: ``30000``)
**Path** *"path"*
- Used as the "optionalRoutingPath" element in the REST path (default: `empty`)
+ Used as the "optionalRoutingPath" element in the REST path (default: empty)
**Topic** *"path"*
- Used as the "topicName" element in the REST path (default: `empty`)
+ Used as the "topicName" element in the REST path (default: empty)
**UseHttps** *true|false*
- Allow application to use HTTPS instead of HTTP (default: `false`)
+ Allow application to use HTTPS instead of HTTP (default: ``false``)
**Username** *"username"*
- VES collector user name (default: `empty`)
+ VES collector user name (default: empty)
**Password** *"passwd"*
- VES collector password (default: `empty`)
-
-**FunctionalRole** *"role"*
- Used as the 'functionalRole' field of 'commonEventHeader' event (default:
- `Collectd VES Agent`)
+ VES collector password (default: empty)
**SendEventInterval** *interval*
This configuration option controls how often (sec) collectd data is sent to
- Vendor Event Listener (default: `20`)
+ Vendor Event Listener (default: ``20``)
**ApiVersion** *version*
- Used as the "apiVersion" element in the REST path (default: `5.1`)
+ Used as the "apiVersion" element in the REST path (default: ``5.1``)
**KafkaPort** *port*
Kafka Port (Default ``9092``)
**KafkaBroker** *host*
Kafka Broker domain name. It can be an IP address or hostname of local or remote server
- (default: localhost)
+ (default: ``localhost``)
-Other collectd.conf configurations
-----------------------------------
-Please ensure that FQDNLookup is set to false
-.. code:: bash
+VES YAML configuration format
+-----------------------------
- FQDNLookup false
+The format of the VES message is generated by the VES application based on the
+YAML schema configuration file provided by user via ``--events-schema``
+command-line option of the application.
-Please ensure that the virt plugin is enabled and configured as follows.
+.. note:: Use ``--help`` option of VES application to see the
+ description of all available options
-.. code:: bash
+.. note:: The detailed installation guide of the VES application is described in
+ the `VES Application User Guide`_.
- LoadPlugin virt
+The message defined by YAML schema should correspond to format defined in
+`VES shema definition`_.
- <Plugin virt>
- Connection "qemu:///system"
- RefreshInterval 60
- HostnameFormat uuid
- PluginInstanceFormat name
- ExtraStats "cpu_util perf"
- </Plugin>
+.. warning:: This section doesn't explain the YAML language itself, so the
+ knowledge of the YAML language is required before continue reading it!
+Since, the VES message output is a JSON format, it's recommended to understand
+how YAML document is converted to JSON before starting to create a new YAML
+definition for the VES. E.g.:
-.. note:: For more detailed information on the `virt` plugin configuration,
- requirements etc., please see the userguide of the collectd virt plugin.
+The following YAML document:
-Please ensure that the cpu plugin is enabled and configured as follows
+.. code:: yaml
-.. code:: bash
+ ---
+ additionalMeasurements:
+ plugin: somename
+ instance: someinstance
- LoadPlugin cpu
+will be converted to JSON like this:
- <Plugin cpu>
- ReportByCpu false
- ValuesPercentage true
- </Plugin>
+.. code:: json
-.. note::
+ {
+ "additionalMeasurements": {
+ "instance": "someinstance",
+ "plugin": "somename"
+ }
+ }
- The ``ReportByCpu`` option should be set to `true` (default)
- if VES application is running on guest machine ('GuestRunning' = true).
+.. note:: The `YAML syntax` section of `PyYAML documentation`_ can be used
+ as a reference for this.
-Please ensure that the aggregation plugin is enabled and configured as follows
-(required if 'GuestRunning' = true)
-.. code:: bash
+VES message types
+-----------------
- LoadPlugin aggregation
+The VES agent can generate two type of messages which are sent to the VES
+collector. Each message type must be specified in the YAML configuration file
+using a specific YAML tag.
- <Plugin aggregation>
- <Aggregation>
- Plugin "cpu"
- Type "percent"
- GroupBy "Host"
- GroupBy "TypeInstance"
- SetPlugin "cpu-aggregation"
- CalculateAverage true
- </Aggregation>
- </Plugin>
+Measurements
+ This type is used to send a message defined in YAML configuration to the VES
+ collector with a specified interval (default is 20 sec and it's configurable
+ via command line option of the application). This type can be specified in
+ the configuration using ``!Measurements`` tag. For instance:
-If application is running on a guest side, it is important to enable uuid plugin
-too. In this case the hostname in event message will be represented as UUID
-instead of system host name.
+ .. code:: yaml
-.. code:: bash
+ ---
+ # My comments
+ My Host Measurements: !Measurements
+ ... # message definition
- LoadPlugin uuid
+Events
+ This type is used to send a message defined in YAML configuration to the VES
+ collector when collectd notification is received from Kafka bus (collectd
+ ``write_kafka`` plugin). This type can be specified in the configuration
+ using ``!Events`` tag. For instance:
-If a custom UUID needs to be provided, the following configuration is required in collectd.conf
-file:
+ .. code:: yaml
-.. code:: bash
+ ---
+ # My comments
+ My Events: !Events
+ ... # event definition
- <Plugin uuid>
- UUIDFile "/etc/uuid"
- </Plugin>
-Where "/etc/uuid" is a file containing custom UUID.
+Collectd metrics in VES
+-----------------------
-Please also ensure that the following plugins are enabled:
+The VES application caches collectd metrics received via Kafka bus. The data
+is stored in a table format. It's important to know it before mapping collectd
+metric values to message defined in YAML configuration file.
-.. code:: bash
+VES collectd metric cache example:
- LoadPlugin disk
- LoadPlugin interface
- LoadPlugin memory
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| host | plugin | plugin_instance | type | type_instace | time | value | ds_name | interval |
++===========+===========+====================+===========+================+===================+========+=========+==========+
+| localhost | cpu | | percent | user | 1509535512.30567 | 16 | value | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| localhost | memory | | memory | free | 1509535573.448014 | 798456 | value | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| localhost | interface | | eth0 | if_packets | 1509544183.956496 | 253 | rx | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| 7ec333e7 | virt | Ubuntu-12.04.5-LTS | percent | virt_cpu_total | 1509544402.378035 | 0.2 | value | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| 7ec333e7 | virt | Ubuntu-12.04.5-LTS | memory | rss | 1509544638.55119 | 123405 | value | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| 7ec333e7 | virt | Ubuntu-12.04.5-LTS | if_octets | vnet1 | 1509544646.27108 | 67 | tx | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| cc659a52 | virt | Ubuntu-16.04 | percent | virt_cpu_total | 1509544745.617207 | 0.3 | value | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| cc659a52 | virt | Ubuntu-16.04 | memory | rss | 1509544754.329411 | 4567 | value | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| cc659a52 | virt | Ubuntu-16.04 | if_octets | vnet0 | 1509544760.720381 | 0 | rx | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
-VES application with collectd notifications example
----------------------------------------------------
+It's possible from YAML configuration file to refer to any field of any row of
+the table via special YAML tags like ``ValueItem`` or ``ArrayItem``. See the
+`Collectd metric reference`_ reference for more details.
-A good example of collectD notification is monitoring of the total CPU usage on a VM
-using the 'threshold' plugin. The following configuration will setup VES plugin to send 'Fault'
-event every time a total VM CPU value is out of range (e.g.: WARNING: VM CPU TOTAL > 50%,
-CRITICAL: VM CPU TOTAL > 96%) and send 'Fault' NORMAL event if the CPU value is back
-to normal. In the example below, there is one VM with two CPUs configured which is running
-on the host with a total of 48 cores. Thus, the threshold value 2.08 (100/48) means that
-one CPU of the VM is fully loaded (e.g.: 50% of total CPU usage of the VM) and 4.0 means
-96% of total CPU usage of the VM. Those values can also be obtained by virt-top
-command line tool.
+.. note:: The `collectd data types file`_ contains map of ``type`` to
+ ``ds_name`` field. It can be used to get possible value for ``ds_name``
+ field. See the `collectd data types description`_ for more details on
+ collectd data types.
-.. code:: bash
- LoadPlugin threshold
+Aging of collectd metric
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+If the metric will not be updated by the collectd during the double metric
+interval time, it will be removed (aged) from VES internal cache.
+
+
+VES YAML references
+-------------------
+
+There are four type of references supported by the YAML format: System,
+Config, Collectd metric and Collectd notification. The format of the reference
+is the following:
+
+.. code:: yaml
+
+ "{<ref type>.<attribute name>}"
+
+.. note:: Possible values for ``<ref type>`` and ``<attribute name>`` are
+ described in farther sections.
+
+
+System reference
+~~~~~~~~~~~~~~~~
+
+This reference is used to get system statistics like time, date etc. The system
+reference (``<ref type>`` = ``system``) can be used in any place of the YAML
+configuration file. This type of reference provides the following attributes:
+
+``hostname``
+ The name of the host where VES application is running.
+
+``id``
+ Unique ID during VES application runtime.
+
+``time``
+ Current time in seconds since the Epoch. For example ``1509641411.631951``.
+
+``date``
+ Current date in ISO 8601 format, ``YYYY-MM-DD``. For instance ``2017-11-02``.
+
+
+For example:
+
+.. code:: yaml
+
+ Date: "{system.date}"
+
+
+Config reference
+~~~~~~~~~~~~~~~~
+
+This reference is used to get VES configuration described in `VES application
+configuration description`_ sectoin. The reference (``<ref type>`` = ``config``)
+can be used in any place of the YAML configuration file. This type of reference
+provides the following attributes:
+
+``interval``
+ Measurements dispatch interval. It referenses to ``SendEventInterval``
+ configuration of the VES application.
+
+
+For example:
+
+.. code:: yaml
+
+ Interval: "{config.interval}"
+
+
+Collectd metric reference
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This reference is used to get the attribute value of collectd metric from the
+VES cache. The reference (``<ref type>`` = ``vl``) can be used ONLY inside
+``Measurements``, ``ValueItem`` and ``ArrayItem`` tags. Using the reference
+inside a helper tag is also allowed if the helper tag is located inside the
+tag where the reference is allowed (e.g.: ``ArrayItem``). The
+``<attribute name>`` name corresponds to the table field name described in
+`Collectd metrics in VES`_ section. For example:
+
+.. code:: yaml
+
+ name: "{vl.type}-{vl.type_instance}"
+
+
+Collectd notification reference
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This reference is used to get the attribute value of received collectd
+notification. The reference (``<ref type>`` = ``n``) can be used ONLY inside
+``Events`` tag. Using the reference inside a helper tag is also allowed if
+the helper tag is located inside the ``Events`` tag. This type of reference
+provides the following attributes:
+
+``host``
+ The hostname of received collectd notification.
+
+``plugin``
+ The plugin name of received collectd notification.
+
+``plugin_instance``
+ The plugin instance of the received collectd notification.
+
+``type``
+ The type name of the received collectd notification.
+
+``type_instance``
+ The type instance of the received collectd notification.
+
+``severity``
+ The severity of the received collectd notification.
+
+``message``
+ The message of the received collectd notification.
+
+
+.. note:: The exact value for each attribute depends on the collectd plugin
+ which may generate the notification. Please refer to the
+ `collectd plugin description`_ document to get more details on the specific
+ collectd plugin.
+
+YAML config example:
+
+.. code:: yaml
+
+ sourceId: "{n.plugin_instance}"
+
+
+Collectd metric mapping YAML tags
+---------------------------------
+
+This section describes the YAML tags used to map collectd metric values in the
+YAML configuration file.
+
+
+Measurements tag
+~~~~~~~~~~~~~~~~
+
+This tag is a YAML map which is used to define the VES measurement message. It's
+allowed to be used multiple times in the document (e.g.: you can define multiple
+VES messages in one YAML document). This tag works in the same way as `ArrayItem
+tag`_ does and all keys have the same description/rules.
+
+
+ValueItem tag
+~~~~~~~~~~~~~
+
+This tag is used to select a collectd metric and get its attribute value using
+`Collectd metric reference`_. The type of this tag is a YAML array of maps with
+the possible keys described below.
+
+``SELECT`` (required)
+ Is a YAML map which describes the select metric criteria. Each key name of the
+ map must correspond to the table field name described in `Collectd metrics in VES`_
+ section.
+
+``VALUE`` (optional)
+ Describes the value to be assigned. If not provided, the default
+ ``!Number "{vl.value}"`` expression is used.
+
+``DEFAULT`` (optional)
+ Describes the default value which will be assigned if no metric is selected
+ by ``SELECT`` criteria.
+
+
+ValueItem tag description example:
+
+.. code:: yaml
+
+ memoryFree: !ValueItem
+ - SELECT:
+ plugin: memory
+ type: memory
+ type_instance: rss
+ - VALUE: !Bytes2Kibibytes "{vl.value}"
+ - DEFAULT: 0
+
+The tag process workflow is described on the figure below.
+
+.. figure:: value-item-parse-workflow.png
+
+ YAML ValueItem tag process workflow
+
+
+ArrayItem tag
+~~~~~~~~~~~~~
+
+This tag is used to select a list of collectd metrics and generate a YAML array
+of YAML items described by ``ITEM-DESC`` key. If no collectd metrics are
+selected by the given criteria, the empty array will be returned.
+
+``SELECT`` (optional)
+ Is a YAML map which describes the select metrics criteria. Each key name of
+ the map must correspond to the table field name described in `Collectd
+ metrics in VES`_ section. The value of the key may be regular expression. To
+ enable regular expression in the value, the YAML string containing ``/`` char
+ at the beginning and at the end should be used. For example:
+
+ .. code:: yaml
+
+ plugin: "/^(?!virt).*$/" # selected all metrics except ``virt`` plugin
+
+ The VES application uses the python RE library to work with regular
+ expression specified in the YAML configuration. Please refer to `python
+ regular expression syntax`_ documentation for more details on a syntax
+ used by the VES.
+
+ Multiple ``SELECT`` keys are allowed by the tag. If nor ``SELECT`` or
+ ``INDEX-KEY`` key is specified, the VES error is generated.
+
+``INDEX-KEY`` (optional)
+ Is a YAML array which describes the unique fields to be selected by the tag.
+ Each element of array is a YAML string which should be one of the table field
+ name described in `Collectd metrics in VES`_ section. Please note, if this
+ key is used, only fields specified by the key can be used as a collectd
+ reference in the ``ITEM-DESC`` key.
+
+``ITEM-DESC`` (required)
+ Is a YAML map which describes each element of the YAML array generated by
+ the tag. Using ``ArrayItem`` tags and other `Helper YAML tags`_ are also
+ allowed in the definition of the key.
+
+In the example below, the ArrayItem tag is used to generate an array of
+``ITEM-DESC`` items for each collectd metrics except ``virt`` plugin with
+unique ``plugin``, ``plugin_instance`` attribute values.
+
+.. code:: yaml
+
+ Measurements: !ArrayItem
+ - SELECT:
+ plugin: "/^(?!virt).*$/"
+ - INDEX-KEY:
+ - plugin
+ - plugin_instance
+ - ITEM-DESC:
+ name: !StripExtraDash "{vl.plugin}-{vl.plugin_instance}"
+
+The tag process workflow is described on the figure below.
+
+.. figure:: array-item-parse-workflow.png
+
+ YAML ArrayItem tag process workflow
+
+
+Collectd event mapping YAML tags
+--------------------------------
+
+This section describes the YAML tags used to map the collectd notification to
+the VES event message in the YAML configuration file.
+
+
+Events tag
+~~~~~~~~~~
+
+This tag is a YAML map which is used to define the VES event message. It's
+allowed to be used multiple times in the document (e.g.: you can map multiple
+collectd notification into VES message in one YAML document). The possible
+keys of the tag are described below.
+
+``CONDITION`` (optional)
+ Is a YAML map which describes the select metric criteria. Each key name of
+ the map must correspond to the name of attribute provided by `Collectd
+ notification reference`_. If no such key provided, any collectd notification
+ will map the defined YAML message.
+
+``ITEM-DESC`` (required)
+ Is a YAML map which describes the message generated by this tag. Only `Helper
+ YAML tags`_ are allowed in the definition of the key.
+
+The example of the VES event message which will be generate by the VES
+application when collectd notification of the ``virt`` plugin is triggered
+is described below.
+
+.. code:: yaml
+
+ ---
+ Virt Event: !Events
+ - ITEM-DESC:
+ event:
+ commonEventHeader:
+ domain: fault
+ eventType: Notification
+ sourceId: &event_sourceId "{n.plugin_instance}"
+ sourceName: *event_sourceId
+ lastEpochMicrosec: !Number "{n.time}"
+ startEpochMicrosec: !Number "{n.time}"
+ faultFields:
+ alarmInterfaceA: !StripExtraDash "{n.plugin}-{n.plugin_instance}"
+ alarmCondition: "{n.severity}"
+ faultFieldsVersion: 1.1
+ - CONDITION:
+ plugin: virt
+
+
+Helper YAML tags
+----------------
+
+This section describes the YAML tags used as utilities for formatting the output
+message. The YAML configuration process workflow is described on the figure
+below.
+
+.. figure:: parse-work-flow.png
+
+ YAML configuration process workflow
+
+
+Convert string to number tag
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``!Number`` tag is used in YAML configuration file to convert string value into
+the number type. For instance:
+
+.. code:: yaml
+
+ lastEpochMicrosec: !Number "3456"
+
+The output of the tag will be the JSON number.
+
+.. code:: json
+
+ {
+ lastEpochMicrosec: 3456
+ }
+
+
+Convert bytes to Kibibytes tag
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``!Bytes2Kibibytes`` tag is used in YAML configuration file to convert
+bytes into kibibytes (1 kibibyte = 1024 bytes). For instance:
+
+.. code:: yaml
+
+ memoryConfigured: !Bytes2Kibibytes 4098
+ memoryConfigured: !Bytes2Kibibytes "1024"
+
+The output of the tag will be the JSON number.
+
+.. code:: json
+
+ {
+ memoryConfigured: 4
+ memoryConfigured: 1
+ }
+
+
+Convert one value to another tag
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``!MapValue`` tag is used in YAML configuration file to map one value
+into another value defined in the configuration. For instance:
+
+.. code:: yaml
+
+ Severity: !MapValue
+ VALUE: Failure
+ TO:
+ Failure: Critical
+ Error: Warnign
+
+The output of the tag will be the mapped value.
+
+.. code:: json
+
+ {
+ Severity: Critical
+ }
+
+
+Strip extra dash tag
+~~~~~~~~~~~~~~~~~~~~
+
+The ``!StripExtraDash`` tag is used in YAML configuration file to strip extra
+dashes in the string (dashes at the beginning, at the end and double dashes).
+For example:
+
+.. code:: yaml
+
+ name: !StripExtraDash string-with--extra-dashes-
+
+The output of the tag will be the JSON string with extra dashes removed.
+
+.. code:: json
+
+ {
+ name: string-with-extra-dashes
+ }
+
+
+Limitations
+-----------
+
+#. Only one document can be defined in the same YAML configuration file.
+
+#. The collectd notification is not supported by `Kafka collectd plugin`_. Due to
+ this limitation, the collectd notifications cannot be received by the VES
+ application and the defined YAML event will not be generated and sent to the
+ VES collector. Please note, that VES YAML format already supports the events
+ definition and the format is descibed in the document.
- <Plugin "threshold">
- <Plugin "virt">
- <Type "percent">
- WarningMax 2.08
- FailureMax 4.0
- Instance "virt_cpu_total"
- </Type>
- </Plugin>
- </Plugin>
-More detailed information on how to configure collectD thresholds can be found at
-https://collectd.org/documentation/manpages/collectd-threshold.5.shtml
+.. _collectd: http://collectd.org/
+.. _Kafka: https://kafka.apache.org/
+.. _`VES`: https://wiki.opnfv.org/display/fastpath/VES+plugin+updates
+.. _`VES shema definition`: https://gerrit.onap.org/r/gitweb?p=demo.git;a=tree;f=vnfs/VES5.0/evel/evel-test-collector/docs/att_interface_definition;hb=refs/heads/master
+.. _`PyYAML documentation`: https://pyyaml.org/wiki/PyYAMLDocumentation
+.. _`collectd plugin description`: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod
+.. _`collectd data types file`: https://github.com/collectd/collectd/blob/master/src/types.db
+.. _`collectd data types description`: https://github.com/collectd/collectd/blob/master/src/types.db.pod
+.. _`python regular expression syntax`: https://docs.python.org/2/library/re.html#regular-expression-syntax
+.. _`Kafka collectd plugin`: https://collectd.org/wiki/index.php/Plugin:Write_Kafka