ves: Add YAML configuration format description

- Clean-up

Change-Id: I2dc3d5924683832ffa0085ea2a2db9ed14f42f5a
Signed-off-by: Mytnyk, Volodymyr <volodymyrx.mytnyk@intel.com>

4 files changed, 538 insertions, 11 deletions

diff --git a/docs/release/userguide/array-item-parse-workflow.png b/docs/release/userguide/array-item-parse-workflow.png
new file mode 100644
index 00000000..22cbefff
--- /dev/null
+++ b/docs/release/userguide/array-item-parse-workflow.png
diff --git a/docs/release/userguide/collectd.ves.userguide.rst b/docs/release/userguide/collectd.ves.userguide.rst
index e7fcf738..1f69b6ad 100644
--- a/docs/release/userguide/collectd.ves.userguide.rst
+++ b/docs/release/userguide/collectd.ves.userguide.rst
@@ -5,7 +5,11 @@
 ==========================
 VES Application User Guide
 ==========================
-The Barometer repository contains a python based application for VES.
+
+The Barometer repository contains a python based application for VES (VNF Event
+Stream) which receives the `collectd`_ specific metrics via `Kafka`_ bus,
+normalizes the metric data into the VES message and sends it into the VES
+collector.
 
 The application currently supports pushing platform relevant metrics through the
 additional measurements field for VES.
@@ -186,9 +190,6 @@ collectd configuration file as following:
 
 Start collectd process as a service as described in :ref:`install-collectd-as-a-service`.
 
-..  Start collectd process as a service as described in `Barometer User Guide
-    <http://artifacts.opnfv.org/barometer/docs/index.html#installing-collectd-as-a-service>`_.
-
 
 Setup VES Test Collector
 ------------------------
@@ -233,7 +234,7 @@ Install dependencies:
 
     $ sudo pip install pyyaml
 
-Clone Barometer repo:
+Clone Barometer repo and start the VES application:
 
 .. code:: bash
 
@@ -339,10 +340,536 @@ configuration options are described below:
   (default: ``localhost``)
 
 
-VES notification support
-------------------------
+VES YAML configuration format
+-----------------------------
+
+The format of the VES message is generated by the VES application based on the
+YAML schema configuration file provided by user via ``--events-schema``
+command-line option of the application.
+
+.. note:: Use ``--help`` option of VES application to see the
+    description of all available options
+
+.. note:: The detailed installation guide of the VES application is described in
+    the `VES Application User Guide`_.
+
+The message defined by YAML schema should correspond to format defined in
+`VES shema definition`_.
+
+.. warning:: This section doesn't explain the YAML language itself, so the
+    knowledge of the YAML language is required before continue reading it!
+
+Since, the VES message output is a JSON format, it's recommended to understand
+how YAML document is converted to JSON before starting to create a new YAML
+definition for the VES. E.g.:
+
+The following YAML document:
+
+.. code:: yaml
+
+    ---
+    additionalMeasurements:
+      plugin: somename
+      instance: someinstance
+
+will be converted to JSON like this:
+
+.. code:: json
+
+    {
+      "additionalMeasurements": {
+        "instance": "someinstance",
+        "plugin": "somename"
+      }
+    }
+
+.. note:: The `YAML syntax` section of `PyYAML documentation`_ can be used
+    as a reference for this.
+
+
+VES message types
+-----------------
+
+The VES agent can generate two type of messages which are sent to the VES
+collector. Each message type must be specified in the YAML configuration file
+using a specific YAML tag.
+
+Measurements
+  This type is used to send a message defined in YAML configuration to the VES
+  collector with a specified interval (default is 20 sec and it's configurable
+  via command line option of the application). This type can be specified in
+  the configuration using ``!Measurements`` tag. For instance:
+
+  .. code:: yaml
+
+      ---
+      # My comments
+      My Host Measurements: !Measurements
+        ... # message definition
+
+Events
+  This type is used to send a message defined in YAML configuration to the VES
+  collector when collectd notification is received from Kafka bus (collectd
+  ``write_kafka`` plugin). This type can be specified in the configuration
+  using ``!Events`` tag. For instance:
+
+  .. code:: yaml
+
+      ---
+      # My comments
+      My Events: !Events
+        ... # event definition
+
+
+Collectd metrics in VES
+-----------------------
+
+The VES application caches collectd metrics received via Kafka bus. The data
+is stored in a table format. It's important to know it before mapping collectd
+metric values to message defined in YAML configuration file.
+
+VES collectd metric cache example:
+
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+|   host    |  plugin   |  plugin_instance   |  type     |  type_instace  |       time        | value  | ds_name | interval |
++===========+===========+====================+===========+================+===================+========+=========+==========+
+| localhost | cpu       |                    | percent   | user           | 1509535512.30567  | 16     | value   | 10       |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| localhost | memory    |                    | memory    | free           | 1509535573.448014 | 798456 | value   | 10       |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| localhost | interface |                    | eth0      | if_packets     | 1509544183.956496 | 253    | rx      | 10       |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| 7ec333e7  | virt      | Ubuntu-12.04.5-LTS | percent   | virt_cpu_total | 1509544402.378035 | 0.2    | value   | 10       |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| 7ec333e7  | virt      | Ubuntu-12.04.5-LTS | memory    | rss            | 1509544638.55119  | 123405 | value   | 10       |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| 7ec333e7  | virt      | Ubuntu-12.04.5-LTS | if_octets | vnet1          | 1509544646.27108  | 67     | tx      | 10       |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| cc659a52  | virt      | Ubuntu-16.04       | percent   | virt_cpu_total | 1509544745.617207 | 0.3    | value   | 10       |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| cc659a52  | virt      | Ubuntu-16.04       | memory    | rss            | 1509544754.329411 | 4567   | value   | 10       |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| cc659a52  | virt      | Ubuntu-16.04       | if_octets | vnet0          | 1509544760.720381 | 0      | rx      | 10       |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+
+It's possible from YAML configuration file to refer to any field of any row of
+the table via special YAML tags like ``ValueItem`` or ``ArrayItem``. See the
+`Collectd metric reference`_ reference for more details.
+
+.. note:: The `collectd data types file`_ contains map of ``type`` to
+    ``ds_name`` field. It can be used to get possible value for ``ds_name``
+    field. See the `collectd data types description`_ for more details on
+    collectd data types.
+
+
+Aging of collectd metric
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+If the metric will not be updated by the collectd during the double metric
+interval time, it will be removed (aged) from VES internal cache.
+
+
+VES YAML references
+-------------------
+
+There are four type of references supported by the YAML format: System,
+Config, Collectd metric and Collectd notification. The format of the reference
+is the following:
+
+.. code:: yaml
+
+    "{<ref type>.<attribute name>}"
+
+.. note:: Possible values for ``<ref type>`` and ``<attribute name>`` are
+    described in farther sections.
+
+
+System reference
+~~~~~~~~~~~~~~~~
+
+This reference is used to get system statistics like time, date etc. The system
+reference (``<ref type>`` = ``system``) can be used in any place of the YAML
+configuration file. This type of reference provides the following attributes:
+
+``hostname``
+  The name of the host where VES application is running.
+
+``id``
+  Unique ID during VES application runtime.
+
+``time``
+  Current time in seconds since the Epoch. For example ``1509641411.631951``.
+
+``date``
+  Current date in ISO 8601 format, ``YYYY-MM-DD``. For instance ``2017-11-02``.
+
+
+For example:
+
+.. code:: yaml
+
+    Date: "{system.date}"
+
+
+Config reference
+~~~~~~~~~~~~~~~~
+
+This reference is used to get VES configuration described in `VES application
+configuration description`_ sectoin. The reference (``<ref type>`` = ``config``)
+can be used in any place of the YAML configuration file. This type of reference
+provides the following attributes:
+
+``interval``
+  Measurements dispatch interval. It referenses to ``SendEventInterval``
+  configuration of the VES application.
+
+
+For example:
+
+.. code:: yaml
+
+    Interval: "{config.interval}"
+
+
+Collectd metric reference
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This reference is used to get the attribute value of collectd metric from the
+VES cache. The reference (``<ref type>`` = ``vl``) can be used ONLY inside
+``Measurements``, ``ValueItem`` and ``ArrayItem`` tags. Using the reference
+inside a helper tag is also allowed if the helper tag is located inside the
+tag where the reference is allowed (e.g.: ``ArrayItem``). The
+``<attribute name>`` name corresponds to the table field name described in
+`Collectd metrics in VES`_ section. For example:
+
+.. code:: yaml
+
+    name: "{vl.type}-{vl.type_instance}"
+
+
+Collectd notification reference
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This reference is used to get the attribute value of received collectd
+notification. The reference (``<ref type>`` = ``n``) can be used ONLY inside
+``Events`` tag. Using the reference inside a helper tag is also allowed if
+the helper tag is located inside the ``Events`` tag. This type of reference
+provides the following attributes:
+
+``host``
+  The hostname of received collectd notification.
+
+``plugin``
+  The plugin name of received collectd notification.
+
+``plugin_instance``
+  The plugin instance of the received collectd notification.
+
+``type``
+  The type name of the received collectd notification.
+
+``type_instance``
+  The type instance of the received collectd notification.
+
+``severity``
+  The severity of the received collectd notification.
+
+``message``
+  The message of the received collectd notification.
+
+
+.. note:: The exact value for each attribute depends on the collectd plugin
+  which may generate the notification. Please refer to the
+  `collectd plugin description`_ document to get more details on the specific
+  collectd plugin.
+
+YAML config example:
+
+.. code:: yaml
+
+    sourceId: "{n.plugin_instance}"
+
+
+Collectd metric mapping YAML tags
+---------------------------------
+
+This section describes the YAML tags used to map collectd metric values in the
+YAML configuration file.
+
+
+Measurements tag
+~~~~~~~~~~~~~~~~
+
+This tag is a YAML map which is used to define the VES measurement message. It's
+allowed to be used multiple times in the document (e.g.: you can define multiple
+VES messages in one YAML document). This tag works in the same way as `ArrayItem
+tag`_ does and all keys have the same description/rules.
+
+
+ValueItem tag
+~~~~~~~~~~~~~
+
+This tag is used to select a collectd metric and get its attribute value using
+`Collectd metric reference`_. The type of this tag is a YAML array of maps with
+the possible keys described below.
+
+``SELECT`` (required)
+  Is a YAML map which describes the select metric criteria. Each key name of the
+  map must correspond to the table field name described in `Collectd metrics in VES`_
+  section.
+
+``VALUE`` (optional)
+  Describes the value to be assigned. If not provided, the default
+  ``!Number "{vl.value}"`` expression is used.
+
+``DEFAULT`` (optional)
+  Describes the default value which will be assigned if no metric is selected
+  by ``SELECT`` criteria.
+
+
+ValueItem tag description example:
+
+.. code:: yaml
+
+    memoryFree: !ValueItem
+      - SELECT:
+          plugin: memory
+          type: memory
+          type_instance: rss
+      - VALUE: !Bytes2Kibibytes "{vl.value}"
+      - DEFAULT: 0
+
+The tag process workflow is described on the figure below.
+
+.. figure:: value-item-parse-workflow.png
+
+    YAML ValueItem tag process workflow
+
+
+ArrayItem tag
+~~~~~~~~~~~~~
+
+This tag is used to select a list of collectd metrics and generate a YAML array
+of YAML items described by ``ITEM-DESC`` key. If no collectd metrics are
+selected by the given criteria, the empty array will be returned.
+
+``SELECT`` (optional)
+  Is a YAML map which describes the select metrics criteria. Each key name of
+  the map must correspond to the table field name described in `Collectd
+  metrics in VES`_ section. The value of the key may be regular expression. To
+  enable regular expression in the value, the YAML string containing ``/`` char
+  at the beginning and at the end should be used. For example:
+
+  .. code:: yaml
+
+      plugin: "/^(?!virt).*$/" # selected all metrics except ``virt`` plugin
+
+  The VES application uses the python RE library to work with regular
+  expression specified in the YAML configuration. Please refer to `python
+  regular expression syntax`_ documentation for more details on a syntax
+  used by the VES.
+
+  Multiple ``SELECT`` keys are allowed by the tag. If nor ``SELECT`` or
+  ``INDEX-KEY`` key is specified, the VES error is generated.
+
+``INDEX-KEY`` (optional)
+  Is a YAML array which describes the unique fields to be selected by the tag.
+  Each element of array is a YAML string which should be one of the table field
+  name described in `Collectd metrics in VES`_ section. Please note, if this
+  key is used, only fields specified by the key can be used as a collectd
+  reference in the ``ITEM-DESC`` key.
+
+``ITEM-DESC`` (required)
+  Is a YAML map which describes each element of the YAML array generated by
+  the tag. Using ``ArrayItem`` tags and other `Helper YAML tags`_ are also
+  allowed in the definition of the key.
+
+In the example below, the ArrayItem tag is used to generate an array of
+``ITEM-DESC`` items for each collectd metrics except ``virt`` plugin with
+unique ``plugin``, ``plugin_instance`` attribute values.
+
+.. code:: yaml
+
+    Measurements: !ArrayItem
+      - SELECT:
+          plugin: "/^(?!virt).*$/"
+      - INDEX-KEY:
+          - plugin
+          - plugin_instance
+      - ITEM-DESC:
+          name: !StripExtraDash "{vl.plugin}-{vl.plugin_instance}"
+
+The tag process workflow is described on the figure below.
+
+.. figure:: array-item-parse-workflow.png
+
+    YAML ArrayItem tag process workflow
+
+
+Collectd event mapping YAML tags
+--------------------------------
+
+This section describes the YAML tags used to map the collectd notification to
+the VES event message in the YAML configuration file.
+
+
+Events tag
+~~~~~~~~~~
+
+This tag is a YAML map which is used to define the VES event message. It's
+allowed to be used multiple times in the document (e.g.: you can map multiple
+collectd notification into VES message in one YAML document). The possible
+keys of the tag are described below.
+
+``CONDITION`` (optional)
+  Is a YAML map which describes the select metric criteria. Each key name of
+  the map must correspond to the name of attribute provided by `Collectd
+  notification reference`_. If no such key provided, any collectd notification
+  will map the defined YAML message.
+
+``ITEM-DESC`` (required)
+  Is a YAML map which describes the message generated by this tag. Only `Helper
+  YAML tags`_ are allowed in the definition of the key.
+
+The example of the VES event message which will be generate by the VES
+application when collectd notification of the ``virt`` plugin is triggered
+is described below.
+
+.. code:: yaml
+
+    ---
+    Virt Event: !Events
+      - ITEM-DESC:
+          event:
+            commonEventHeader:
+              domain: fault
+              eventType: Notification
+              sourceId: &event_sourceId "{n.plugin_instance}"
+              sourceName: *event_sourceId
+              lastEpochMicrosec: !Number "{n.time}"
+              startEpochMicrosec: !Number "{n.time}"
+            faultFields:
+              alarmInterfaceA: !StripExtraDash "{n.plugin}-{n.plugin_instance}"
+              alarmCondition: "{n.severity}"
+              faultFieldsVersion: 1.1
+      - CONDITION:
+          plugin: virt
+
+
+Helper YAML tags
+----------------
+
+This section describes the YAML tags used as utilities for formatting the output
+message. The YAML configuration process workflow is described on the figure
+below.
+
+.. figure:: parse-work-flow.png
+
+    YAML configuration process workflow
+
+
+Convert string to number tag
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``!Number`` tag is used in YAML configuration file to convert string value into
+the number type. For instance:
+
+.. code:: yaml
+
+    lastEpochMicrosec: !Number "3456"
+
+The output of the tag will be the JSON number.
+
+.. code:: json
+
+    {
+      lastEpochMicrosec: 3456
+    }
+
+
+Convert bytes to Kibibytes tag
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``!Bytes2Kibibytes`` tag is used in YAML configuration file to convert
+bytes into kibibytes (1 kibibyte = 1024 bytes). For instance:
+
+.. code:: yaml
+
+    memoryConfigured: !Bytes2Kibibytes 4098
+    memoryConfigured: !Bytes2Kibibytes "1024"
+
+The output of the tag will be the JSON number.
+
+.. code:: json
+
+    {
+      memoryConfigured: 4
+      memoryConfigured: 1
+    }
+
+
+Convert one value to another tag
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``!MapValue`` tag is used in YAML configuration file to map one value
+into another value defined in the configuration. For instance:
+
+.. code:: yaml
+
+    Severity: !MapValue
+      VALUE: Failure
+      TO:
+        Failure: Critical
+        Error: Warnign
+
+The output of the tag will be the mapped value.
+
+.. code:: json
+
+    {
+      Severity: Critical
+    }
+
+
+Strip extra dash tag
+~~~~~~~~~~~~~~~~~~~~
+
+The ``!StripExtraDash`` tag is used in YAML configuration file to strip extra
+dashes in the string (dashes at the beginning, at the end and double dashes).
+For example:
+
+.. code:: yaml
+
+    name: !StripExtraDash string-with--extra-dashes-
+
+The output of the tag will be the JSON string with extra dashes removed.
+
+.. code:: json
+
+    {
+      name: string-with-extra-dashes
+    }
+
+
+Limitations
+-----------
+
+#. Only one document can be defined in the same YAML configuration file.
+
+#. The collectd notification is not supported by `Kafka collectd plugin`_. Due to
+   this limitation, the collectd notifications cannot be received by the VES
+   application and the defined YAML event will not be generated and sent to the
+   VES collector. Please note, that VES YAML format already supports the events
+   definition and the format is descibed in the document.
+
 
-The VES application already supports YAML notification definitions but due to
-the collectd Kafka plugin limitations, collectd notifications cannot be received
-by the VES application. Thus, the VES notification (defined by YAML) will not be
-generated and sent to VES collector.
+.. _collectd: http://collectd.org/
+.. _Kafka: https://kafka.apache.org/
+.. _`VES`: https://wiki.opnfv.org/display/fastpath/VES+plugin+updates
+.. _`VES shema definition`: https://gerrit.onap.org/r/gitweb?p=demo.git;a=tree;f=vnfs/VES5.0/evel/evel-test-collector/docs/att_interface_definition;hb=refs/heads/master
+.. _`PyYAML documentation`: https://pyyaml.org/wiki/PyYAMLDocumentation
+.. _`collectd plugin description`: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod
+.. _`collectd data types file`: https://github.com/collectd/collectd/blob/master/src/types.db
+.. _`collectd data types description`: https://github.com/collectd/collectd/blob/master/src/types.db.pod
+.. _`python regular expression syntax`: https://docs.python.org/2/library/re.html#regular-expression-syntax
+.. _`Kafka collectd plugin`: https://collectd.org/wiki/index.php/Plugin:Write_Kafka
diff --git a/docs/release/userguide/parse-work-flow.png b/docs/release/userguide/parse-work-flow.png
new file mode 100644
index 00000000..59cb70a8
--- /dev/null
+++ b/docs/release/userguide/parse-work-flow.png
diff --git a/docs/release/userguide/value-item-parse-workflow.png b/docs/release/userguide/value-item-parse-workflow.png
new file mode 100644
index 00000000..0aff3191
--- /dev/null
+++ b/docs/release/userguide/value-item-parse-workflow.png