VES Application: Documentation changes to userguide.

- Information on how kafka works with VES app.
- Added how-to guide to setup kakfa & dependencies.

Change-Id: Iaa973201b7178dbb902dd9efc9ccf91b5a423658
Signed-off-by: Alan Carey <alan.carey@intel.com>

1 files changed, 194 insertions, 33 deletions

diff --git a/docs/release/userguide/collectd.ves.userguide.rst b/docs/release/userguide/collectd.ves.userguide.rst
index fdf90187..7077cecd 100644
--- a/docs/release/userguide/collectd.ves.userguide.rst
+++ b/docs/release/userguide/collectd.ves.userguide.rst
@@ -2,51 +2,180 @@
 .. http://creativecommons.org/licenses/by/4.0
 .. (c) OPNFV, Intel Corporation and others.
 
-==============================
-collectd VES plugin User Guide
-==============================
-The Barometer repository contains a python based write plugin for VES.
+==========================
+VES Application User Guide
+==========================
+The Barometer repository contains a python based application for VES.
 
-The plugin currently supports pushing platform relevant metrics through the
+The application currently supports pushing platform relevant metrics through the
 additional measurements field for VES.
 
-**Please note**: Hardcoded configuration values will be modified so that they
-are configurable through the configuration file.
+Collectd has a write_kafka plugin that will send collectd metrics and values to
+a Kafka Broker.
+The VES application uses Kafka Consumer to receive metrics from the Kafka
+Broker.
 
 Installation Instructions:
 --------------------------
-1. Clone this repo
+1. Clone this repo:
+
+    .. code:: bash
+
+        git clone https://gerrit.opnfv.org/gerrit/barometer
+
 2. Install collectd
 
-.. code:: bash
+    .. code:: bash
+
+        $ sudo apt-get install collectd
+
+    CentOS 7.x use:
+
+    .. code:: bash
+
+        $ sudo yum install collectd
+
+    .. note:: You may need to add epel repository if the above does not work.
+
+    .. code:: bash
 
-   $ sudo apt-get install collectd
+        $ sudo yun install epel-release
 
 3. Modify the collectd configuration script: `/etc/collectd/collectd.conf`
 
-.. code:: bash
+    .. code:: bash
 
-    <LoadPlugin python>
-      Globals true
-    </LoadPlugin>
-
-    <Plugin python>
-      ModulePath "/path/to/your/python/modules"
-      LogTraces true
-      Interactive false
-      Import "ves_plugin"
-    <Module ves_plugin>
-    # VES plugin configuration (see next section below)
-    </Module>
-    </Plugin>
+        <Plugin write_kafka>
+            Property "metadata.broker.list" "localhost:9092"
+            <Topic "collectd">
+                Format JSON
+            </Topic>
+        </Plugin>
+
+    .. note::
+
+        The above configuration is for a single host setup. Simply change localhost to remote
+        server IP addess or hostname.
+
+Install Kafka Broker
+--------------------
+
+1. Dependencies: install JAVA & Zookeeper.
+
+    JAVA:
+
+    .. code:: bash
+
+        $ sudo apt install default-jre
+
+    CentOS 7.x use:
+
+    .. code:: bash
+
+        $ sudo yum install java-1.6.0-openjdk
+
+    Zookeeper:
+
+    .. code:: bash
+
+        $ sudo apt install zookeeperd
+
+    CentOS 7.x use:
+
+    .. code:: bash
+
+        $ sudo yum install zookeeper
+
+    .. note:: You may need to add the the repository that contains zookeeper
+
+    .. code:: bash
+
+        $ sudo yum install
+        https://archive.cloudera.com/cdh5/one-click-install/redhat/7/x86_64/cloudera-cdh-5-0.x86_64.rpm
+
+    CentOS 7.x start zookeeper:
+
+    .. code:: bash
+
+        $ sudo zookeeper-server start
+
+    To test if Zookeeper is running as a daemon.
+
+    .. code:: bash
+
+        $ sudo telnet localhost 2181
+
+    Type 'ruok' & hit enter.
+    Expected response is 'imok'. Zookeeper is running fine.
 
-where "/path/to/your/python/modules" is the path to ves_plugin.py,
-which is located in this repo.
+    .. note::
 
-VES python plugin configuration description:
---------------------------------------------
+        VES doesn't work with version 0.9.4 of kafka-python.
+        The recommended/tested version is 1.3.3.
 
-**Note** Details of the Vendor Event Listener REST service
+    .. code:: bash
+
+        $ sudo pip install kafka-python
+
+2. Download Kafka:
+
+    .. code:: bash
+
+        $ sudo wget "http://www-eu.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz"
+
+3. Extract the archive:
+
+    .. code:: bash
+
+        $ sudo tar -xvzf kafka_2.11-0.11.0.0.tgz
+
+4. Configure Kafka Server:
+
+    .. code:: bash
+
+        $ sudo vi kafka_2.11-0.11.0.0/config/server.properties
+
+    By default Kafka does not allow you to delete topics. Please uncomment:
+
+    .. code:: bash
+
+        delete.topic.enable=true
+
+5. Start the Kafka Server.
+
+    Run 'kafka-server-start.sh' with nohup as a background process:
+
+    .. code:: bash
+
+        $ sudo nohup kafka_2.11-0.11.0.0/bin/kafka-server-start.sh \
+          kafka_2.11-0.11.0.0/config/server.properties > kafka_2.11-0.11.0.0/kafka.log 2>&1 &
+
+6. Test Kafka Broker Installation
+
+    To test if the installation worked correctly there are two scripts, producer and consumer scripts.
+    These will allow you to see messages pushed to broker and receive from broker.
+
+    Producer (Publish "Hello World"):
+
+    .. code:: bash
+
+        $ echo "Hello, World" | kafka_2.11-0.11.0.0/bin/kafka-console-producer.sh \
+          --broker-list localhost:9092 --topic TopicTest > /dev/null
+
+    Consumer (Receive "Hello World"):
+
+    .. code:: bash
+
+        $ kafka_2.11-0.11.0.0/bin/kafka-console-consumer.sh --zookeeper \
+          localhost:2181 --topic TopicTest --from-beginning
+
+
+VES application configuration description:
+------------------------------------------
+
+Within the VES directory there is a configuration file called 'ves_app.conf'.
+
+.. note:: Details of the Vendor Event Listener REST service
 
 REST resources are defined with respect to a ServerRoot:
 
@@ -76,7 +205,7 @@ REST resources are of the form:
   Used as the "topicName" element in the REST  path (default: `empty`)
 
 **UseHttps** *true|false*
-  Allow plugin to use HTTPS instead of HTTP (default: `false`)
+  Allow application to use HTTPS instead of HTTP (default: `false`)
 
 **Username** *"username"*
   VES collector user name (default: `empty`)
@@ -95,6 +224,13 @@ REST resources are of the form:
 **ApiVersion** *version*
   Used as the "apiVersion" element in the REST path (default: `5.1`)
 
+**KafkaPort** *port*
+  Kafka Port (Default ``9092``)
+
+**KafkaBroker** *host*
+  Kafka Broker domain name. It can be an IP address or hostname of local or remote server
+  (default: localhost)
+
 Other collectd.conf configurations
 ----------------------------------
 Please ensure that FQDNLookup is set to false
@@ -132,7 +268,32 @@ Please ensure that the cpu plugin is enabled and configured as follows
         ValuesPercentage true
     </Plugin>
 
-To report the host name as a UUID the uuid plugin can be used.
+.. note::
+
+    The ``ReportByCpu`` option should be set to `true` (default)
+    if VES application is running on guest machine ('GuestRunning' = true).
+
+Please ensure that the aggregation plugin is enabled and configured as follows
+(required if 'GuestRunning' = true)
+
+.. code:: bash
+
+    LoadPlugin aggregation
+
+    <Plugin aggregation>
+        <Aggregation>
+                Plugin "cpu"
+                Type "percent"
+                GroupBy "Host"
+                GroupBy "TypeInstance"
+                SetPlugin "cpu-aggregation"
+                CalculateAverage true
+        </Aggregation>
+    </Plugin>
+
+If application is running on a guest side, it is important to enable uuid plugin
+too. In this case the hostname in event message will be represented as UUID
+instead of system host name.
 
 .. code:: bash
 
@@ -157,8 +318,8 @@ Please also ensure that the following plugins are enabled:
     LoadPlugin interface
     LoadPlugin memory
 
-VES plugin notification example
--------------------------------
+VES application with collectd notifications example
+---------------------------------------------------
 
 A good example of collectD notification is monitoring of the total CPU usage on a VM
 using the 'threshold' plugin. The following configuration will setup VES plugin to send 'Fault'