diff options
Diffstat (limited to 'docs/release')
| -rw-r--r-- | docs/release/userguide/array-item-parse-workflow.png | bin | 0 -> 24192 bytes | |||
| -rw-r--r-- | docs/release/userguide/collectd-ves-guest.conf | 29 | ||||
| -rw-r--r-- | docs/release/userguide/collectd-ves-host.conf | 26 | ||||
| -rw-r--r-- | docs/release/userguide/collectd.ves.userguide.rst | 834 | ||||
| -rw-r--r-- | docs/release/userguide/docker.userguide.rst | 326 | ||||
| -rw-r--r-- | docs/release/userguide/feature.userguide.rst | 11 | ||||
| -rw-r--r-- | docs/release/userguide/index.rst | 1 | ||||
| -rw-r--r-- | docs/release/userguide/parse-work-flow.png | bin | 0 -> 3949 bytes | |||
| -rw-r--r-- | docs/release/userguide/value-item-parse-workflow.png | bin | 0 -> 31526 bytes |
9 files changed, 1070 insertions, 157 deletions
diff --git a/docs/release/userguide/array-item-parse-workflow.png b/docs/release/userguide/array-item-parse-workflow.png Binary files differnew file mode 100644 index 00000000..22cbefff --- /dev/null +++ b/docs/release/userguide/array-item-parse-workflow.png diff --git a/docs/release/userguide/collectd-ves-guest.conf b/docs/release/userguide/collectd-ves-guest.conf new file mode 100644 index 00000000..2ed814d5 --- /dev/null +++ b/docs/release/userguide/collectd-ves-guest.conf @@ -0,0 +1,29 @@ +LoadPlugin logfile +<Plugin logfile> + LogLevel info + File "/opt/collectd/var/log/collectd.log" + Timestamp true + PrintSeverity false +</Plugin> + +LoadPlugin cpu +<Plugin cpu> + ReportByCpu true + ReportByState true + ValuesPercentage true +</Plugin> + +LoadPlugin interface +LoadPlugin memory +LoadPlugin load +LoadPlugin disk +LoadPlugin uuid + +LoadPlugin write_kafka +<Plugin write_kafka> + Property "metadata.broker.list" "localhost:9092" + <Topic "collectd"> + Format JSON + </Topic> +</Plugin> + diff --git a/docs/release/userguide/collectd-ves-host.conf b/docs/release/userguide/collectd-ves-host.conf new file mode 100644 index 00000000..76706b5c --- /dev/null +++ b/docs/release/userguide/collectd-ves-host.conf @@ -0,0 +1,26 @@ +LoadPlugin logfile +<Plugin logfile> + LogLevel info + File "/opt/collectd/var/log/collectd.log" + Timestamp true + PrintSeverity false +</Plugin> + +LoadPlugin cpu + +LoadPlugin virt +<Plugin virt> + Connection "qemu:///system" + RefreshInterval 60 + HostnameFormat uuid + PluginInstanceFormat name + ExtraStats "cpu_util" +</Plugin> + +LoadPlugin write_kafka +<Plugin write_kafka> + Property "metadata.broker.list" "localhost:9092" + <Topic "collectd"> + Format JSON + </Topic> +</Plugin> diff --git a/docs/release/userguide/collectd.ves.userguide.rst b/docs/release/userguide/collectd.ves.userguide.rst index 7077cecd..1f69b6ad 100644 --- a/docs/release/userguide/collectd.ves.userguide.rst +++ b/docs/release/userguide/collectd.ves.userguide.rst @@ -5,57 +5,19 @@ ========================== VES Application User Guide ========================== -The Barometer repository contains a python based application for VES. + +The Barometer repository contains a python based application for VES (VNF Event +Stream) which receives the `collectd`_ specific metrics via `Kafka`_ bus, +normalizes the metric data into the VES message and sends it into the VES +collector. The application currently supports pushing platform relevant metrics through the additional measurements field for VES. -Collectd has a write_kafka plugin that will send collectd metrics and values to -a Kafka Broker. -The VES application uses Kafka Consumer to receive metrics from the Kafka -Broker. - -Installation Instructions: --------------------------- -1. Clone this repo: - - .. code:: bash - - git clone https://gerrit.opnfv.org/gerrit/barometer - -2. Install collectd - - .. code:: bash - - $ sudo apt-get install collectd - - CentOS 7.x use: - - .. code:: bash - - $ sudo yum install collectd +Collectd has a ``write_kafka`` plugin that sends collectd metrics and values to +a Kafka Broker. The VES application uses Kafka Consumer to receive metrics +from the Kafka Broker. - .. note:: You may need to add epel repository if the above does not work. - - .. code:: bash - - $ sudo yun install epel-release - -3. Modify the collectd configuration script: `/etc/collectd/collectd.conf` - - .. code:: bash - - <Plugin write_kafka> - Property "metadata.broker.list" "localhost:9092" - <Topic "collectd"> - Format JSON - </Topic> - </Plugin> - - .. note:: - - The above configuration is for a single host setup. Simply change localhost to remote - server IP addess or hostname. Install Kafka Broker -------------------- @@ -86,7 +48,9 @@ Install Kafka Broker $ sudo yum install zookeeper - .. note:: You may need to add the the repository that contains zookeeper + .. note:: You may need to add the repository that contains zookeeper. + To do so, follow the step below and try to install `zookeeper` again + using steps above. Otherwise, skip next step. .. code:: bash @@ -99,11 +63,20 @@ Install Kafka Broker $ sudo zookeeper-server start + if you get the error message like ``ZooKeeper data directory is missing at /var/lib/zookeeper`` + during the start of zookeeper, initialize zookeeper data directory using + the command below and start zookeeper again, otherwise skip the next step. + + .. code:: bash + + $ sudo /usr/lib/zookeeper/bin/zkServer-initialize.sh + No myid provided, be sure to specify it in /var/lib/zookeeper/myid if using non-standalone + To test if Zookeeper is running as a daemon. .. code:: bash - $ sudo telnet localhost 2181 + $ telnet localhost 2181 Type 'ruok' & hit enter. Expected response is 'imok'. Zookeeper is running fine. @@ -115,25 +88,26 @@ Install Kafka Broker .. code:: bash + $ sudo yum install python-pip $ sudo pip install kafka-python 2. Download Kafka: .. code:: bash - $ sudo wget "http://www-eu.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz" + $ wget "http://www-eu.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz" 3. Extract the archive: .. code:: bash - $ sudo tar -xvzf kafka_2.11-0.11.0.0.tgz + $ tar -xvzf kafka_2.11-0.11.0.0.tgz 4. Configure Kafka Server: .. code:: bash - $ sudo vi kafka_2.11-0.11.0.0/config/server.properties + $ vi kafka_2.11-0.11.0.0/config/server.properties By default Kafka does not allow you to delete topics. Please uncomment: @@ -170,180 +144,732 @@ Install Kafka Broker localhost:2181 --topic TopicTest --from-beginning -VES application configuration description: ------------------------------------------- +Install collectd +---------------- + +Install development tools: -Within the VES directory there is a configuration file called 'ves_app.conf'. +.. code:: bash + + $ sudo yum group install 'Development Tools' -.. note:: Details of the Vendor Event Listener REST service +.. The libkafka installed via yum pkg manager is 0.11.0 which doesn't work with + collectd (compilation issue). Thus, we have to use the library installed + from sources using latest stable version which works with collectd. -REST resources are defined with respect to a ServerRoot: +Install Apache Kafka C/C++ client library: .. code:: bash - ServerRoot = https://{Domain}:{Port}/{optionalRoutingPath} + $ git clone https://github.com/edenhill/librdkafka.git ~/librdkafka + $ cd ~/librdkafka + $ git checkout -b v0.9.5 v0.9.5 + $ ./configure --prefix=/usr + $ make + $ sudo make install + +Build collectd with Kafka support: + +.. code:: bash + + $ git clone https://github.com/collectd/collectd.git ~/collectd + $ cd ~/collectd + $ ./build.sh + $ ./configure --with-librdkafka=/usr --without-perl-bindings --enable-perl=no + $ make && sudo make install + +Configure and start collectd. Create ``/opt/collectd/etc/collectd.conf`` +collectd configuration file as following: + +.. note:: The following collectd configuration file allows user to run VES + application in the guest mode. To run the VES in host mode, please follow + the `Configure VES in host mode`_ steps. + +.. include:: collectd-ves-guest.conf + :code: bash + +Start collectd process as a service as described in :ref:`install-collectd-as-a-service`. + + +Setup VES Test Collector +------------------------ + +.. note:: Test Collector setup is required only for VES application testing + purposes. + +Install dependencies: -REST resources are of the form: +.. code:: bash + + $ sudo pip install jsonschema + +Clone VES Test Collector: + +.. code:: bash + + $ git clone https://github.com/att/evel-test-collector.git ~/evel-test-collector + +Modify VES Test Collector config file to point to existing log directory and +schema file: .. code:: bash + $ sed -i.back 's/^\(log_file[ ]*=[ ]*\).*/\1collector.log/' ~/evel-test-collector/config/collector.conf + $ sed -i.back 's/^\(schema_file[ ]*=.*\)event_format_updated.json$/\1CommonEventFormat.json/' ~/evel-test-collector/config/collector.conf + +Start VES Test Collector: + +.. code:: bash + + $ cd ~/evel-test-collector/code/collector + $ nohup python ./collector.py --config ../../config/collector.conf > collector.stdout.log & + + +Setup VES application (guest mode) +---------------------------------- + +Install dependencies: + +.. code:: bash + + $ sudo pip install pyyaml + +Clone Barometer repo and start the VES application: + +.. code:: bash + + $ git clone https://gerrit.opnfv.org/gerrit/barometer ~/barometer + $ cd ~/barometer/3rd_party/collectd-ves-app/ves_app + $ nohup python ves_app.py --events-schema=guest.yaml --config=ves_app_config.conf > ves_app.stdout.log & + +.. note:: + + The above configuration is used for a localhost. The VES application can be + configured to use remote real VES collector and remote Kafka server. To do + so, the IP addresses/host names needs to be changed in ``collector.conf`` + and ``ves_app_config.conf`` files accordingly. + + +Configure VES in host mode +-------------------------- + +Running the VES in host mode looks like steps described in +`Setup VES application (guest mode)`_ but with the following exceptions: + +- The ``host.yaml`` configuration file should be used instead of ``guest.yaml`` + file when VES application is running. + +- Collectd should be running on host machine only. + +- Addition ``libvirtd`` dependencies needs to be installed on a host where + collectd daemon is running. To install those dependencies, see :ref:`virt-plugin` + section of Barometer user guide. + +- At least one VM instance should be up and running by hypervisor on the host. + +- The next (minimum) configuration needs to be provided to collectd to be able + to generate the VES message to VES collector. + + .. include:: collectd-ves-host.conf + :code: bash + + to apply this configuration, the ``/opt/collectd/etc/collectd.conf`` file + needs to be modified based on example above and collectd daemon needs to + be restarted using the command below: + + .. code:: bash + + $ sudo systemctl restart collectd + +.. note:: The list of the plugins can be extented depends on your needs. + + +VES application configuration description +----------------------------------------- + +**Details of the Vendor Event Listener REST service** + +REST resources are defined with respect to a ``ServerRoot``:: + + ServerRoot = https://{Domain}:{Port}/{optionalRoutingPath} + +REST resources are of the form:: + {ServerRoot}/eventListener/v{apiVersion}` {ServerRoot}/eventListener/v{apiVersion}/{topicName}` {ServerRoot}/eventListener/v{apiVersion}/eventBatch` +Within the VES directory (``3rd_party/collectd-ves-app/ves_app``) there is a +configuration file called ``ves_app.conf``. The description of the +configuration options are described below: + **Domain** *"host"* VES domain name. It can be IP address or hostname of VES collector - (default: `127.0.0.1`) + (default: ``127.0.0.1``) **Port** *port* - VES port (default: `30000`) + VES port (default: ``30000``) **Path** *"path"* - Used as the "optionalRoutingPath" element in the REST path (default: `empty`) + Used as the "optionalRoutingPath" element in the REST path (default: empty) **Topic** *"path"* - Used as the "topicName" element in the REST path (default: `empty`) + Used as the "topicName" element in the REST path (default: empty) **UseHttps** *true|false* - Allow application to use HTTPS instead of HTTP (default: `false`) + Allow application to use HTTPS instead of HTTP (default: ``false``) **Username** *"username"* - VES collector user name (default: `empty`) + VES collector user name (default: empty) **Password** *"passwd"* - VES collector password (default: `empty`) - -**FunctionalRole** *"role"* - Used as the 'functionalRole' field of 'commonEventHeader' event (default: - `Collectd VES Agent`) + VES collector password (default: empty) **SendEventInterval** *interval* This configuration option controls how often (sec) collectd data is sent to - Vendor Event Listener (default: `20`) + Vendor Event Listener (default: ``20``) **ApiVersion** *version* - Used as the "apiVersion" element in the REST path (default: `5.1`) + Used as the "apiVersion" element in the REST path (default: ``5.1``) **KafkaPort** *port* Kafka Port (Default ``9092``) **KafkaBroker** *host* Kafka Broker domain name. It can be an IP address or hostname of local or remote server - (default: localhost) + (default: ``localhost``) -Other collectd.conf configurations ----------------------------------- -Please ensure that FQDNLookup is set to false -.. code:: bash +VES YAML configuration format +----------------------------- - FQDNLookup false +The format of the VES message is generated by the VES application based on the +YAML schema configuration file provided by user via ``--events-schema`` +command-line option of the application. -Please ensure that the virt plugin is enabled and configured as follows. +.. note:: Use ``--help`` option of VES application to see the + description of all available options -.. code:: bash +.. note:: The detailed installation guide of the VES application is described in + the `VES Application User Guide`_. - LoadPlugin virt +The message defined by YAML schema should correspond to format defined in +`VES shema definition`_. - <Plugin virt> - Connection "qemu:///system" - RefreshInterval 60 - HostnameFormat uuid - PluginInstanceFormat name - ExtraStats "cpu_util perf" - </Plugin> +.. warning:: This section doesn't explain the YAML language itself, so the + knowledge of the YAML language is required before continue reading it! +Since, the VES message output is a JSON format, it's recommended to understand +how YAML document is converted to JSON before starting to create a new YAML +definition for the VES. E.g.: -.. note:: For more detailed information on the `virt` plugin configuration, - requirements etc., please see the userguide of the collectd virt plugin. +The following YAML document: -Please ensure that the cpu plugin is enabled and configured as follows +.. code:: yaml -.. code:: bash + --- + additionalMeasurements: + plugin: somename + instance: someinstance - LoadPlugin cpu +will be converted to JSON like this: - <Plugin cpu> - ReportByCpu false - ValuesPercentage true - </Plugin> +.. code:: json -.. note:: + { + "additionalMeasurements": { + "instance": "someinstance", + "plugin": "somename" + } + } - The ``ReportByCpu`` option should be set to `true` (default) - if VES application is running on guest machine ('GuestRunning' = true). +.. note:: The `YAML syntax` section of `PyYAML documentation`_ can be used + as a reference for this. -Please ensure that the aggregation plugin is enabled and configured as follows -(required if 'GuestRunning' = true) -.. code:: bash +VES message types +----------------- - LoadPlugin aggregation +The VES agent can generate two type of messages which are sent to the VES +collector. Each message type must be specified in the YAML configuration file +using a specific YAML tag. - <Plugin aggregation> - <Aggregation> - Plugin "cpu" - Type "percent" - GroupBy "Host" - GroupBy "TypeInstance" - SetPlugin "cpu-aggregation" - CalculateAverage true - </Aggregation> - </Plugin> +Measurements + This type is used to send a message defined in YAML configuration to the VES + collector with a specified interval (default is 20 sec and it's configurable + via command line option of the application). This type can be specified in + the configuration using ``!Measurements`` tag. For instance: -If application is running on a guest side, it is important to enable uuid plugin -too. In this case the hostname in event message will be represented as UUID -instead of system host name. + .. code:: yaml -.. code:: bash + --- + # My comments + My Host Measurements: !Measurements + ... # message definition - LoadPlugin uuid +Events + This type is used to send a message defined in YAML configuration to the VES + collector when collectd notification is received from Kafka bus (collectd + ``write_kafka`` plugin). This type can be specified in the configuration + using ``!Events`` tag. For instance: -If a custom UUID needs to be provided, the following configuration is required in collectd.conf -file: + .. code:: yaml -.. code:: bash + --- + # My comments + My Events: !Events + ... # event definition - <Plugin uuid> - UUIDFile "/etc/uuid" - </Plugin> -Where "/etc/uuid" is a file containing custom UUID. +Collectd metrics in VES +----------------------- -Please also ensure that the following plugins are enabled: +The VES application caches collectd metrics received via Kafka bus. The data +is stored in a table format. It's important to know it before mapping collectd +metric values to message defined in YAML configuration file. -.. code:: bash +VES collectd metric cache example: - LoadPlugin disk - LoadPlugin interface - LoadPlugin memory ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| host | plugin | plugin_instance | type | type_instace | time | value | ds_name | interval | ++===========+===========+====================+===========+================+===================+========+=========+==========+ +| localhost | cpu | | percent | user | 1509535512.30567 | 16 | value | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| localhost | memory | | memory | free | 1509535573.448014 | 798456 | value | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| localhost | interface | | eth0 | if_packets | 1509544183.956496 | 253 | rx | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| 7ec333e7 | virt | Ubuntu-12.04.5-LTS | percent | virt_cpu_total | 1509544402.378035 | 0.2 | value | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| 7ec333e7 | virt | Ubuntu-12.04.5-LTS | memory | rss | 1509544638.55119 | 123405 | value | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| 7ec333e7 | virt | Ubuntu-12.04.5-LTS | if_octets | vnet1 | 1509544646.27108 | 67 | tx | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| cc659a52 | virt | Ubuntu-16.04 | percent | virt_cpu_total | 1509544745.617207 | 0.3 | value | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| cc659a52 | virt | Ubuntu-16.04 | memory | rss | 1509544754.329411 | 4567 | value | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ +| cc659a52 | virt | Ubuntu-16.04 | if_octets | vnet0 | 1509544760.720381 | 0 | rx | 10 | ++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+ -VES application with collectd notifications example ---------------------------------------------------- +It's possible from YAML configuration file to refer to any field of any row of +the table via special YAML tags like ``ValueItem`` or ``ArrayItem``. See the +`Collectd metric reference`_ reference for more details. -A good example of collectD notification is monitoring of the total CPU usage on a VM -using the 'threshold' plugin. The following configuration will setup VES plugin to send 'Fault' -event every time a total VM CPU value is out of range (e.g.: WARNING: VM CPU TOTAL > 50%, -CRITICAL: VM CPU TOTAL > 96%) and send 'Fault' NORMAL event if the CPU value is back -to normal. In the example below, there is one VM with two CPUs configured which is running -on the host with a total of 48 cores. Thus, the threshold value 2.08 (100/48) means that -one CPU of the VM is fully loaded (e.g.: 50% of total CPU usage of the VM) and 4.0 means -96% of total CPU usage of the VM. Those values can also be obtained by virt-top -command line tool. +.. note:: The `collectd data types file`_ contains map of ``type`` to + ``ds_name`` field. It can be used to get possible value for ``ds_name`` + field. See the `collectd data types description`_ for more details on + collectd data types. -.. code:: bash - LoadPlugin threshold +Aging of collectd metric +~~~~~~~~~~~~~~~~~~~~~~~~ + +If the metric will not be updated by the collectd during the double metric +interval time, it will be removed (aged) from VES internal cache. + + +VES YAML references +------------------- + +There are four type of references supported by the YAML format: System, +Config, Collectd metric and Collectd notification. The format of the reference +is the following: + +.. code:: yaml + + "{<ref type>.<attribute name>}" + +.. note:: Possible values for ``<ref type>`` and ``<attribute name>`` are + described in farther sections. + + +System reference +~~~~~~~~~~~~~~~~ + +This reference is used to get system statistics like time, date etc. The system +reference (``<ref type>`` = ``system``) can be used in any place of the YAML +configuration file. This type of reference provides the following attributes: + +``hostname`` + The name of the host where VES application is running. + +``id`` + Unique ID during VES application runtime. + +``time`` + Current time in seconds since the Epoch. For example ``1509641411.631951``. + +``date`` + Current date in ISO 8601 format, ``YYYY-MM-DD``. For instance ``2017-11-02``. + + +For example: + +.. code:: yaml + + Date: "{system.date}" + + +Config reference +~~~~~~~~~~~~~~~~ + +This reference is used to get VES configuration described in `VES application +configuration description`_ sectoin. The reference (``<ref type>`` = ``config``) +can be used in any place of the YAML configuration file. This type of reference +provides the following attributes: + +``interval`` + Measurements dispatch interval. It referenses to ``SendEventInterval`` + configuration of the VES application. + + +For example: + +.. code:: yaml + + Interval: "{config.interval}" + + +Collectd metric reference +~~~~~~~~~~~~~~~~~~~~~~~~~ + +This reference is used to get the attribute value of collectd metric from the +VES cache. The reference (``<ref type>`` = ``vl``) can be used ONLY inside +``Measurements``, ``ValueItem`` and ``ArrayItem`` tags. Using the reference +inside a helper tag is also allowed if the helper tag is located inside the +tag where the reference is allowed (e.g.: ``ArrayItem``). The +``<attribute name>`` name corresponds to the table field name described in +`Collectd metrics in VES`_ section. For example: + +.. code:: yaml + + name: "{vl.type}-{vl.type_instance}" + + +Collectd notification reference +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This reference is used to get the attribute value of received collectd +notification. The reference (``<ref type>`` = ``n``) can be used ONLY inside +``Events`` tag. Using the reference inside a helper tag is also allowed if +the helper tag is located inside the ``Events`` tag. This type of reference +provides the following attributes: + +``host`` + The hostname of received collectd notification. + +``plugin`` + The plugin name of received collectd notification. + +``plugin_instance`` + The plugin instance of the received collectd notification. + +``type`` + The type name of the received collectd notification. + +``type_instance`` + The type instance of the received collectd notification. + +``severity`` + The severity of the received collectd notification. + +``message`` + The message of the received collectd notification. + + +.. note:: The exact value for each attribute depends on the collectd plugin + which may generate the notification. Please refer to the + `collectd plugin description`_ document to get more details on the specific + collectd plugin. + +YAML config example: + +.. code:: yaml + + sourceId: "{n.plugin_instance}" + + +Collectd metric mapping YAML tags +--------------------------------- + +This section describes the YAML tags used to map collectd metric values in the +YAML configuration file. + + +Measurements tag +~~~~~~~~~~~~~~~~ + +This tag is a YAML map which is used to define the VES measurement message. It's +allowed to be used multiple times in the document (e.g.: you can define multiple +VES messages in one YAML document). This tag works in the same way as `ArrayItem +tag`_ does and all keys have the same description/rules. + + +ValueItem tag +~~~~~~~~~~~~~ + +This tag is used to select a collectd metric and get its attribute value using +`Collectd metric reference`_. The type of this tag is a YAML array of maps with +the possible keys described below. + +``SELECT`` (required) + Is a YAML map which describes the select metric criteria. Each key name of the + map must correspond to the table field name described in `Collectd metrics in VES`_ + section. + +``VALUE`` (optional) + Describes the value to be assigned. If not provided, the default + ``!Number "{vl.value}"`` expression is used. + +``DEFAULT`` (optional) + Describes the default value which will be assigned if no metric is selected + by ``SELECT`` criteria. + + +ValueItem tag description example: + +.. code:: yaml + + memoryFree: !ValueItem + - SELECT: + plugin: memory + type: memory + type_instance: rss + - VALUE: !Bytes2Kibibytes "{vl.value}" + - DEFAULT: 0 + +The tag process workflow is described on the figure below. + +.. figure:: value-item-parse-workflow.png + + YAML ValueItem tag process workflow + + +ArrayItem tag +~~~~~~~~~~~~~ + +This tag is used to select a list of collectd metrics and generate a YAML array +of YAML items described by ``ITEM-DESC`` key. If no collectd metrics are +selected by the given criteria, the empty array will be returned. + +``SELECT`` (optional) + Is a YAML map which describes the select metrics criteria. Each key name of + the map must correspond to the table field name described in `Collectd + metrics in VES`_ section. The value of the key may be regular expression. To + enable regular expression in the value, the YAML string containing ``/`` char + at the beginning and at the end should be used. For example: + + .. code:: yaml + + plugin: "/^(?!virt).*$/" # selected all metrics except ``virt`` plugin + + The VES application uses the python RE library to work with regular + expression specified in the YAML configuration. Please refer to `python + regular expression syntax`_ documentation for more details on a syntax + used by the VES. + + Multiple ``SELECT`` keys are allowed by the tag. If nor ``SELECT`` or + ``INDEX-KEY`` key is specified, the VES error is generated. + +``INDEX-KEY`` (optional) + Is a YAML array which describes the unique fields to be selected by the tag. + Each element of array is a YAML string which should be one of the table field + name described in `Collectd metrics in VES`_ section. Please note, if this + key is used, only fields specified by the key can be used as a collectd + reference in the ``ITEM-DESC`` key. + +``ITEM-DESC`` (required) + Is a YAML map which describes each element of the YAML array generated by + the tag. Using ``ArrayItem`` tags and other `Helper YAML tags`_ are also + allowed in the definition of the key. + +In the example below, the ArrayItem tag is used to generate an array of +``ITEM-DESC`` items for each collectd metrics except ``virt`` plugin with +unique ``plugin``, ``plugin_instance`` attribute values. + +.. code:: yaml + + Measurements: !ArrayItem + - SELECT: + plugin: "/^(?!virt).*$/" + - INDEX-KEY: + - plugin + - plugin_instance + - ITEM-DESC: + name: !StripExtraDash "{vl.plugin}-{vl.plugin_instance}" + +The tag process workflow is described on the figure below. + +.. figure:: array-item-parse-workflow.png + + YAML ArrayItem tag process workflow + + +Collectd event mapping YAML tags +-------------------------------- + +This section describes the YAML tags used to map the collectd notification to +the VES event message in the YAML configuration file. + + +Events tag +~~~~~~~~~~ + +This tag is a YAML map which is used to define the VES event message. It's +allowed to be used multiple times in the document (e.g.: you can map multiple +collectd notification into VES message in one YAML document). The possible +keys of the tag are described below. + +``CONDITION`` (optional) + Is a YAML map which describes the select metric criteria. Each key name of + the map must correspond to the name of attribute provided by `Collectd + notification reference`_. If no such key provided, any collectd notification + will map the defined YAML message. + +``ITEM-DESC`` (required) + Is a YAML map which describes the message generated by this tag. Only `Helper + YAML tags`_ are allowed in the definition of the key. + +The example of the VES event message which will be generate by the VES +application when collectd notification of the ``virt`` plugin is triggered +is described below. + +.. code:: yaml + + --- + Virt Event: !Events + - ITEM-DESC: + event: + commonEventHeader: + domain: fault + eventType: Notification + sourceId: &event_sourceId "{n.plugin_instance}" + sourceName: *event_sourceId + lastEpochMicrosec: !Number "{n.time}" + startEpochMicrosec: !Number "{n.time}" + faultFields: + alarmInterfaceA: !StripExtraDash "{n.plugin}-{n.plugin_instance}" + alarmCondition: "{n.severity}" + faultFieldsVersion: 1.1 + - CONDITION: + plugin: virt + + +Helper YAML tags +---------------- + +This section describes the YAML tags used as utilities for formatting the output +message. The YAML configuration process workflow is described on the figure +below. + +.. figure:: parse-work-flow.png + + YAML configuration process workflow + + +Convert string to number tag +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``!Number`` tag is used in YAML configuration file to convert string value into +the number type. For instance: + +.. code:: yaml + + lastEpochMicrosec: !Number "3456" + +The output of the tag will be the JSON number. + +.. code:: json + + { + lastEpochMicrosec: 3456 + } + + +Convert bytes to Kibibytes tag +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``!Bytes2Kibibytes`` tag is used in YAML configuration file to convert +bytes into kibibytes (1 kibibyte = 1024 bytes). For instance: + +.. code:: yaml + + memoryConfigured: !Bytes2Kibibytes 4098 + memoryConfigured: !Bytes2Kibibytes "1024" + +The output of the tag will be the JSON number. + +.. code:: json + + { + memoryConfigured: 4 + memoryConfigured: 1 + } + + +Convert one value to another tag +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``!MapValue`` tag is used in YAML configuration file to map one value +into another value defined in the configuration. For instance: + +.. code:: yaml + + Severity: !MapValue + VALUE: Failure + TO: + Failure: Critical + Error: Warnign + +The output of the tag will be the mapped value. + +.. code:: json + + { + Severity: Critical + } + + +Strip extra dash tag +~~~~~~~~~~~~~~~~~~~~ + +The ``!StripExtraDash`` tag is used in YAML configuration file to strip extra +dashes in the string (dashes at the beginning, at the end and double dashes). +For example: + +.. code:: yaml + + name: !StripExtraDash string-with--extra-dashes- + +The output of the tag will be the JSON string with extra dashes removed. + +.. code:: json + + { + name: string-with-extra-dashes + } + + +Limitations +----------- + +#. Only one document can be defined in the same YAML configuration file. + +#. The collectd notification is not supported by `Kafka collectd plugin`_. Due to + this limitation, the collectd notifications cannot be received by the VES + application and the defined YAML event will not be generated and sent to the + VES collector. Please note, that VES YAML format already supports the events + definition and the format is descibed in the document. - <Plugin "threshold"> - <Plugin "virt"> - <Type "percent"> - WarningMax 2.08 - FailureMax 4.0 - Instance "virt_cpu_total" - </Type> - </Plugin> - </Plugin> -More detailed information on how to configure collectD thresholds can be found at -https://collectd.org/documentation/manpages/collectd-threshold.5.shtml +.. _collectd: http://collectd.org/ +.. _Kafka: https://kafka.apache.org/ +.. _`VES`: https://wiki.opnfv.org/display/fastpath/VES+plugin+updates +.. _`VES shema definition`: https://gerrit.onap.org/r/gitweb?p=demo.git;a=tree;f=vnfs/VES5.0/evel/evel-test-collector/docs/att_interface_definition;hb=refs/heads/master +.. _`PyYAML documentation`: https://pyyaml.org/wiki/PyYAMLDocumentation +.. _`collectd plugin description`: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod +.. _`collectd data types file`: https://github.com/collectd/collectd/blob/master/src/types.db +.. _`collectd data types description`: https://github.com/collectd/collectd/blob/master/src/types.db.pod +.. _`python regular expression syntax`: https://docs.python.org/2/library/re.html#regular-expression-syntax +.. _`Kafka collectd plugin`: https://collectd.org/wiki/index.php/Plugin:Write_Kafka diff --git a/docs/release/userguide/docker.userguide.rst b/docs/release/userguide/docker.userguide.rst new file mode 100644 index 00000000..c6f56b2b --- /dev/null +++ b/docs/release/userguide/docker.userguide.rst @@ -0,0 +1,326 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) <optionally add copywriters name> + +=================================== +OPNFV Barometer Docker User Guide +=================================== + +.. contents:: + :depth: 3 + :local: + +Barometer docker image description +----------------------------------- +.. Describe the specific features and how it is realised in the scenario in a brief manner +.. to ensure the user understand the context for the user guide instructions to follow. + +The intention of this user guide is to outline how to install and test the +barometer docker image that can be built from the Dockerfile available in the +barometer repository. + +.. note:: + The Dockerfile is available in the docker/ directory in the barometer repo. + The Dockerfile builds a CentOS 7 docker image. + +The barometer docker image gives you a collectd installation that includes all +the barometer plugins. + +.. note:: + The container MUST be run as a privileged container. + +Collectd is a daemon which collects system performance statistics periodically +and provides a variety of mechanisms to publish the collected metrics. It +supports more than 90 different input and output plugins. Input plugins +retrieve metrics and publish them to the collectd deamon, while output plugins +publish the data they receive to an end point. collectd also has infrastructure +to support thresholding and notification. + +Barometer docker image has enabled the following collectd plugins (in addition +to the standard collectd plugins): + +* hugepages plugin +* Open vSwitch events Plugin +* Open vSwitch stats Plugin +* mcelog plugin +* PMU plugin +* RDT plugin +* virt +* SNMP Agent + +Plugins and third party applications in Barometer repository that will be available in the +docker image: + +* Open vSwitch PMD stats +* ONAP VES application +* gnocchi plugin +* aodh plugin +* Legacy/IPMI + + +Installing Docker +----------------- +.. Describe the specific capabilities and usage for <XYZ> feature. +.. Provide enough information that a user will be able to operate the feature on a deployed scenario. + +On Ubuntu +^^^^^^^^^^ +.. note:: + * sudo permissions are required to install docker. + * These instructions are for Ubuntu 16.10 + +To install docker: + +.. code:: bash + + $ sudo apt-get install curl + $ sudo curl -fsSL https://get.docker.com/ | sh + $ sudo usermod -aG docker <username> + $ sudo systemctl status docker + +Replace <username> above with an appropriate user name. + +On CentOS +^^^^^^^^^^ +.. note:: + * sudo permissions are required to install docker. + * These instructions are for CentOS 7 + +To install docker: + +.. code:: bash + + $ sudo yum remove docker docker-common docker-selinux docker-engine + $ sudo yum install -y yum-utils device-mapper-persistent-data lvm2 + $ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo + $ sudo yum-config-manager --enable docker-ce-edge + $ sudo yum-config-manager --enable docker-ce-test + $ sudo yum install docker-ce + $ sudo usermod -aG docker <username> + $ sudo systemctl status docker + +Replace <username> above with an appropriate user name. + +.. note:: + If this is the first time you are installing a package from a recently added + repository, you will be prompted to accept the GPG key, and the key’s + fingerprint will be shown. Verify that the fingerprint is correct, and if so, + accept the key. The fingerprint should match060A 61C5 1B55 8A7F 742B 77AA C52F + EB6B 621E 9F35. + + Retrieving key from https://download.docker.com/linux/centos/gpg + Importing GPG key 0x621E9F35: + Userid : "Docker Release (CE rpm) <docker@docker.com>" + Fingerprint: 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35 + From : https://download.docker.com/linux/centos/gpg + Is this ok [y/N]: y + +Proxy Configuration: +^^^^^^^^^^^^^^^^^^^^ +.. note:: + This applies for both CentOS and Ubuntu. + +If you are behind an HTTP or HTTPS proxy server, you will need to add this +configuration in the Docker systemd service file. + +1. Create a systemd drop-in directory for the docker service: + +.. code:: bash + + $ sudo mkdir -p /etc/systemd/system/docker.service.d + +2. Create a file +called /etc/systemd/system/docker.service.d/http-proxy.conf that adds +the HTTP_PROXY environment variable: + +.. code:: bash + + [Service] + Environment="HTTP_PROXY=http://proxy.example.com:80/" + +Or, if you are behind an HTTPS proxy server, create a file +called /etc/systemd/system/docker.service.d/https-proxy.conf that adds +the HTTPS_PROXY environment variable: + +.. code:: bash + + [Service] + Environment="HTTPS_PROXY=https://proxy.example.com:443/" + +Or create a single file with all the proxy configurations: +/etc/systemd/system/docker.service.d/proxy.conf + +.. code:: bash + + [Service] + Environment="HTTP_PROXY=http://proxy.example.com:80/" + Environment="HTTPS_PROXY=https://proxy.example.com:443/" + Environment="FTP_PROXY=ftp://proxy.example.com:443/" + Environment="NO_PROXY=localhost" + +3. Flush changes: + +.. code:: bash + + $ sudo systemctl daemon-reload + +4. Restart Docker: + +.. code:: bash + + $ sudo systemctl restart docker + +5. Check docker environment variables: + +.. code:: bash + + sudo systemctl show --property=Environment docker + +Test docker installation +^^^^^^^^^^^^^^^^^^^^^^^^ +.. note:: + This applies for both CentOS and Ubuntu. + +.. code:: bash + + $ sudo docker run hello-world + +The output should be something like: + +.. code:: bash + + Unable to find image 'hello-world:latest' locally + latest: Pulling from library/hello-world + 5b0f327be733: Pull complete + Digest: sha256:07d5f7800dfe37b8c2196c7b1c524c33808ce2e0f74e7aa00e603295ca9a0972 + Status: Downloaded newer image for hello-world:latest + + Hello from Docker! + This message shows that your installation appears to be working correctly. + + To generate this message, Docker took the following steps: + 1. The Docker client contacted the Docker daemon. + 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. + 3. The Docker daemon created a new container from that image which runs the + executable that produces the output you are currently reading. + 4. The Docker daemon streamed that output to the Docker client, which sent it + to your terminal. + +To try something more ambitious, you can run an Ubuntu container with: + +.. code:: bash + + $ docker run -it ubuntu bash + +Build the barometer docker image +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code:: bash + + $ cd barometer + $ sudo docker build -t barometer_image --build-arg http_proxy=`echo $http_proxy` \ + --build-arg https_proxy=`echo $https_proxy` -f docker/Dockerfile . + +.. note:: + In the above mentioned ``docker build`` command, http_proxy & https_proxy arguments needs to be passed only if system is behind an HTTP or HTTPS proxy server. + +Check the docker images: + +.. code:: bash + + $ sudo docker images + +Output should contain a barometer image: + +.. code:: + + REPOSITORY TAG IMAGE ID CREATED SIZE + barometer_image latest 05f2a3edd96b 3 hours ago 1.2GB + centos 7 196e0ce0c9fb 4 weeks ago 197MB + centos latest 196e0ce0c9fb 4 weeks ago 197MB + hello-world latest 05a3bd381fc2 4 weeks ago 1.84kB + +Run the barometer docker image: + +.. code:: bash + + $ sudo docker run -tid --net=host -v `pwd`/../src/collectd_sample_configs:/opt/collectd/etc/collectd.conf.d \ + -v /var/run:/var/run -v /tmp:/tmp --privileged barometer_image /run_collectd.sh + +.. note:: + The docker barometer image contains configuration for all the collectd plugins. In the command + above we are overriding /opt/collectd/etc/collectd.conf.d by mounting a host directory + `pwd`/../src/collectd_sample_configs thta contains only the sample configurations we are interested + in running. + +To make some changes run: + +.. code:: bash + + sudo docker exec -tid barometer_image /bin/bash + +Check your docker image is running + +.. code:: bash + + sudo docker ps + +Build the influxdb + Grafana docker images +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Start by installing docker compose: + +.. code:: bash + + $ sudo curl -L https://github.com/docker/compose/releases/download/1.17.0/docker-compose-`uname -s`-`uname -m` -o /usr/bin/docker-compose + +.. note:: + Use the latest Compose release number in the download command. The above command is an example, + and it may become out-of-date. To ensure you have the latest version, check the Compose repository + release page on GitHub. + +2. Apply executable permissions to the binary: + +.. code:: bash + + $ sudo chmod +x /usr/bin/docker-compose + +3. Test the installation. + +.. code:: bash + + $ sudo docker-compose --version + +4. Run the get_types_db.sh script in barometer/docker + +5. Run the docker containers: + +.. code:: bash + + $ sudo docker-compose up -d + +6. Check your docker images are running + +.. code:: bash + + $ sudo docker ps + +7. Run the script to create the CPU dashboard barometer/docker: + +.. code:: bash + + $ ./configure_grafana.sh + +8. Connect to <host_ip>:3000 with a browser and log into grafana: admin/admin + +Testing the docker image +^^^^^^^^^^^^^^^^^^^^^^^^ + +TODO + +References +^^^^^^^^^^^ +.. [1] https://docs.docker.com/engine/admin/systemd/#httphttps-proxy +.. [2] https://docs.docker.com/engine/installation/linux/docker-ce/centos/#install-using-the-repository +.. [3] https://docs.docker.com/engine/userguide/ + diff --git a/docs/release/userguide/feature.userguide.rst b/docs/release/userguide/feature.userguide.rst index 6c269b14..699412d9 100644 --- a/docs/release/userguide/feature.userguide.rst +++ b/docs/release/userguide/feature.userguide.rst @@ -714,6 +714,7 @@ Then you can run the mcelog test suite with This will inject different classes of errors and check that the mcelog triggers runs. There will be some kernel messages about page offlining attempts. The test will also lose a few pages of memory in your system (not significant). + .. note:: This test will kill any running mcelog, which needs to be restarted manually afterwards. @@ -1037,6 +1038,8 @@ https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod For more details on AgentX subagent, please see: http://www.net-snmp.org/tutorial/tutorial-5/toolkit/demon/ +.. _virt-plugin: + virt plugin ^^^^^^^^^^^^ Repo: https://github.com/collectd/collectd @@ -1172,6 +1175,8 @@ statistics are disabled. They can be enabled with ``ExtraStats`` option. For more information on the plugin parameters, please see: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod +.. _install-collectd-as-a-service: + Installing collectd as a service ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ **NOTE**: In an OPNFV installation, collectd is installed and configured as a @@ -1208,9 +1213,9 @@ Reload Additional useful plugins ^^^^^^^^^^^^^^^^^^^^^^^^^^ -* **Exec Plugin** : Can be used to show you when notifications are being - generated by calling a bash script that dumps notifications to file. (handy - for debug). Modify /opt/collectd/etc/collectd.conf: +**Exec Plugin** : Can be used to show you when notifications are being +generated by calling a bash script that dumps notifications to file. (handy +for debug). Modify /opt/collectd/etc/collectd.conf: .. code:: bash diff --git a/docs/release/userguide/index.rst b/docs/release/userguide/index.rst index a6ec261f..e880f3a9 100644 --- a/docs/release/userguide/index.rst +++ b/docs/release/userguide/index.rst @@ -19,6 +19,7 @@ OPNFV Barometer User Guide feature.userguide collectd.ves.userguide.rst + docker.userguide.rst .. The feature.userguide.rst file should contain the text for this document .. additional documents can be added to this directory and added in the right order .. to this file as a list below. diff --git a/docs/release/userguide/parse-work-flow.png b/docs/release/userguide/parse-work-flow.png Binary files differnew file mode 100644 index 00000000..59cb70a8 --- /dev/null +++ b/docs/release/userguide/parse-work-flow.png diff --git a/docs/release/userguide/value-item-parse-workflow.png b/docs/release/userguide/value-item-parse-workflow.png Binary files differnew file mode 100644 index 00000000..0aff3191 --- /dev/null +++ b/docs/release/userguide/value-item-parse-workflow.png |
