summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
-rw-r--r--docker/barometer-collectd/Dockerfile2
-rw-r--r--docker/compose/influxdb-grafana/docker-compose.yml2
-rw-r--r--docker/compose/ves/docker-compose.yml5
-rw-r--r--docs/release/userguide/docker.userguide.rst299
-rw-r--r--docs/release/userguide/feature.userguide.rst29
-rw-r--r--puppet-barometer/templates/mcelog.conf.erb6
6 files changed, 281 insertions, 62 deletions
diff --git a/docker/barometer-collectd/Dockerfile b/docker/barometer-collectd/Dockerfile
index 585e5d05..895d3d08 100644
--- a/docker/barometer-collectd/Dockerfile
+++ b/docker/barometer-collectd/Dockerfile
@@ -22,3 +22,5 @@ RUN pip install -r ${openstack_plugins}/collectd-openstack-plugins/requirements.
COPY run_collectd.sh /run_collectd.sh
RUN chmod +x /run_collectd.sh
+
+ENTRYPOINT ["/run_collectd.sh"]
diff --git a/docker/compose/influxdb-grafana/docker-compose.yml b/docker/compose/influxdb-grafana/docker-compose.yml
index 7cb2e750..eee72ccb 100644
--- a/docker/compose/influxdb-grafana/docker-compose.yml
+++ b/docker/compose/influxdb-grafana/docker-compose.yml
@@ -9,6 +9,7 @@ services:
ports:
- 8086:8086
- 25826:25826
+ container_name: bar-influxdb
grafana:
image: opnfv/barometer-grafana
network_mode: host
@@ -16,3 +17,4 @@ services:
- /var/lib/grafana
ports:
- 3000:3000
+ container_name: bar-grafana
diff --git a/docker/compose/ves/docker-compose.yml b/docker/compose/ves/docker-compose.yml
index b4729585..8aefdb36 100644
--- a/docker/compose/ves/docker-compose.yml
+++ b/docker/compose/ves/docker-compose.yml
@@ -2,18 +2,21 @@ version: '3'
services:
zookeeper:
- image: 31z4/zookeeper
+ image: zookeeper:3.4.11
network_mode: host
ports:
- 2181:2181
+ container_name: zookeeper
kafka:
image: opnfv/barometer-kafka
network_mode: host
ports:
- 9092:9092
+ container_name: bar-kafka
ves:
image: opnfv/barometer-ves
network_mode: host
command: host.yaml
+ container_name: bar-ves
diff --git a/docs/release/userguide/docker.userguide.rst b/docs/release/userguide/docker.userguide.rst
index 2a78bc95..b422d949 100644
--- a/docs/release/userguide/docker.userguide.rst
+++ b/docs/release/userguide/docker.userguide.rst
@@ -10,12 +10,28 @@ OPNFV Barometer Docker User Guide
:depth: 3
:local:
-The intention of this user guide is to outline how to install and test the
-Barometer projects Collectd, Influxdb and Grafana docker images which can be built from the Dockerfiles
-available within the barometer repository.
+The intention of this user guide is to outline how to install and test the Barometer project's
+docker images. The `OPNFV docker hub <https://hub.docker.com/u/opnfv/?page=1>`_ contains 5 docker
+images from the Barometer project:
+ 1. `Collectd docker image <https://hub.docker.com/r/opnfv/barometer-collectd/>`_
+ 2. `Influxdb docker image <https://hub.docker.com/r/opnfv/barometer-influxdb/>`_
+ 3. `Grafana docker image <https://hub.docker.com/r/opnfv/barometer-grafana/>`_
+ 4. `Kafka docker image <https://hub.docker.com/r/opnfv/barometer-kafka/>`_
+ 5. `VES application docker image <https://hub.docker.com/r/opnfv/barometer-ves/>`_
-Barometer docker images description
+For description of images please see section `Barometer Docker Images Description`_
+
+For steps to build and run Collectd image please see section `Build and Run Collectd Docker Image`_
+
+For steps to build and run InfluxDB and Grafana images please see section `Build and Run InfluxDB and Grafana Docker Images`_
+
+For steps to build and run VES and Kafka images please see section `Build and Run VES and Kafka Docker Images`_
+
+For overview of running VES application with Kafka please see the `VES Application User Guide
+<http://docs.opnfv.org/en/latest/submodules/barometer/docs/release/userguide/collectd.ves.userguide.html>`_
+
+Barometer Docker Images Description
-----------------------------------
.. Describe the specific features and how it is realised in the scenario in a brief manner
@@ -60,14 +76,24 @@ docker image:
* aodh plugin
* Legacy/IPMI
-InfluxDB + Grafana Images
-^^^^^^^^^^^^^^^^^^^^^^^^^
+InfluxDB + Grafana Docker Images
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The Barometer project's InfluxDB and Grafana docker images are 2 docker images that database and graph
statistics reported by the Barometer collectd docker. InfluxDB is an open-source time series database
tool which stores the data from collectd for future analysis via Grafana, which is a open-source
metrics anlytics and visualisation suite which can be accessed through any browser.
+VES + Kafka Docker Images
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The Barometer project's VES application and Kafka docker images are based on a CentOS 7 image. Kafka
+docker image has a dependancy on `Zookeeper <https://zookeeper.apache.org/>`_. Kafka must be able to
+connect and register with an instance of Zookeeper that is either running on local or remote host.
+Kafka recieves and stores metrics recieved from Collectd. VES application pulls latest metrics from Kafka
+which it normalizes into VES format for sending to a VES collector. Please see details in `VES Application User Guide
+<http://docs.opnfv.org/en/latest/submodules/barometer/docs/release/userguide/collectd.ves.userguide.html>`_
+
Installing Docker
-----------------
.. Describe the specific capabilities and usage for <XYZ> feature.
@@ -76,8 +102,8 @@ Installing Docker
On Ubuntu
^^^^^^^^^^
.. note::
- * sudo permissions are required to install docker.
- * These instructions are for Ubuntu 16.10
+ * sudo permissions are required to install docker.
+ * These instructions are for Ubuntu 16.10
To install docker:
@@ -93,8 +119,8 @@ Replace <username> above with an appropriate user name.
On CentOS
^^^^^^^^^^
.. note::
- * sudo permissions are required to install docker.
- * These instructions are for CentOS 7
+ * sudo permissions are required to install docker.
+ * These instructions are for CentOS 7
To install docker:
@@ -189,7 +215,7 @@ Or create a single file with all the proxy configurations:
Test docker installation
^^^^^^^^^^^^^^^^^^^^^^^^
.. note::
- This applies for both CentOS and Ubuntu.
+ This applies for both CentOS and Ubuntu.
.. code:: bash
@@ -222,8 +248,20 @@ To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
+Build and Run Collectd Docker Image
+-----------------------------------
+
+Download the collectd docker image
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+If you wish to use a pre-built barometer image, you can pull the barometer
+image from https://hub.docker.com/r/opnfv/barometer-collectd/
+
+.. code:: bash
+
+ $ docker pull opnfv/barometer-collectd
+
Build the collectd docker image
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: bash
@@ -252,16 +290,6 @@ Output should contain a barometer-collectd image:
centos latest 196e0ce0c9fb 4 weeks ago 197MB
hello-world latest 05a3bd381fc2 4 weeks ago 1.84kB
-Download the collectd docker image
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-If you want to use a pre-built barometer image, you can pull the barometer
-image from https://hub.docker.com/r/opnfv/barometer-collectd/
-
-.. code:: bash
-
- $ docker pull opnfv/barometer-collectd
-
-
Run the collectd docker image
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: bash
@@ -270,12 +298,12 @@ Run the collectd docker image
-v /var/run:/var/run -v /tmp:/tmp --privileged opnfv/barometer-collectd /run_collectd.sh
.. note::
- The docker collectd image contains configuration for all the collectd plugins. In the command
- above we are overriding /opt/collectd/etc/collectd.conf.d by mounting a host directory
- `pwd`/../src/collectd_sample_configs that contains only the sample configurations we are interested
- in running. *It's important to do this if you don't have DPDK, or RDT installed on the host*.
- Sample configurations can be found at:
- https://github.com/opnfv/barometer/tree/master/src/collectd/collectd_sample_configs
+ The docker collectd image contains configuration for all the collectd plugins. In the command
+ above we are overriding /opt/collectd/etc/collectd.conf.d by mounting a host directory
+ `pwd`/../src/collectd_sample_configs that contains only the sample configurations we are interested
+ in running. *It's important to do this if you don't have DPDK, or RDT installed on the host*.
+ Sample configurations can be found at:
+ https://github.com/opnfv/barometer/tree/master/src/collectd/collectd_sample_configs
To make some changes when the container is running run:
@@ -289,8 +317,8 @@ Check your docker image is running
sudo docker ps
-Build the influxdb + Grafana docker images
-------------------------------------------
+Build and Run InfluxDB and Grafana docker images
+------------------------------------------------
Overview
^^^^^^^^
@@ -314,14 +342,14 @@ Incase where a folder is mounted to this volume only files included in this fold
inside barometer-grafana. To ensure all default files are also loaded please ensure they are included in
volume folder been mounted. Appropriate example are given in section `Run the Grafana docker image`_
-Download the InfluxDB and Grafana docker image
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Download the InfluxDB and Grafana docker images
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you wish to use pre-built barometer project's influxdb and grafana images, you can pull the
images from https://hub.docker.com/r/opnfv/barometer-influxdb/ and https://hub.docker.com/r/opnfv/barometer-grafana/
.. note::
- If your preference is to build images locally please see sections `Build the InfluxDB Image`_ and
- `Build the Grafana Image`_
+ If your preference is to build images locally please see sections `Build InfluxDB Docker Image`_ and
+ `Build Grafana Docker Image`_
.. code:: bash
@@ -329,15 +357,15 @@ images from https://hub.docker.com/r/opnfv/barometer-influxdb/ and https://hub.d
$ docker pull opnfv/barometer-grafana
.. note::
- If you have pulled the pre-built barometer-influxdb and barometer-grafana images there is no
- requirement to complete steps outlined in sections `Build the InfluxDB Image`_ and
- `Build the Grafana Image`_ and you can proceed directly to section
- `Run the Influxdb and Grafana Images`_ If you wish to run the barometer-influxdb and
- barometer-grafana images via Docker Compose proceed directly to section
- `Docker Compose`_.
-
-Build the InfluxDB Image
-^^^^^^^^^^^^^^^^^^^^^^^^^
+ If you have pulled the pre-built barometer-influxdb and barometer-grafana images there is no
+ requirement to complete steps outlined in sections `Build InfluxDB Docker Image`_ and
+ `Build Grafana Docker Image`_ and you can proceed directly to section
+ `Run the Influxdb and Grafana Images`_ If you wish to run the barometer-influxdb and
+ barometer-grafana images via Docker Compose proceed directly to section
+ `Docker Compose`_.
+
+Build InfluxDB docker image
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
Build influxdb image from Dockerfile
@@ -345,11 +373,11 @@ Build influxdb image from Dockerfile
$ cd barometer/docker/barometer-influxdb
$ sudo docker build -t opnfv/barometer-influxdb --build-arg http_proxy=`echo $http_proxy` \
- --build-arg https_proxy=`echo $https_proxy` -f Dockerfile .
+ --build-arg https_proxy=`echo $https_proxy` -f Dockerfile .
.. note::
- In the above mentioned ``docker build`` command, http_proxy & https_proxy arguments needs to
- be passed only if system is behind an HTTP or HTTPS proxy server.
+ In the above mentioned ``docker build`` command, http_proxy & https_proxy arguments needs to
+ be passed only if system is behind an HTTP or HTTPS proxy server.
Check the docker images:
@@ -364,8 +392,8 @@ Output should contain an influxdb image:
REPOSITORY TAG IMAGE ID CREATED SIZE
opnfv/barometer-influxdb latest 1e4623a59fe5 3 days ago 191MB
-Build the Grafana Image
-^^^^^^^^^^^^^^^^^^^^^^^
+Build Grafana docker image
+^^^^^^^^^^^^^^^^^^^^^^^^^^
Build Grafana image from Dockerfile
@@ -373,10 +401,11 @@ Build Grafana image from Dockerfile
$ cd barometer/docker/barometer-grafana
$ sudo docker build -t opnfv/barometer-grafana --build-arg http_proxy=`echo $http_proxy` \
- --build-arg https_proxy=`echo $https_proxy` -f Dockerfile .
+ --build-arg https_proxy=`echo $https_proxy` -f Dockerfile .
.. note::
- In the above mentioned ``docker build`` command, http_proxy & https_proxy arguments needs to be passed only if system is behind an HTTP or HTTPS proxy server.
+ In the above mentioned ``docker build`` command, http_proxy & https_proxy arguments needs to
+ be passed only if system is behind an HTTP or HTTPS proxy server.
Check the docker images:
@@ -394,7 +423,7 @@ Output should contain an influxdb image:
Run the Influxdb and Grafana Images
-----------------------------------
-Run the InfluxDB docker image
+Run the InfluxDB docker image
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: bash
@@ -419,13 +448,16 @@ Connecting to an influxdb instance running on local system and adding own custom
.. code:: bash
- $ sudo docker run -tid --net=host -v /var/lib/grafana:/var/lib/grafana -v ${PWD}/dashboards:/opt/grafana/dashboards -p 3000:3000 opnfv/barometer-grafana
+ $ sudo docker run -tid --net=host -v /var/lib/grafana:/var/lib/grafana -v ${PWD}/dashboards:/opt/grafana/dashboards \
+ -p 3000:3000 opnfv/barometer-grafana
-Connecting to an influxdb instance running on remote system with hostname of someserver and IP address of 192.168.121.111
+Connecting to an influxdb instance running on remote system with hostname of someserver and IP address
+of 192.168.121.111
.. code:: bash
- $ sudo docker run -tid --net=host -v /var/lib/grafana:/var/lib/grafana -p 3000:3000 -e influxdb_host=someserver --add-host someserver:192.168.121.111 opnfv/barometer-grafana
+ $ sudo docker run -tid --net=host -v /var/lib/grafana:/var/lib/grafana -p 3000:3000 -e \
+ influxdb_host=someserver --add-host someserver:192.168.121.111 opnfv/barometer-grafana
To make some changes when the container is running run:
@@ -441,6 +473,152 @@ Check your docker image is running
Connect to <host_ip>:3000 with a browser and log into grafana: admin/admin
+
+Build and Run VES and Kafka Docker Images
+------------------------------------------
+
+Download VES and Kafka docker images
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you wish to use pre-built barometer project's VES and kafka images, you can pull the
+images from https://hub.docker.com/r/opnfv/barometer-ves/ and https://hub.docker.com/r/opnfv/barometer-kafka/
+
+.. note::
+ If your preference is to build images locally please see sections `Build the Kafka Image`_ and
+ `Build VES Image`_
+
+.. code:: bash
+
+ $ docker pull opnfv/barometer-kafka
+ $ docker pull opnfv/barometer-ves
+
+.. note::
+ If you have pulled the pre-built images there is no requirement to complete steps outlined
+ in sections `Build Kafka Docker Image`_ and `Build VES Docker Image`_ and you can proceed directly to section
+ `Run Kafka Docker Image`_ If you wish to run the docker images via Docker Compose proceed directly to section `Docker Compose`_.
+
+Build Kafka docker image
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Build Kafka docker image:
+
+.. code:: bash
+
+ $ cd barometer/docker/barometer-kafka
+ $ sudo docker build -t opnfv/barometer-kafka --build-arg http_proxy=`echo $http_proxy` \
+ --build-arg https_proxy=`echo $https_proxy` -f Dockerfile .
+
+.. note::
+ In the above mentioned ``docker build`` command, http_proxy & https_proxy arguments needs
+ to be passed only if system is behind an HTTP or HTTPS proxy server.
+
+Check the docker images:
+
+.. code:: bash
+
+ $ sudo docker images
+
+Output should contain a barometer image:
+
+.. code::
+
+ REPOSITORY TAG IMAGE ID CREATED SIZE
+ opnfv/barometer-kafka latest 05f2a3edd96b 3 hours ago 1.2GB
+
+Build VES docker image
+^^^^^^^^^^^^^^^^^^^^^^
+
+Build VES application docker image:
+
+.. code:: bash
+
+ $ cd barometer/docker/barometer-ves
+ $ sudo docker build -t opnfv/barometer-ves --build-arg http_proxy=`echo $http_proxy` \
+ --build-arg https_proxy=`echo $https_proxy` -f Dockerfile .
+
+.. note::
+ In the above mentioned ``docker build`` command, http_proxy & https_proxy arguments needs
+ to be passed only if system is behind an HTTP or HTTPS proxy server.
+
+Check the docker images:
+
+.. code:: bash
+
+ $ sudo docker images
+
+Output should contain a barometer image:
+
+.. code::
+
+ REPOSITORY TAG IMAGE ID CREATED SIZE
+ opnfv/barometer-ves latest 05f2a3edd96b 3 hours ago 1.2GB
+
+Run Kafka docker image
+^^^^^^^^^^^^^^^^^^^^^^
+
+.. note::
+ Before running Kafka an instance of Zookeeper must be running for the Kafka broker to register
+ with. Zookeeper can be running locally or on a remote platform. Kafka's broker_id and address of
+ its zookeeper instance can be configured by setting values for environmental variables 'broker_id'
+ and 'zookeeper_node'. In instance where 'broker_id' and/or 'zookeeper_node' is not set the default
+ setting of broker_id=0 and zookeeper_node=localhost is used. In intance where Zookeeper is running
+ on same node as Kafka and there is a one to one relationship between Zookeeper and Kafka, default
+ setting can be used. The docker argument `add-host` adds hostname and IP address to
+ /etc/hosts file in container
+
+Run zookeeper docker image:
+
+.. code:: bash
+
+ $ sudo docker run -tid --net=host -p 2181:2181 zookeeper:3.4.11
+
+Run kafka docker image which connects with a zookeeper instance running on same node with a 1:1 relationship
+
+.. code:: bash
+
+ $ sudo docker run -tid --net=host -p 9092:9092 opnfv/barometer-kafka
+
+
+Run kafka docker image which connects with a zookeeper instance running on a node with IP address of
+192.168.121.111 using broker ID of 1
+
+.. code:: bash
+
+ $ sudo docker run -tid --net=host -p 9092:9092 --env broker_id=1 --env zookeeper_node=zookeeper --add-host \
+ zookeeper:192.168.121.111 opnfv/barometer-kafka
+
+Run VES Application docker image
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. note::
+ VES application uses configuration file ves_app_config.conf from directory
+ barometer/3rd_party/collectd-ves-app/ves_app/config/ and host.yaml file from
+ barometer/3rd_party/collectd-ves-app/ves_app/yaml/ by default. If you wish to use a custom config
+ file it should be mounted to mount point /opt/ves/config/ves_app_config.conf. To use an alternative yaml
+ file from folder barometer/3rd_party/collectd-ves-app/ves_app/yaml the name of the yaml file to use
+ should be passed as an additional command. If you wish to use a custom file the file should be
+ mounted to mount point /opt/ves/yaml/ Please see examples below
+
+Run VES docker image with default configuration
+
+.. code:: bash
+
+ $ sudo docker run -tid --net=host opnfv/barometer-ves
+
+Run VES docker image with guest.yaml files from barometer/3rd_party/collectd-ves-app/ves_app/yaml/
+
+.. code:: bash
+
+ $ sudo docker run -tid --net=host opnfv/barometer-ves guest.yaml
+
+
+Run VES docker image with using custom config and yaml files. In example below yaml/ folder cotains
+file named custom.yaml
+
+.. code:: bash
+
+ $ sudo docker run -tid --net=host -v ${PWD}/custom.config:/opt/ves/config/ves_app_config.conf \
+ -v ${PWD}/yaml/:/opt/ves/yaml/ opnfv/barometer-ves custom.yaml
+
Docker Compose
--------------
@@ -451,9 +629,9 @@ On the node where you want to run influxdb + grafana or the node where you want
zookeeper and Kafka containers together:
.. note::
- The default configuration for all these containers is to run on the localhost. If this is not
- the model you want to use then please make the appropriate configuration changes before launching
- the docker containers.
+ The default configuration for all these containers is to run on the localhost. If this is not
+ the model you want to use then please make the appropriate configuration changes before launching
+ the docker containers.
1. Start by installing docker compose
@@ -462,9 +640,9 @@ zookeeper and Kafka containers together:
$ sudo curl -L https://github.com/docker/compose/releases/download/1.17.0/docker-compose-`uname -s`-`uname -m` -o /usr/bin/docker-compose
.. note::
- Use the latest Compose release number in the download command. The above command is an example,
- and it may become out-of-date. To ensure you have the latest version, check the Compose repository
- release page on GitHub.
+ Use the latest Compose release number in the download command. The above command is an example,
+ and it may become out-of-date. To ensure you have the latest version, check the Compose repository
+ release page on GitHub.
2. Apply executable permissions to the binary:
@@ -522,3 +700,4 @@ References
.. [2] https://docs.docker.com/engine/installation/linux/docker-ce/centos/#install-using-the-repository
.. [3] https://docs.docker.com/engine/userguide/
+
diff --git a/docs/release/userguide/feature.userguide.rst b/docs/release/userguide/feature.userguide.rst
index 21b29dca..acef4232 100644
--- a/docs/release/userguide/feature.userguide.rst
+++ b/docs/release/userguide/feature.userguide.rst
@@ -376,6 +376,18 @@ Building and installing *jevents* library:
$ make
$ sudo make install
+Download the Hardware Events that are relevant to your CPU, download the appropriate
+CPU event list json file:
+
+.. code:: bash
+
+ $ wget https://raw.githubusercontent.com/andikleen/pmu-tools/master/event_download.py
+ $ python event_download.py
+
+This will download the json files to the location: $HOME/.cache/pmu-events/. If you don't want to
+download these files to the aforementioned location, set the environment variable XDG_CACHE_HOME to
+the location you want the files downloaded to.
+
Building and installing collectd:
.. code:: bash
@@ -403,6 +415,23 @@ include:
ReportSoftwareEvents true
</Plugin>
+If you want to monitor Intel CPU specific CPU events, make sure to enable the
+additional two options shown below:
+
+.. code:: bash
+
+ <Plugin intel_pmu>
+ ReportHardwareCacheEvents true
+ ReportKernelPMUEvents true
+ ReportSoftwareEvents true
+ EventList "$HOME/.cache/pmu-events/GenuineIntel-6-2D-core.json"
+ HardwareEvents "L2_RQSTS.CODE_RD_HIT,L2_RQSTS.CODE_RD_MISS" "L2_RQSTS.ALL_CODE_RD"
+ </Plugin>
+
+.. note::
+ If you set XDG_CACHE_HOME to anything other than the variable above - you will need to modify
+ the path for the EventList configuration.
+
For more information on the plugin parameters, please see:
https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod
diff --git a/puppet-barometer/templates/mcelog.conf.erb b/puppet-barometer/templates/mcelog.conf.erb
index dfe24b43..ad310fb7 100644
--- a/puppet-barometer/templates/mcelog.conf.erb
+++ b/puppet-barometer/templates/mcelog.conf.erb
@@ -15,7 +15,11 @@
<LoadPlugin mcelog>
Interval 1
</LoadPlugin>
+
<Plugin "mcelog">
- McelogClientSocket "/var/run/mcelog-client"
+ <Memory>
+ McelogClientSocket "/var/run/mcelog-client"
+ PersistentNotification false
+ </Memory>
</Plugin>