summaryrefslogtreecommitdiffstats
path: root/docs/lma/logs
diff options
context:
space:
mode:
Diffstat (limited to 'docs/lma/logs')
-rw-r--r--docs/lma/logs/devguide.rst145
-rw-r--r--docs/lma/logs/userguide.rst156
2 files changed, 242 insertions, 59 deletions
diff --git a/docs/lma/logs/devguide.rst b/docs/lma/logs/devguide.rst
new file mode 100644
index 00000000..7aeaad29
--- /dev/null
+++ b/docs/lma/logs/devguide.rst
@@ -0,0 +1,145 @@
+====================
+Logs Developer Guide
+====================
+
+Ansible Client-side
+-------------------
+
+Ansible File Organisation
+^^^^^^^^^^^^^^^^^^^^^^^^^
+Files Structure::
+
+ ansible-client
+ ├── ansible.cfg
+ ├── hosts
+ ├── playbooks
+ │ └── setup.yaml
+ └── roles
+ ├── clean-td-agent
+ │ └── tasks
+ │ └── main.yml
+ └── td-agent
+ ├── files
+ │ └── td-agent.conf
+ └── tasks
+ └── main.yml
+
+Summary of roles
+^^^^^^^^^^^^^^^^
+====================== ======================
+Roles Description
+====================== ======================
+``td-agent`` Install Td-agent & change configuration file
+``clean-td-agent`` Unistall Td-agent
+====================== ======================
+
+Configurable Parameters
+^^^^^^^^^^^^^^^^^^^^^^^
+====================================================== ====================== ======================
+File (ansible-client/roles/) Parameter Description
+====================================================== ====================== ======================
+``td-agent/files/td-agent.conf`` host Fluentd-server IP
+``td-agent/files/td-agent.conf`` port Fluentd-Server Port
+====================================================== ====================== ======================
+
+Ansible Server-side
+-------------------
+
+Ansible File Organisation
+^^^^^^^^^^^^^^^^^^^^^^^^^
+Files Structure::
+
+ ansible-server
+ ├── ansible.cfg
+ ├── group_vars
+ │ └── all.yml
+ ├── hosts
+ ├── playbooks
+ │ └── setup.yaml
+ └── roles
+ ├── clean-logging
+ │ └── tasks
+ │ └── main.yml
+ ├── k8s-master
+ │ └── tasks
+ │ └── main.yml
+ ├── k8s-pre
+ │ └── tasks
+ │ └── main.yml
+ ├── k8s-worker
+ │ └── tasks
+ │ └── main.yml
+ ├── logging
+ │ ├── files
+ │ │ ├── elastalert
+ │ │ │ ├── ealert-conf-cm.yaml
+ │ │ │ ├── ealert-key-cm.yaml
+ │ │ │ ├── ealert-rule-cm.yaml
+ │ │ │ └── elastalert.yaml
+ │ │ ├── elasticsearch
+ │ │ │ ├── elasticsearch.yaml
+ │ │ │ └── user-secret.yaml
+ │ │ ├── fluentd
+ │ │ │ ├── fluent-cm.yaml
+ │ │ │ ├── fluent-service.yaml
+ │ │ │ └── fluent.yaml
+ │ │ ├── kibana
+ │ │ │ └── kibana.yaml
+ │ │ ├── namespace.yaml
+ │ │ ├── nginx
+ │ │ │ ├── nginx-conf-cm.yaml
+ │ │ │ ├── nginx-key-cm.yaml
+ │ │ │ ├── nginx-service.yaml
+ │ │ │ └── nginx.yaml
+ │ │ ├── persistentVolume.yaml
+ │ │ └── storageClass.yaml
+ │ └── tasks
+ │ └── main.yml
+ └── nfs
+ └── tasks
+ └── main.yml
+
+Summary of roles
+^^^^^^^^^^^^^^^^
+====================== ======================
+Roles Description
+====================== ======================
+``k8s-pre`` Pre-requisite for installing K8s, like installing docker & K8s, disable swap etc.
+``k8s-master`` Reset K8s & make a master
+``k8s-worker`` Join woker nodes with token
+``logging`` EFK & elastalert setup in K8s
+``clean logging`` Remove EFK & elastalert setup from K8s
+``nfs`` Start a NFS server to store Elasticsearch data
+====================== ======================
+
+Configurable Parameters
+^^^^^^^^^^^^^^^^^^^^^^^
+========================================================================= ============================================ ======================
+File (ansible-server/roles/) Parameter name Description
+========================================================================= ============================================ ======================
+**Role: logging**
+``logging/files/persistentVolume.yaml`` storage Increase or Decrease Storage size of Persistent Volume size for each VM
+``logging/files/kibana/kibana.yaml`` version To Change the Kibana Version
+``logging/files/kibana/kibana.yaml`` count To increase or decrease the replica
+``logging/files/elasticsearch/elasticsearch.yaml`` version To Change the Elasticsearch Version
+``logging/files/elasticsearch/elasticsearch.yaml`` nodePort To Change Service Port
+``logging/files/elasticsearch/elasticsearch.yaml`` storage Increase or Decrease Storage size of Elasticsearch data for each VM
+``logging/files/elasticsearch/elasticsearch.yaml`` nodeAffinity -> values (hostname) In which VM Elasticsearch master or data pod will run (change the hostname to run the Elasticsearch master or data pod on a specific node)
+``logging/files/elasticsearch/user-secret.yaml`` stringData Add Elasticsearch User & its roles (`Elastic Docs <https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-users-and-roles.html#k8s_file_realm>`_)
+``logging/files/fluentd/fluent.yaml`` replicas To increase or decrease the replica
+``logging/files/fluentd/fluent-service.yaml`` nodePort To Change Service Port
+``logging/files/fluentd/fluent-cm.yaml`` index_template.json -> number_of_replicas To increase or decrease replica of data in Elasticsearch
+``logging/files/fluentd/fluent-cm.yaml`` fluent.conf Server port & other Fluentd Configuration
+``logging/files/nginx/nginx.yaml`` replicas To increase or decrease the replica
+``logging/files/nginx/nginx-service.yaml`` nodePort To Change Service Port
+``logging/files/nginx/nginx-key-cm.yaml`` kibana-access.key, kibana-access.pem Key file for HTTPs Connection
+``logging/files/nginx/nginx-conf-cm.yaml`` - Nginx Configuration
+``logging/files/elastalert/elastalert.yaml`` replicas To increase or decrease the replica
+``logging/files/elastalert/ealert-key-cm.yaml`` elastalert.key, elastalert.pem Key file for HTTPs Connection
+``logging/files/elastalert/ealert-conf-cm.yaml`` run_every How often ElastAlert will query Elasticsearch
+``logging/files/elastalert/ealert-conf-cm.yaml`` alert_time_limit If an alert fails for some reason, ElastAlert will retry sending the alert until this time period has elapsed
+``logging/files/elastalert/ealert-conf-cm.yaml`` es_host, es_port Elasticsearch Serivce name & port in K8s
+``logging/files/elastalert/ealert-rule-cm.yaml`` http_post_url Alert Receiver IP (`Elastalert Rule Config <https://elastalert.readthedocs.io/en/latest/ruletypes.html>`_)
+**Role: nfs**
+``nfs/tasks/main.yml`` line Path of NFS storage
+========================================================================= ============================================ ======================
diff --git a/docs/lma/logs/userguide.rst b/docs/lma/logs/userguide.rst
index b410ee6c..9b616fe7 100644
--- a/docs/lma/logs/userguide.rst
+++ b/docs/lma/logs/userguide.rst
@@ -1,14 +1,16 @@
-=================
-Table of Contents
-=================
-.. contents::
-.. section-numbering::
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Intel Corporation, AT&T, Red Hat, Spirent, Ixia and others.
-Setup
-======
+.. OPNFV VSPERF Documentation master file.
+
+***************
+Logs User Guide
+***************
Prerequisites
--------------------------
+=============
+
- Require 3 VMs to setup K8s
- ``$ sudo yum install ansible``
- ``$ pip install openshift pyyaml kubernetes`` (required for ansible K8s module)
@@ -23,19 +25,21 @@ Prerequisites
====================================================================== ======================
Architecture
---------------
+============
.. image:: images/setup.png
Installation - Clientside
--------------------------
+=========================
Nodes
-`````
+-----
+
- **Node1** = 10.10.120.21
- **Node4** = 10.10.120.24
How installation is done?
-`````````````````````````
+-------------------------
+
- TD-agent installation
``$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh``
- Copy the TD-agent config file in **Node1**
@@ -46,10 +50,11 @@ How installation is done?
``$ sudo service td-agent restart``
Installation - Serverside
--------------------------
+=========================
Nodes
-`````
+-----
+
Inside Jumphost - POD12
- **VM1** = 10.10.120.211
- **VM2** = 10.10.120.203
@@ -57,7 +62,8 @@ Inside Jumphost - POD12
How installation is done?
-`````````````````````````
+-------------------------
+
**Using Ansible:**
- **K8s**
- **Elasticsearch:** 1 Master & 1 Data node at each VM
@@ -70,12 +76,14 @@ How installation is done?
- ``/srv/nfs/data``
How to setup?
-`````````````
+-------------
+
- **To setup K8s cluster and EFK:** Run the ansible-playbook ``ansible/playbooks/setup.yaml``
- **To clean everything:** Run the ansible-playbook ``ansible/playbooks/clean.yaml``
Do we have HA?
-````````````````
+--------------
+
Yes
Configuration
@@ -83,33 +91,39 @@ Configuration
K8s
---
+
Path of all yamls (Serverside)
-````````````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
``ansible-server/roles/logging/files/``
K8s namespace
-`````````````
+^^^^^^^^^^^^^
+
``logging``
K8s Service details
-````````````````````
+^^^^^^^^^^^^^^^^^^^
+
``$ kubectl get svc -n logging``
Elasticsearch Configuration
---------------------------
Elasticsearch Setup Structure
-`````````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
.. image:: images/elasticsearch.png
Elasticsearch service details
-`````````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
| **Service Name:** ``logging-es-http``
| **Service Port:** ``9200``
| **Service Type:** ``ClusterIP``
How to get elasticsearch default username & password?
-`````````````````````````````````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
- User1 (custom user):
| **Username:** ``elasticsearch``
| **Password:** ``password123``
@@ -120,7 +134,8 @@ How to get elasticsearch default username & password?
| ``$ echo $PASSWORD``
How to increase replica of any index?
-````````````````````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
| $ curl -k -u "elasticsearch:password123" -H 'Content-Type: application/json' -XPUT "https://10.10.120.211:9200/indexname*/_settings" -d '
| {
| "index" : {
@@ -128,51 +143,60 @@ How to increase replica of any index?
| }'
Index Life
-```````````
+^^^^^^^^^^
**30 Days**
Kibana Configuration
--------------------
Kibana Service details
-````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^
+
| **Service Name:** ``logging-kb-http``
| **Service Port:** ``5601``
| **Service Type:** ``ClusterIP``
Nginx Configuration
---------------------
+-------------------
+
IP
-````
-https://10.10.120.211:32000
+^^
+
+The IP address with https. Ex: "10.10.120.211:32000"
Nginx Setup Structure
-`````````````````````
+^^^^^^^^^^^^^^^^^^^^^
+
.. image:: images/nginx.png
Ngnix Service details
-`````````````````````
+^^^^^^^^^^^^^^^^^^^^^
+
| **Service Name:** ``nginx``
| **Service Port:** ``32000``
| **Service Type:** ``NodePort``
Why NGINX is used?
-```````````````````
+^^^^^^^^^^^^^^^^^^
+
`Securing ELK using Nginx <https://logz.io/blog/securing-elk-nginx/>`_
Nginx Configuration
-````````````````````
+^^^^^^^^^^^^^^^^^^^
+
**Path:** ``ansible-server/roles/logging/files/nginx/nginx-conf-cm.yaml``
Fluentd Configuration - Clientside (Td-agent)
---------------------------------------------
Fluentd Setup Structure
-````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^
+
.. image:: images/fluentd-cs.png
Log collection paths
-`````````````````````
+^^^^^^^^^^^^^^^^^^^^
+
- ``/tmp/result*/*.log``
- ``/tmp/result*/*.dat``
- ``/tmp/result*/*.csv``
@@ -181,21 +205,25 @@ Log collection paths
- ``/var/log/sriovdp/*.log.*``
- ``/var/log/pods/**/*.log``
-Logs sends to
-`````````````
+Logs sent to
+^^^^^^^^^^^^
+
Another fluentd instance of K8s cluster (K8s Master: 10.10.120.211) at Jumphost.
Td-agent logs
-`````````````
+^^^^^^^^^^^^^
+
Path of td-agent logs: ``/var/log/td-agent/td-agent.log``
Td-agent configuration
-````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^
+
| Path of conf file: ``/etc/td-agent/td-agent.conf``
| **If any changes is made in td-agent.conf then restart the td-agent service,** ``$ sudo service td-agent restart``
Config Description
-````````````````````
+^^^^^^^^^^^^^^^^^^
+
- Get the logs from collection path
- | Convert to this format
| {
@@ -210,21 +238,24 @@ Fluentd Configuration - Serverside
----------------------------------
Fluentd Setup Structure
-````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^
+
.. image:: images/fluentd-ss.png
Fluentd Service details
-````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^
+
| **Service Name:** ``fluentd``
| **Service Port:** ``32224``
| **Service Type:** ``NodePort``
-Logs sends to
-`````````````
-Elasticsearch service (https://logging-es-http:9200)
+Logs sent to
+^^^^^^^^^^^^
+Elasticsearch service (Example: logging-es-http at port 9200)
Config Description
-````````````````````
+^^^^^^^^^^^^^^^^^^
+
- **Step 1**
- Get the logs from Node1 & Node4
- **Step 2**
@@ -264,10 +295,11 @@ Config Description
================================ ======================
Elastalert
-----------
+==========
Send alert if
-``````````````
+-------------
+
- Blacklist
- "Failed to run test"
- "Failed to execute in '30' seconds"
@@ -283,7 +315,8 @@ Send alert if
- vswitch_duration > 3 sec
How to configure alert?
-````````````````````````
+-----------------------
+
- Add your rule in ``ansible/roles/logging/files/elastalert/ealert-rule-cm.yaml`` (`Elastalert Rule Config <https://elastalert.readthedocs.io/en/latest/ruletypes.html>`_)
| name: anything
| type: <check-above-link> #The RuleType to use
@@ -291,12 +324,13 @@ How to configure alert?
| realert:
| minutes: 0 #to get alert for all cases after each interval
| alert: post #To send alert as HTTP POST
- | http_post_url: "http://url"
+ | http_post_url: # Provide URL
- Mount this file to elastalert pod in ``ansible/roles/logging/files/elastalert/elastalert.yaml``.
Alert Format
-````````````
+------------
+
{"type": "pattern-match", "label": "failed", "index": "node4-20200815", "log": "error-log-line", "log-path": "/tmp/result/file.log", "reson": "error-message" }
Data Management
@@ -305,37 +339,41 @@ Data Management
Elasticsearch
-------------
+Q&As
+^^^^
+
Where data is stored now?
-`````````````````````````
Data is stored in NFS server with 1 replica of each index (default). Path of data are following:
+
- ``/srv/nfs/data (VM1)``
- ``/srv/nfs/data (VM2)``
- ``/srv/nfs/data (VM3)``
- ``/srv/nfs/master (VM1)``
- ``/srv/nfs/master (VM2)``
- ``/srv/nfs/master (VM3)``
-If user wants to change from NFS to local storage
-``````````````````````````````````````````````````
+
+If user wants to change from NFS to local storage, can he do it?
Yes, user can do this, need to configure persistent volume. (``ansible-server/roles/logging/files/persistentVolume.yaml``)
Do we have backup of data?
-````````````````````````````
-1 replica of each index
+Yes. 1 replica of each index
When K8s restart, the data is still accessible?
-`````````````````````````````````````````````````````
Yes (If data is not deleted from /srv/nfs/data)
Troubleshooting
===============
+
If no logs receiving in Elasticsearch
---------------------------------------
+-------------------------------------
+
- Check IP & port of server-fluentd in client config.
- Check client-fluentd logs, ``$ sudo tail -f /var/log/td-agent/td-agent.log``
- Check server-fluentd logs, ``$ sudo kubectl logs -n logging <fluentd-pod-name>``
If no notification received
---------------------------
+
- Search your "log" in Elasticsearch.
- Check config of elastalert
- Check IP of alert-receiver
@@ -345,4 +383,4 @@ Reference
- `Elastic cloud on K8s <https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html>`_
- `HA Elasticsearch on K8s <https://www.elastic.co/blog/high-availability-elasticsearch-on-kubernetes-with-eck-and-gke>`_
- `Fluentd Configuration <https://docs.fluentd.org/configuration/config-file>`_
-- `Elastalert Rule Config <https://elastalert.readthedocs.io/en/latest/ruletypes.html>`_ \ No newline at end of file
+- `Elastalert Rule Config <https://elastalert.readthedocs.io/en/latest/ruletypes.html>`_