DOCS: Cleanup for Jerma Release.

This patch cleans up documentation folder.
1. Broken links are fixed.
2. LMA docs reorganized.
3. Placeholder for K8S and Openstack Support.

Remove tls_verify = False line

JIRA: VSPERF-627
Signed-off-by: Sridhar K. N. Rao <sridhar.rao@spirent.com>
Change-Id: I5ef93919755d351b79fa0bd0416d7d6baeb134a9

2 files changed, 242 insertions, 59 deletions

diff --git a/docs/lma/logs/devguide.rst b/docs/lma/logs/devguide.rst
new file mode 100644
index 00000000..7aeaad29
--- /dev/null
+++ b/docs/lma/logs/devguide.rst
@@ -0,0 +1,145 @@
+====================
+Logs Developer Guide
+====================
+
+Ansible Client-side
+-------------------
+
+Ansible File Organisation
+^^^^^^^^^^^^^^^^^^^^^^^^^
+Files Structure::
+
+    ansible-client
+    ├── ansible.cfg
+    ├── hosts
+    ├── playbooks
+    │   └── setup.yaml
+    └── roles
+        ├── clean-td-agent
+        │   └── tasks
+        │       └── main.yml
+        └── td-agent
+            ├── files
+            │   └── td-agent.conf
+            └── tasks
+                └── main.yml
+
+Summary of roles
+^^^^^^^^^^^^^^^^
+====================== ======================
+Roles                  Description
+====================== ======================
+``td-agent``           Install Td-agent & change configuration file
+``clean-td-agent``     Unistall Td-agent
+====================== ======================
+
+Configurable Parameters
+^^^^^^^^^^^^^^^^^^^^^^^
+====================================================== ====================== ======================
+File (ansible-client/roles/)                           Parameter              Description
+====================================================== ====================== ======================
+``td-agent/files/td-agent.conf``                       host                   Fluentd-server IP
+``td-agent/files/td-agent.conf``                       port                   Fluentd-Server Port
+====================================================== ====================== ======================
+
+Ansible Server-side
+-------------------
+
+Ansible File Organisation
+^^^^^^^^^^^^^^^^^^^^^^^^^
+Files Structure::
+
+      ansible-server
+      ├── ansible.cfg
+      ├── group_vars
+      │   └── all.yml
+      ├── hosts
+      ├── playbooks
+      │   └── setup.yaml
+      └── roles
+          ├── clean-logging
+          │   └── tasks
+          │       └── main.yml
+          ├── k8s-master
+          │   └── tasks
+          │       └── main.yml
+          ├── k8s-pre
+          │   └── tasks
+          │       └── main.yml
+          ├── k8s-worker
+          │   └── tasks
+          │       └── main.yml
+          ├── logging
+          │   ├── files
+          │   │   ├── elastalert
+          │   │   │   ├── ealert-conf-cm.yaml
+          │   │   │   ├── ealert-key-cm.yaml
+          │   │   │   ├── ealert-rule-cm.yaml
+          │   │   │   └── elastalert.yaml
+          │   │   ├── elasticsearch
+          │   │   │   ├── elasticsearch.yaml
+          │   │   │   └── user-secret.yaml
+          │   │   ├── fluentd
+          │   │   │   ├── fluent-cm.yaml
+          │   │   │   ├── fluent-service.yaml
+          │   │   │   └── fluent.yaml
+          │   │   ├── kibana
+          │   │   │   └── kibana.yaml
+          │   │   ├── namespace.yaml
+          │   │   ├── nginx
+          │   │   │   ├── nginx-conf-cm.yaml
+          │   │   │   ├── nginx-key-cm.yaml
+          │   │   │   ├── nginx-service.yaml
+          │   │   │   └── nginx.yaml
+          │   │   ├── persistentVolume.yaml
+          │   │   └── storageClass.yaml
+          │   └── tasks
+          │       └── main.yml
+          └── nfs
+              └── tasks
+                  └── main.yml
+
+Summary of roles
+^^^^^^^^^^^^^^^^
+====================== ======================
+Roles                  Description
+====================== ======================
+``k8s-pre``            Pre-requisite for installing K8s, like installing docker & K8s, disable swap etc.
+``k8s-master``         Reset K8s & make a master
+``k8s-worker``         Join woker nodes with token
+``logging``            EFK & elastalert setup in K8s
+``clean logging``      Remove EFK & elastalert setup from K8s
+``nfs``                Start a NFS server to store Elasticsearch data
+====================== ======================
+
+Configurable Parameters
+^^^^^^^^^^^^^^^^^^^^^^^
+========================================================================= ============================================ ======================
+File (ansible-server/roles/)                                              Parameter name                               Description
+========================================================================= ============================================ ======================
+**Role: logging**
+``logging/files/persistentVolume.yaml``                                   storage                                      Increase or Decrease Storage size of Persistent Volume size for each VM
+``logging/files/kibana/kibana.yaml``                                      version                                      To Change the Kibana Version
+``logging/files/kibana/kibana.yaml``                                      count                                        To increase or decrease the replica
+``logging/files/elasticsearch/elasticsearch.yaml``                        version                                      To Change the Elasticsearch Version
+``logging/files/elasticsearch/elasticsearch.yaml``                        nodePort                                     To Change Service Port
+``logging/files/elasticsearch/elasticsearch.yaml``                        storage                                      Increase or Decrease Storage size of Elasticsearch data for each VM
+``logging/files/elasticsearch/elasticsearch.yaml``                        nodeAffinity -> values (hostname)              In which VM Elasticsearch master or data pod will run (change the hostname to run the Elasticsearch master or data pod on a specific node)
+``logging/files/elasticsearch/user-secret.yaml``                          stringData                                   Add Elasticsearch User & its roles (`Elastic Docs <https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-users-and-roles.html#k8s_file_realm>`_)
+``logging/files/fluentd/fluent.yaml``                                     replicas                                     To increase or decrease the replica
+``logging/files/fluentd/fluent-service.yaml``                             nodePort                                     To Change Service Port
+``logging/files/fluentd/fluent-cm.yaml``                                  index_template.json -> number_of_replicas    To increase or decrease replica of data in Elasticsearch
+``logging/files/fluentd/fluent-cm.yaml``                                  fluent.conf                                  Server port & other Fluentd Configuration
+``logging/files/nginx/nginx.yaml``                                        replicas                                     To increase or decrease the replica
+``logging/files/nginx/nginx-service.yaml``                                nodePort                                     To Change Service Port
+``logging/files/nginx/nginx-key-cm.yaml``                                 kibana-access.key, kibana-access.pem         Key file for HTTPs Connection
+``logging/files/nginx/nginx-conf-cm.yaml``                                -                                            Nginx Configuration
+``logging/files/elastalert/elastalert.yaml``                              replicas                                     To increase or decrease the replica
+``logging/files/elastalert/ealert-key-cm.yaml``                           elastalert.key, elastalert.pem               Key file for HTTPs Connection
+``logging/files/elastalert/ealert-conf-cm.yaml``                          run_every                                    How often ElastAlert will query Elasticsearch
+``logging/files/elastalert/ealert-conf-cm.yaml``                          alert_time_limit                             If an alert fails for some reason, ElastAlert will retry sending the alert until this time period has elapsed
+``logging/files/elastalert/ealert-conf-cm.yaml``                          es_host, es_port                             Elasticsearch Serivce name & port in K8s
+``logging/files/elastalert/ealert-rule-cm.yaml``                          http_post_url                                Alert Receiver IP (`Elastalert Rule Config <https://elastalert.readthedocs.io/en/latest/ruletypes.html>`_)
+**Role: nfs**
+``nfs/tasks/main.yml``                                                    line                                         Path of NFS storage
+========================================================================= ============================================ ======================
diff --git a/docs/lma/logs/userguide.rst b/docs/lma/logs/userguide.rst
index b410ee6c..9b616fe7 100644
--- a/docs/lma/logs/userguide.rst
+++ b/docs/lma/logs/userguide.rst
@@ -1,14 +1,16 @@
-=================
-Table of Contents
-=================
-.. contents::
-.. section-numbering::
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Intel Corporation, AT&T, Red Hat, Spirent, Ixia  and others.
 
-Setup
-======
+.. OPNFV VSPERF Documentation master file.
+
+***************
+Logs User Guide
+***************
 
 Prerequisites
--------------------------
+=============
+
 - Require 3 VMs to setup K8s
 - ``$ sudo yum install ansible``
 - ``$ pip install openshift pyyaml kubernetes`` (required for ansible K8s module)
@@ -23,19 +25,21 @@ Prerequisites
    ====================================================================== ======================
 
 Architecture
---------------
+============
 .. image:: images/setup.png
 
 Installation - Clientside
--------------------------
+=========================
 
 Nodes
-`````
+-----
+
 - **Node1** = 10.10.120.21
 - **Node4** = 10.10.120.24
 
 How installation is done?
-`````````````````````````
+-------------------------
+
 - TD-agent installation
    ``$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh``
 - Copy the TD-agent config file in **Node1**
@@ -46,10 +50,11 @@ How installation is done?
    ``$ sudo service td-agent restart``
 
 Installation - Serverside
--------------------------
+=========================
 
 Nodes
-`````
+-----
+
 Inside Jumphost - POD12
    - **VM1** = 10.10.120.211
    - **VM2** = 10.10.120.203
@@ -57,7 +62,8 @@ Inside Jumphost - POD12
 
 
 How installation is done?
-`````````````````````````
+-------------------------
+
 **Using Ansible:**
    - **K8s**
       - **Elasticsearch:** 1 Master & 1 Data node at each VM
@@ -70,12 +76,14 @@ How installation is done?
       - ``/srv/nfs/data``
 
 How to setup?
-`````````````
+-------------
+
 - **To setup K8s cluster and EFK:** Run the ansible-playbook ``ansible/playbooks/setup.yaml``
 - **To clean everything:** Run the ansible-playbook ``ansible/playbooks/clean.yaml``
 
 Do we have HA?
-````````````````
+--------------
+
 Yes
 
 Configuration
@@ -83,33 +91,39 @@ Configuration
 
 K8s
 ---
+
 Path of all yamls (Serverside)
-````````````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 ``ansible-server/roles/logging/files/``
 
 K8s namespace
-`````````````
+^^^^^^^^^^^^^
+
 ``logging``
 
 K8s Service details
-````````````````````
+^^^^^^^^^^^^^^^^^^^
+
 ``$ kubectl get svc -n logging``
 
 Elasticsearch Configuration
 ---------------------------
 
 Elasticsearch Setup Structure
-`````````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
 .. image:: images/elasticsearch.png
 
 Elasticsearch service details
-`````````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
 | **Service Name:** ``logging-es-http``
 | **Service Port:** ``9200``
 | **Service Type:** ``ClusterIP``
 
 How to get elasticsearch default username & password?
-`````````````````````````````````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
 - User1 (custom user):
     | **Username:** ``elasticsearch``
     | **Password:** ``password123``
@@ -120,7 +134,8 @@ How to get elasticsearch default username & password?
     | ``$ echo $PASSWORD``
 
 How to increase replica of any index?
-````````````````````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
 | $ curl -k -u "elasticsearch:password123" -H 'Content-Type: application/json' -XPUT  "https://10.10.120.211:9200/indexname*/_settings" -d '
 | {
 |   "index" : {
@@ -128,51 +143,60 @@ How to increase replica of any index?
 | }'
 
 Index Life
-```````````
+^^^^^^^^^^
 **30 Days**
 
 Kibana Configuration
 --------------------
 
 Kibana Service details
-````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^
+
 | **Service Name:** ``logging-kb-http``
 | **Service Port:** ``5601``
 | **Service Type:** ``ClusterIP``
 
 Nginx Configuration
---------------------
+-------------------
+
 IP
-````
-https://10.10.120.211:32000
+^^
+
+The IP address with https. Ex: "10.10.120.211:32000"
 
 Nginx Setup Structure
-`````````````````````
+^^^^^^^^^^^^^^^^^^^^^
+
 .. image:: images/nginx.png
 
 Ngnix Service details
-`````````````````````
+^^^^^^^^^^^^^^^^^^^^^
+
 | **Service Name:** ``nginx``
 | **Service Port:** ``32000``
 | **Service Type:** ``NodePort``
 
 Why NGINX is used?
-```````````````````
+^^^^^^^^^^^^^^^^^^
+
 `Securing ELK using Nginx <https://logz.io/blog/securing-elk-nginx/>`_
 
 Nginx Configuration
-````````````````````
+^^^^^^^^^^^^^^^^^^^
+
 **Path:** ``ansible-server/roles/logging/files/nginx/nginx-conf-cm.yaml``
 
 Fluentd Configuration - Clientside (Td-agent)
 ---------------------------------------------
 
 Fluentd Setup Structure
-````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^
+
 .. image:: images/fluentd-cs.png
 
 Log collection paths
-`````````````````````
+^^^^^^^^^^^^^^^^^^^^
+
 - ``/tmp/result*/*.log``
 - ``/tmp/result*/*.dat``
 - ``/tmp/result*/*.csv``
@@ -181,21 +205,25 @@ Log collection paths
 - ``/var/log/sriovdp/*.log.*``
 - ``/var/log/pods/**/*.log``
 
-Logs sends to
-`````````````
+Logs sent to
+^^^^^^^^^^^^
+
 Another fluentd instance of K8s cluster (K8s Master: 10.10.120.211) at Jumphost.
 
 Td-agent logs
-`````````````
+^^^^^^^^^^^^^
+
 Path of td-agent logs: ``/var/log/td-agent/td-agent.log``
 
 Td-agent configuration
-````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^
+
 | Path of conf file: ``/etc/td-agent/td-agent.conf``
 | **If any changes is made in td-agent.conf then restart the td-agent service,** ``$ sudo service td-agent restart``
 
 Config Description
-````````````````````
+^^^^^^^^^^^^^^^^^^
+
 - Get the logs from collection path
 - | Convert to this format
   | {
@@ -210,21 +238,24 @@ Fluentd Configuration - Serverside
 ----------------------------------
 
 Fluentd Setup Structure
-````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^
+
 .. image:: images/fluentd-ss.png
 
 Fluentd Service details
-````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^
+
 | **Service Name:** ``fluentd``
 | **Service Port:** ``32224``
 | **Service Type:** ``NodePort``
 
-Logs sends to
-`````````````
-Elasticsearch service (https://logging-es-http:9200)
+Logs sent to
+^^^^^^^^^^^^
+Elasticsearch service (Example: logging-es-http at port 9200)
 
 Config Description
-````````````````````
+^^^^^^^^^^^^^^^^^^
+
 - **Step 1**
    - Get the logs from Node1 & Node4
 - **Step 2**
@@ -264,10 +295,11 @@ Config Description
    ================================ ======================
 
 Elastalert
-----------
+==========
 
 Send alert if
-``````````````
+-------------
+
 - Blacklist
     - "Failed to run test"
     - "Failed to execute in '30' seconds"
@@ -283,7 +315,8 @@ Send alert if
     - vswitch_duration > 3 sec
 
 How to configure alert?
-````````````````````````
+-----------------------
+
 - Add your rule in ``ansible/roles/logging/files/elastalert/ealert-rule-cm.yaml`` (`Elastalert Rule Config <https://elastalert.readthedocs.io/en/latest/ruletypes.html>`_)
     | name: anything
     | type: <check-above-link> #The RuleType to use
@@ -291,12 +324,13 @@ How to configure alert?
     | realert:
     |   minutes: 0    #to get alert for all cases after each interval
     | alert: post #To send alert as HTTP POST
-    | http_post_url: "http://url"
+    | http_post_url: # Provide URL
 
 - Mount this file to elastalert pod in ``ansible/roles/logging/files/elastalert/elastalert.yaml``.
 
 Alert Format
-````````````
+------------
+
 {"type": "pattern-match", "label": "failed", "index": "node4-20200815", "log": "error-log-line", "log-path": "/tmp/result/file.log", "reson": "error-message" }
 
 Data Management
@@ -305,37 +339,41 @@ Data Management
 Elasticsearch
 -------------
 
+Q&As
+^^^^
+
 Where data is stored now?
-`````````````````````````
 Data is stored in NFS server with 1 replica of each index (default). Path of data are following:
+  
   - ``/srv/nfs/data (VM1)``
   - ``/srv/nfs/data (VM2)``
   - ``/srv/nfs/data (VM3)``
   - ``/srv/nfs/master (VM1)``
   - ``/srv/nfs/master (VM2)``
   - ``/srv/nfs/master (VM3)``
-If user wants to change from NFS to local storage
-``````````````````````````````````````````````````
+
+If user wants to change from NFS to local storage, can he do it?
 Yes, user can do this, need to configure persistent volume. (``ansible-server/roles/logging/files/persistentVolume.yaml``)
 
 Do we have backup of data?
-````````````````````````````
-1 replica of each index
+Yes. 1 replica of each index
 
 When K8s restart, the data is still accessible?
-`````````````````````````````````````````````````````
 Yes (If data is not deleted from /srv/nfs/data)
 
 Troubleshooting
 ===============
+
 If no logs receiving in Elasticsearch
---------------------------------------
+-------------------------------------
+
 - Check IP & port of server-fluentd in client config.
 - Check client-fluentd logs, ``$ sudo tail -f /var/log/td-agent/td-agent.log``
 - Check server-fluentd logs, ``$ sudo kubectl logs -n logging <fluentd-pod-name>``
 
 If no notification received
 ---------------------------
+
 - Search your "log" in Elasticsearch.
 - Check config of elastalert
 - Check IP of alert-receiver
@@ -345,4 +383,4 @@ Reference
 - `Elastic cloud on K8s <https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html>`_
 - `HA Elasticsearch on K8s <https://www.elastic.co/blog/high-availability-elasticsearch-on-kubernetes-with-eck-and-gke>`_
 - `Fluentd Configuration <https://docs.fluentd.org/configuration/config-file>`_
-- `Elastalert Rule Config <https://elastalert.readthedocs.io/en/latest/ruletypes.html>`_
\ No newline at end of file
+- `Elastalert Rule Config <https://elastalert.readthedocs.io/en/latest/ruletypes.html>`_