###############################################################################
# Copyright (c) 2017 Koren Lev (Cisco Systems), Yaron Yogev (Cisco Systems)   #
# and others                                                                  #
#                                                                             #
# All rights reserved. This program and the accompanying materials            #
# are made available under the terms of the Apache License, Version 2.0       #
# which accompanies this distribution, and is available at                    #
# http://www.apache.org/licenses/LICENSE-2.0                                  #
###############################################################################
| Calipso.io
| Monitoring Guide

|image0|

Project “Calipso” tries to illuminate complex virtual networking with
real time operational state visibility for large and highly distributed
Virtual Infrastructure Management (VIM).

We believe that Stability is driven by accurate Visibility.

Calipso provides visible insights using smart discovery and virtual
topological representation in graphs, with monitoring per object in the
graph inventory to reduce error vectors and troubleshooting, maintenance
cycles for VIM operators and administrators.

Table of Contents

Calipso.io Monitoring Guide 1

1 Monitoring deployment options 3

1.1 Calipso monitoring provisioning 3

1.2 Calipso-sensu container 5

2 Monitoring configurations 5

2.1 Calipso monitoring configuration templates 6

2.2 Calipso Apex monitoring integration 11

Monitoring deployment options 
==============================

    Calipso uses sensu framework that includes a customized calipso
    sensu server and a set of customized clients, for details on the
    framework see:

    https://sensuapp.org/docs/latest/overview/what-is-sensu.html

    Calipso uses this framework but further configure and customizes the
    install, per:

    https://sensuapp.org/docs/1.0/installation/

    Server and clients are deployed with rabbitmq over SSL as the
    client-server transport:

    https://sensuapp.org/docs/1.0/reference/ssl.html

    Calipso then deploys customized checks and handlers as new types of
    plugins:

    http://sensu-plugins.io/plugins/

    In order to monitor all virtual components as described in
    Calipso-model document, Calipso offers two major functions:

1. Calipso scan module: customizes and automatically deploys sensu,
   config files and all needed plugins and scripts on all clients and
   the central server.

2. Calipso sensu module: listens for the results of the customized
   events and updates the inventories with state and statuses, while
   generating all related messages.

Calipso monitoring provisioning
-------------------------------

    Calipso administrator should instruct the main scanning engine
    whether or not to deploy monitoring across the VIM environment
    hosts. This is done first by the Calipso scan container that uses a
    set of “environment\_config” parameters (see calipso-model guide for
    details).

    A full scan of an environment is a mandatory pre-requisite for
    monitoring, because calipso monitoring provisioning uses inventory
    discovered data as values of several configuration items to be
    deployed onto the sensu framework.

    Here are the important parts of an “environment\_config” document
    that needs to be defined either through UI or API as a pre-requisite
    for monitoring provisioning:

    {

    **"enable\_monitoring": true**, // this must be “true” for any
    monitoring result

    **"name"** **: "my-env-with-monitoring",** // name is needed as
    usual

    **"distribution" :** // other details removed ….

    **"configuration" : [** // in configuration: a monitoring section is
    defined

    **{**

    **"name" : "Monitoring",** // name of this configuration section

    **"type" : "Sensu",** // instruct to use a sensu type of monitoring
    framework

    **"rabbitmq\_port" : 5671,** // the SSL port used by sensu clients
    to server

    **"rabbitmq\_pass" : "dummy\_pwd",** // clients-to-server rabbitmq
    password

    **"rabbitmq\_user" : "sensu",** // clients-to-server rabbitmq
    username

    **"server\_ip" : "10.0.0.1",** // the calipso sensu container ip
    address

    **"ssh\_user": "root",** // user used inside the sensu server
    (container)

    **"ssh\_password": "dummy\_pwd",** // root pwd used inside the sensu
    server

    **"ssh\_port" : 20022, //** ssh port where sensu container offers
    ssh access

    **"server\_name”: "sensu\_server", /**/ name to use for the sensu
    server api

    **"env\_type" : "production",** // a type of environment to setup
    (sensu’s ‘type’)

    **"config\_folder" : "/local\_dir/sensu\_config",** // location to
    place config files

    **"api\_port" : 4567** // sensu server api port to connect to

    **"provision" : "None" // \*\* see provision options described next
    ! \*\***

    **}, **

    **]**

    \*\* **Provision** options are:

1. **“None”:** does not deploy any monitoring configurations

2. **“DB”:** creates all necessary configuration files in the mongoDB,
   placed in “monitoring\_config” collection.

3. **“Files”:** creates all necessary configuration files in the
   mongoDB, placed in “monitoring\_config” collection and also in
   ‘config\_folder’ location on the server.

4. **“Deploy”:** Does what “Files” is doing, but also goes and deploys
   all configuration files and scripts on all servers on the
   environment.

    *This is a high level representation of the calipso monitoring
    provisioning logic:*

    |image1|

Calipso-sensu container
-----------------------

    Once sensu clients and all needed configurations and plugins are
    deployed properly, the sensu server should start receiving results
    and update the relevant inventory objects with their states and
    statuses.

    The calipso-sensu container is a pre-built sensu server customized
    for calipso design.

    The following services are exposed on the calipso-sensu container:

1. Calipso monitoring handling app running in:
   /home/scan/calipso\_prod/app/monitoring/handlers/monitor.py

2. Sensu-api service accessible on port 4567.

3. Calipso container bash access through ssh on port 20022 (see
   quickstart-guide).

4. Sensu-server service listening as rabbitmq over ssl on port 5671.

5. Rabbitmq management server on port 15672 (http://server-ip:15672/ to
   access).

6. Sensu uchiwa UI listening on port 3000 (http://server-ip:3000 to
   access).

    All the above services are maintained by calipso and customized for
    calipso’s virtual inventory monitoring design.

Monitoring configurations 
==========================

    You can access calipso-sensus container by ssh, using:

    **ssh scan@localhost -p 20022** with a default password “scan”.

    Inside the calipso-sensu container the calipso monitoring
    application is maintained at:

    **/home/scan/calipso\_prod/app/monitoring**

    SSL keys are shipped with the calipso-sensu container and maintained
    at:

    **/etc/sensu/ssl **

    Check scripts used to run a health check against monitored objects
    are maintained at:

    **/home/scan/calipso\_prod/app/monitoring/checks**

    Handling scripts to grab and handle checks results from clients are
    maintained at:

    **/home/scan/calipso\_prod/app/monitoring/handlers**

    Configuration deployment setup scripts are maintained at:

    **/home/scan/calipso\_prod/app/monitoring/setup**

Calipso monitoring configuration templates
------------------------------------------

    Calipso application maintained a customizable model for all the
    configuration files deployed onto the sensu server and clients at
    provisioning stage of its main scanning engine.

    The following configuration files are customizable:

    *Server side:*

    **redis.json**

    **transport,json**

    **rabbitmq.json**

    **api.json**

    **client.json**

    **filters.json**

    **handlers.json**

    *Clients side:*

    **transport,json**

    **rabbitmq.json**

    **client.json**

    **customized checks to integrate into client.json **

    We have defined a default set of configuration parameters in those
    files per our knowledge and best practices, so a typical user would
    not need to edit any of the above set of configuration files and
    their options. In case any of these needs customizations we offer a
    very granular model maintained in Calipso mongoDB under the
    **“monitoring\_config\_templates”** collection.

    Here is an example of such template, and its options:

    **"type" : "client.json",** // this will be used for building
    client.json files

    **"order" : "1",** // if several of these templates are configured,
    take a priority to use

    **"side" : "client", //**\ client.json file deployed onto the client
    side (env hosts)

    **"config" : {** // the actual configuration txt follows …

    **"client" : {**

    **"address" : "{client\_name}",** // take the name of host from scan
    inventory

    **"subscriptions" : [** //a future option

    **], **

    **"environment" : "{env\_name}",** // name of the VIM environment
    for logs

    **"name" : "{client\_name}" //** take the name of host from scan
    inventory

    **}, **

    **"api" : {**

    **"host" : "{server\_ip}", //** take the server\_ip of
    environment\_config

    **"port" : NumberInt(4567)** // use this port for interfacing with
    the server

    **}**

    **}, **

    **"monitoring\_system" : "sensu"**

    **}**

    The above is just a simple example, login to mongoDB and check
    “monitoring\_config\_templates” collection for recent information on
    deployment files and their configuration options.

    The results of the monitoring provisioning are placed, by default,
    in mongoDB at the collection – “\ **monitoring\_config**\ ”, locally
    on the calipso-scan container at **/local\_dir/sensu\_config** and
    finally on the server (calipso-sensu container) and on all the hosts
    (clients). Here is an example of the resulted client.json file on
    one of the environment hosts (example deployment from real VIM
    environment):

    **{**

    **"api": {**

    **"host": "korlev-calipso-dev.cisco.com",**

    **"port": 4567**

    **},**

    **"checks": {**

    **"host\_pnic\_eno16777728-00---..58..---50---..58..---56---..58..---ac---..58..---e8---..58..---97":
    {**

    **"command": "check\_pnic\_ovs.py eno16777728",**

    **"handlers": [**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **},**

    **"host\_pnic\_eno33554952-00---..58..---50---..58..---56---..58..---ac---..58..---c9---..58..---a2":
    {**

    **"command": "check\_pnic\_ovs.py eno33554952",**

    **"handlers": [**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **},**

    **"otep\_node-6.cisco.com-otep\_vxlan-c0a80201": {**

    **"command": "check\_ping.py -c 10 -i 0.5 -p 4f532d444e41 -w 10 -s
    256 -f 192.168.2.2 -t 192.168.2.1 -W 1%/301.11/600 -C
    10%/1020.12/2000",**

    **"handlers": [**

    **"default",**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **},**

    **"otep\_node-6.cisco.com-otep\_vxlan-c0a80203": {**

    **"command": "check\_ping.py -c 10 -i 0.5 -p 4f532d444e41 -w 10 -s
    256 -f 192.168.2.2 -t 192.168.2.3 -W 1%/301.11/600 -C
    10%/1020.12/2000",**

    **"handlers": [**

    **"default",**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **},**

    **"vedge\_bc865c43-3dc5-4940-af1d-b4be59df1bd0": {**

    **"command": "check\_vedge\_ovs.py",**

    **"handlers": [**

    **"default",**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **},**

    **"vservice\_qdhcp-6c5ddc76-fcd7-4bdd-bff4-1d08b88b96ca": {**

    **"command": "PYTHONPATH=/etc/sensu/plugins check\_vservice.py dhcp
    qdhcp-6c5ddc76-fcd7-4bdd-bff4-1d08b88b96ca",**

    **"handlers": [**

    **"default",**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **},**

    **"vservice\_qdhcp-721f9c95-3042-4840-b8a4-83968c1e92b6": {**

    **"command": "PYTHONPATH=/etc/sensu/plugins check\_vservice.py dhcp
    qdhcp-721f9c95-3042-4840-b8a4-83968c1e92b6",**

    **"handlers": [**

    **"default",**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **},**

    **"vservice\_qdhcp-cc7ea40b-bb11-4b51-8e51-1a3b7abd283d": {**

    **"command": "PYTHONPATH=/etc/sensu/plugins check\_vservice.py dhcp
    qdhcp-cc7ea40b-bb11-4b51-8e51-1a3b7abd283d",**

    **"handlers": [**

    **"default",**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **},**

    **"vservice\_qrouter-1833846f-573e-45ef-8c87-3f7df530cdbd": {**

    **"command": "PYTHONPATH=/etc/sensu/plugins check\_vservice.py
    router qrouter-1833846f-573e-45ef-8c87-3f7df530cdbd",**

    **"handlers": [**

    **"default",**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **}**

    **},**

    **"client": {**

    **"address": "Mirantis-Liberty-node-6.cisco.com",**

    **"environment": "Mirantis-Liberty",**

    **"name": "Mirantis-Liberty-node-6.cisco.com",**

    **"subscriptions": []**

    **}**

    **}**

    All sensu configuration files, keys and scripts are eventually
    deployed, both on clients and on server side at the following
    locations:

    *SSL keys:* **/etc/sensu/ssl**

    *Calipso monitoring checks*: **/etc/sensu/plugins **

    *Configuration files:* **/etc/sensu/conf.d **

    The calipso-scan container is in-charge of the actual deployment (in
    case environment\_config is configured with needed details and
    “provision” = “Deploy.

    Calipso-scan then uses the calipso-sensu as the target “sensu
    server” to deploy and all the environment pre-discovered hosts as
    “sensu clients” to deploy, all pointing their monitoring results
    back to the calipso-sensu container.

    Calipso-scan uses ssh to access all hosts (through the master-host,
    see admin-guide) and also to access calipso-sensu container on port
    20022 to upload all customized files and places them in the above
    locations.

Calipso Apex monitoring integration
-----------------------------------

    For OPNFV version ‘P’ of the calipso application, farther automation
    has been developed for ‘zero touch’ automation. Calipso has a
    built-in ‘apex-configurator’ that runs at apex install phase
    (current scenario: os-nosdn-calipso-noha) and deploys the sensu
    clients themselves with all needed configurations, per apex install
    parameters and customizes the calipso-sensu container accordingly…no
    manual UI or API steps are needed for end-to-end functionality.

.. |image0| image:: media/image1.png
   :width: 6.50000in
   :height: 4.27153in
.. |image1| image:: media/image10.png
   :width: 6.50000in
   :height: 3.62708in