| Calipso.io
| Objects Model
Copyright (c) 2017 Koren Lev (Cisco Systems), Yaron Yogev (Cisco Systems) and others
All rights reserved. This program and the accompanying materials
are made available under the terms of the Apache License, Version 2.0
which accompanies this distribution, and is available at
http://www.apache.org/licenses/LICENSE-2.0
|image0|
Project “Calipso” tries to illuminate complex virtual networking with
real time operational state visibility for large and highly distributed
Virtual Infrastructure Management (VIM).
Calipso provides visible insights using smart discovery and virtual
topological representation in graphs, with monitoring per object in the
graph inventory to reduce error vectors and troubleshooting, maintenance
cycles for VIM operators and administrators.
Calipso model, described in this document, was *built for
multi-environment and many VIM variances*, the model was tested
successfully (as of Aug 27\ :sup:`th`) against 60 different VIM
variances (Distributions, Versions, Networking Drivers and Types).
Table of Contents
Calipso.io Objects Model 1
1 Environments config 4
2 Inventory objects 6
2.1 Host 6
2.2 physical NIC (pNIC) 7
2.3 Bond 7
2.4 Instance 7
2.5 virtual Service (vService) 7
2.6 Network 7
2.7 virtual NIC (vNIC) 7
2.8 Port 8
2.9 virtual Connector (vConnector) 8
2.10 virtual Edge (vEdge) 8
2.11 Overlay-Tunnel-Endpoint (OTEP) 8
2.12 Network\_segment 8
2.13 Network\_Agent 8
2.14 Looking up Calipso objects details 9
3 Link Objects 10
3.1 Link types 11
4 Clique objects 11
4.1 Clique types 11
5 Supported Environments 12
6 System collections 14
6.1 Attributes\_for\_hover\_on\_data 14
6.2 Clique\_constraints 14
6.3 Connection\_tests 14
6.4 Messages 14
6.5 Network\_agent\_types 14
6.6 Roles, Users 15
6.7 Statistics 15
6.8 Constants 15
6.9 Constants-env\_types 15
6.10 Constants-log\_levels 15
6.11 Constants-mechanism\_drivers 15
6.12 Constants-type\_drivers 15
6.13 Constants-environment\_monitoring\_types 15
6.14 Constants-link\_states 15
6.15 Constants-environment\_provision\_types 15
6.16 Constants-environment\_operational\_status 16
6.17 Constants-link\_types 16
6.18 Constants-monitoring\_sides 16
6.19 Constants-object\_types 16
6.20 Constants-scans\_statuses 16
6.21 Constants-distributions 16
6.22 Constants-distribution\_versions 16
6.23 Constants-message\_source\_systems 16
6.24 Constants-object\_types\_for\_links 16
6.25 Constants-scan\_object\_types 17
Environments config
===================
Environment is defined as a certain type of Virtual Infrastructure
facility the runs under a single unified Management (like an
OpenStack facility).
Everything in Calipso application rely on environments config, this
is maintained in the **“environments\_config”** collection in the
mongo Calipso DB.
Environment configs are pushed down to Calipso DB either through UI
or API (and only in OPNFV case Calipso provides an automated program
to build all needed environments\_config parameters for an ‘Apex’
distribution automatically).
When scanning and discovering items Calipso uses this configuration
document for successful scanning results, here is an example of an
environment config document:
**{ **
**"name": "DEMO-ENVIRONMENT-SCHEME", **
**"enable\_monitoring": true, **
**"last\_scanned": "filled-by-scanning", **
**"app\_path": "/home/scan/calipso\_prod/app", **
**"type": "environment", **
**"distribution": "Mirantis", **
**"distribution\_version": "8.0”, **
**"mechanism\_drivers": ["OVS”], **
**"type\_drivers": "vxlan"**
**"operational": "stopped", **
**"listen": true, **
**"scanned": false, **
**"configuration": [**
**{**
**"name": "OpenStack", **
**"port":”5000”, **
**"user": "adminuser", **
**"pwd": "dummy\_pwd", **
**"host": "10.0.0.1", **
**"admin\_token": "dummy\_token"**
**}, **
**{**
**"name": "mysql", **
**"pwd": "dummy\_pwd", **
**"host": "10.0.0.1", **
**"port": “3307”, **
**"user": "mysqluser"**
**}, **
**{**
**"name": "CLI", **
**"user": "sshuser", **
**"host": "10.0.0.1", **
**"pwd": "dummy\_pwd"**
**}, **
**{**
**"name": "AMQP", **
**"pwd": "dummy\_pwd", **
**"host": "10.0.0.1", **
**"port": “5673”, **
**"user": "rabbitmquser"**
**}, **
**{**
**"name": "Monitoring", **
**"ssh\_user": "root", **
**"server\_ip": "10.0.0.1", **
**"ssh\_password": "dummy\_pwd", **
**"rabbitmq\_pass": "dummy\_pwd", **
**"rabbitmq\_user": "sensu", **
**"rabbitmq\_port": “5671”, **
**"provision": "None", **
**"env\_type": "production", **
**"ssh\_port": “20022”, **
**"config\_folder": "/local\_dir/sensu\_config", **
**"server\_name": "sensu\_server", **
**"type": "Sensu", **
**"api\_port": NumberInt(4567)**
**}, **
**{**
**"name": "ACI", **
**"user": "admin", **
**"host": "10.1.1.104", **
**"pwd": "dummy\_pwd"**
**}**
**], **
**"user": "wNLeBJxNDyw8G7Ssg", **
**"auth": {**
**"view-env": [**
**"wNLeBJxNDyw8G7Ssg"**
**], **
**"edit-env": [**
**"wNLeBJxNDyw8G7Ssg"**
**]**
**}, **
**}**
Here is a brief explanation of the purpose of major keys in this
environment configuration doc:
**Distribution**: captures type of VIM, used for scanning of
objects, links and cliques.
**Distribution\_version**: captures version of VIM distribution,
used for scanning of objects, links and cliques.
**Mechanism\_driver**: captures virtual switch type used by the VIM,
used for scanning of objects, links and cliques.
**Type\_driver**: captures virtual switch tunneling type used by the
switch, used for scanning of objects, links and cliques.
**Listen**: defines whether or not to use Calipso listener against
the VIM BUS for updating inventory in real-time from VIM events.
**Scanned**: defines whether or not Calipso ran a full and a
successful scan against this environment.
**Last\_scanned**: end time of last scan.
**Operational**: defines whether or not VIM environment endpoints
are up and running.
**Enable\_monitoring**: defines whether or not Calipso should deploy
monitoring of the inventory objects running inside all environment
hosts.
**Configuration-OpenStack**: defines credentials for OpenStack API
endpoints access.
**Configuration-mysql**: defines credentials for OpenStack DB
access.
**Configuration-CLI**: defines credentials for servers CLI access.
**Configuration-AMQP**: defines credentials for OpenStack BUS
access.
**Configuration-Monitoring**: defines credentials and setup for
Calipso sensu server (see monitoring-guide for details).
**Configuration-ACI**: defines credentials for ACI switched
management API, if exists.
**User and auth**: used for UI authorizations to view and edit this
environment.
**App-path**: defines the root directory of the scanning
application.
Inventory objects
=================
Calipso’s success in scanning, discovering and analyzing many (60 as
of 27\ :sup:`th` Aug 2017) variances of virtual infrastructures lies
with its objects model and relationship definitions (model was
tested even against a vSphere VMware environment).
Those objects are the real-time processes and systems that are built
by workers and agents on the virtual infrastructure servers.
All Calipso objects are maintained in the **“inventory”**
collection.
Here are the major objects defined in Calipso inventory in order to
capture the real-time state of networking:
Host
----
It’s the physical server that runs all virtual objects, typically a
hypervisor or a containers hosting machine.
It’s typically a bare-metal server, in some cases it might be
virtual (running “nesting” VMs as second virtualization layer inside
it).
physical NIC (pNIC)
-------------------
It’s the physical Ethernet Network Interface Card attached to the
Host, typically several of those are available on a host, in some
cases few of those are grouped (bundled) together into etherchannel
bond interfaces.
For capturing data from real infrastructure devices Calipso created
2 types of pNICs: host\_pnic (pNICs on the servers) and switch\_pnic
(pNICs on the physical switches). Calipso currently discovers host
to switch physical connections only in some types of switches (Cisco
ACI as of Aug 27\ :sup:`th` 2017).
Bond
----
It’s a logical Network Interface using etherchannel standard
protocols to form a group of pNICs providing enhanced throughput
for communications to/from the host.
Calipso currently maintains bond details inside a host\_pnic object.
Instance
--------
It’s the virtual server created for running a certain application or
function. Typically it’s a Virtual Machine, sometimes it’s a
Container.
virtual Service (vService)
--------------------------
It’s a process/system that provides some type of networking service
to instances running on networks, some might be deployed as
namespaces and some might deploy as VM or Container, for example:
DHCP server, Router, Firewall, Load-Balancer, VPN service and
others. Calipso categorized vServices accordingly.
Network
-------
It’s an abstracted object, illustrating and representing all the
components (see below) that builds and provides communication
services for several instances and vServices.
virtual NIC (vNIC)
------------------
There are 2 types - instance vNIC and vService vNIC:
- Instance vNIC: It’s the virtual Network Interface Card attached to
the Instance and used by it for communications from/to that instance.
- vService vNIC: It’s the virtual Network Interface Card attached to
the vService used by it for communications from/to that vService.
Port
----
It’s an abstracted object representing the attachment point for an
instance or a vService into the network, in reality it’s fulfilled
by deployment of vNICs on hosts.
virtual Connector (vConnector)
------------------------------
It’s a process/system that provides layer 2 isolation for a specific
network inside the host (isolating traffic from other networks).
Examples: Linux Bridge, Bridge-group, port-group etc.
virtual Edge (vEdge)
--------------------
It’s a process/system that provides switching and routing services
for instances and/or vServices running on a specific host. It
function as an edge device between virtual components running on
that host and the pNICs on that host, making sure traffic is
maintained and still isolated across different networks.
Examples: Open Virtual Switch, Midonet, VPP.
Overlay-Tunnel-Endpoint (OTEP)
------------------------------
It’s an abstracted object representing the end-point on the host
that runs a certain tunneling technology to provide isolation across
networks and hosts for packets leaving and entering the pNICs of a
specific host. Examples: VXLAN tunnels endpoints, GRE tunnels
endpoints etc.
Network\_segment
----------------
It’s the specific segment used inside the “overlay tunnel” to
represent traffic from a specific network, this depends on the
specific type (encapsulation) of the OTEP.
Calipso currently maintains segments details inside a network
object.
Network\_Agent
--------------
It’s a controlling software running on the hosts for orchestrating
the lifecycle of the above virtual components. Examples: DHCP agent,
L3 agent, OVS agent, Metadata agent etc.
Looking up Calipso objects details
----------------------------------
As explained in more details in Calipso admin-guide, the underlying
database used is mongoDB. All major objects discovered by Calipso
scanning module are maintained in the “inventory” collection and
those document includes detailed attributes captured from the
infrastructure about those objects, here are the main objects
quarries to use for grabbing each of the above object types from
Calipso’s inventory:
**{type:"vnic"}**
**{type:"vservice"}**
**{type:"instance"}**
**{type:"host\_pnic"}**
**{type:"switch\_pnic"}**
**{type:"vconnector"}**
**{type:"vedge"}**
**{type:"network"}**
**{type:"network\_agent"}**
**{type:"otep"}**
**{type:"host"}**
**{type:"port"}**
All Calipso modules (visualization, monitoring and analysis) rely on
those objects as baseline inventory items for any further
computation.
Here is an example of a query made using mongo Chef Client
application:
|image1|
\* See Calipso API-guide for details on looking up those objects
through the Calipso API.
The following simplified UML illustrates the way Calipso objects
relationships are maintained in a VIM of type OpenStack:
|image2|
Link Objects
============
Calipso analyzes all objects in its inventory for relationships,
finding in real-time, which object is attached to which object and
then creates a link object representing this relationship. This
analysis finds a link that is “single hop away” - a connection from
certain object to certain object that is attached to it directly.
Derived relationships (A to B and B to C = A to C) is maintained as
‘cliques’.
Links objects are maintained in the **“links”** collection.
Link types
----------
Based on the specific VIM distribution, distribution version,
mechanism driver and type driver a set of links are discovered
automatically by Calipso scanning module. Each link type is
bi-directional, it means that if a connection is discovered from A
to B, a connection also exists from B to A.
Here is the list of link types that might be discovered from a
certain environment in the current release:
**{"link\_type": "instance-vnic"}**
**{"link\_type": "vnic-vconnector"}**
**{"link\_type": "vconnector-vedge"}**
**{"link\_type": "vedge-host\_pnic"}**
**{"link\_type: "host\_pnic-network"}**
**{"link\_type": "vedge-otep"}**
**{"link\_type": "otep-vconnector"}**
**{"link\_type": "otep-host\_pnic"}**
**{"link\_type": "vconnector-host\_pnic"}**
**{"link\_type": "vnic-vedge"}**
**{"link\_type": "vservice-vnic"}**
**{"link\_type": "switch\_pnic-host\_pnic"}**
**{"link\_type": "switch\_pnic-switch\_pnic"}**
**{"link\_type": "switch\_pnic-switch"}**
A successful completion of scanning and discovery means that all
inventory objects, link objects and clique objects (see below) are
found and accurately representing real-time state of the virtual
networking on the specific environment.
Clique objects
==============
Cliques are lists of links. Clique represent a certain path in the
virtual networking infrastructure that an administrator is
interested in, this is made to allow easier searching and finding of
certain points of interest (“focal point”).
Clique types
------------
Based on the specific VIM distribution, distribution version,
mechanism driver and type driver variance, Calipso scanning module
search for specific cliques using a model that is pre-populated in
its **“clique\_types”** collection, and it depends on the
environment variance, here is an example of a clique\_type:
**{ **
**"environment" : "Apex-Euphrates", **
**"link\_types" : [**
**"instance-vnic", **
**"vnic-vconnector", **
**"vconnector-vedge", **
**"vedge-otep", **
**"otep-host\_pnic", **
**"host\_pnic-network"**
**], **
**"name": "instance\_clique\_for\_opnfv", **
**"focal\_point\_type": "instance"**
**}**
The above model instruct the Calipso scanner to create cliques with
the above list of link types for a “focal\_point” that is an
“instance” type of object. We believe this is a highly customized
model for analysis of dependencies for many use cases. We have
included several clique types, common across variances supported in
this release.
The cliques themselves are then maintained in the **“cliques**\ ”
collection.
To clarify this concept, here is an example for an implementation
use case in the Calipso UI module:
When the user of the UI clicks on a certain object of type=instance,
he expresses he’s wish to see a graph representing the path taken by
traffic from that specific instance (as the root source of traffic,
on that specific network) all the way down to the host pNIC and the
(abstracted) network itself.
A successful completion of scanning and discovery means that all
inventory objects, link objects and clique objects (based on the
environment clique types) are found and accurately representing
real-time state of the virtual networking on the specific
environment.
Supported Environments
======================
As of Aug 27\ :sup:`th` 2017, Calipso application supports 60
different VIM environment variances and with each release the
purpose of the application is to maintain support and add more
variances per the VIM development cycles. The latest supported
variance and the specific functions of Calipso available for that
specific variance is captured in the **“supported\_environments”**
collection, here are two examples of that ‘supported’ model:
**1.**
**{ **
**"environment" : {**
**"distribution" : "Apex", **
**"distribution\_version" : ["Euphrates"], **
**"mechanism\_drivers" : "OVS", **
**"type\_drivers" : "vxlan"**
**}, **
**"features" : {**
**"listening" : true, **
**"scanning" : true, **
**"monitoring" : false**
**}**
**}**
**2.**
**{ **
**"environment" : {**
**"distribution" : "Mirantis", **
**"distribution\_version": ["6.0", "7.0", "8.0", "9.0", "9.1",
"10.0"], **
**"mechanism\_drivers" : "OVS", **
**"type\_drivers" : "vxlan"**
**}, **
**"features" : {**
**"listening" : true, **
**"scanning" : true, **
**"monitoring" : true**
**}**
**}**
The examples above defines for Calipso application that:
1. For an ‘Apex’ environment of version ‘Euphrates’ using OVS and vxlan,
Calipso can scan/discover all details (objects, links, cliques) but
is not yet monitoring those discovered objects.
2. For a “Mirantis” environment of versions 6.0 to 10.0 using OVS and
vxlan, Calipso can scan/discover all details (objects, links,
cliques) and also monitor those discovered objects.
With each calipso release more “supported\_environments” should be
added.
System collections
==================
Calipso uses other system collections to maintain its data for scanning,
event handling, monitoring and for helping to operate the API and UI
modules, here is the recent list of collections not covered yet in other
written guides:
Attributes\_for\_hover\_on\_data
--------------------------------
This collection maintains a list of documents describing what will be
presented on the UI popup screen when the use hover-on a specific object
type, it details which parameters or attributed from the object’s data
will be shown on the screen, making this popup fully customized.
Clique\_constraints
-------------------
Defines the logic on which cliques are built, currently network is the
main focus of the UI (central point of connection for all cliques in the
system), but this is customizable.
When building a clique graph, Calipso defaults to traversing all nodes
edges (links) in the graph.
In some cases we want to limit the graph so it will not expand too much
(nor forever).
For example: when we build the graph for a specific instance, we limit
the graph to only take objects from the network on which this instance
resides - otherwise the graph will show objects related to other
instances.
The constraint validation is done by checking value V from the focal
point F on the links.
For example, if an n instance has network X, we check that each link we
use either has network X (attribute “network” has value X), or does not
have the “network” attribute.
Connection\_tests
-----------------
This collection keeps requests from the UI or API to test the different adapters (API, DB, and CLI etc)
and their connections to the underlying VIM, making sure dynamic and real-time data will be maintained during discovery.
Messages
--------
Aggregates all loggings from the different systems, source\_system of
logs currently defined as “OpenStack” (the VIM), “Sensu” (the Monitoring
module) and “Calipso” (logs of the application itself. Messages have 6
levels of severity and can be browsed in the UI and through Calipso API.
Network\_agent\_types
---------------------
Lists the types of networking agents supported on the VIM (per
distribution and version).
Roles, Users
------------
Basic RBAC facility to authorize calispo UI users for certain calipso
functionalities on the UI.
Statistics
----------
Built for detailed analysis and future functionalities, used today for
traffic analysis (capturing samples of throughputs per session on VPP
based environments).
Constants
---------
This is an aggregated collection for many types of documents that are
required mostly by the UI and basic functionality on some scanning
classes (‘fetchers’).
Constants-env\_types
--------------------
Type of environments to allow for configuration on sensu monitoring
framework.
Constants-log\_levels
---------------------
Severity levels for messages generated.
Constants-mechanism\_drivers
----------------------------
Mechanism-drivers allowed for UI users.
Constants-type\_drivers
-----------------------
Type-drivers allowed for UI users.
Constants-environment\_monitoring\_types
----------------------------------------
Currently only “Sensu” is available, might be used for other monitoring
systems integrations.
Constants-link\_states
----------------------
Provides statuses for link objects, based on monitoring results.
Constants-environment\_provision\_types
---------------------------------------
The types of deployment options available for monitoring (see
monitoring-guide for details).
Constants-environment\_operational\_status
------------------------------------------
Captures the overall (aggregated) status of a curtained environment.
Constants-link\_types
---------------------
Lists the connections and relationships options for objects in the
inventory.
Constants-monitoring\_sides
---------------------------
Used for monitoring auto configurations of clients and servers.
Constants-object\_types
-----------------------
Lists the type of objects supported through scanning (inventory
objects).
Constants-scans\_statuses
-------------------------
During scans, several statuses are shown on the UI, based on the
specific stage and results.
Constants-distributions
-----------------------
Lists the VIM distributions.
Constants-distribution\_versions
--------------------------------
Lists the VIM different versions of different distributions.
Constants-message\_source\_systems
----------------------------------
The list of systems that can generate logs and messages.
Constants-object\_types\_for\_links
-----------------------------------
Object\_types used only for link popups on UI.
Constants-scan\_object\_types
-----------------------------
Object\_types used during scanning, see development-guide for details.
Constants-configuration_targets
Names of the configuration targets used in the configuration section of environment configs.
.. |image0| image:: media/image1.png
:width: 6.50000in
:height: 4.27153in
.. |image1| image:: media/image8.png
:width: 6.50000in
:height: 2.43750in
.. |image2| image:: media/image9.png
:width: 7.02325in
:height: 5.22917in