summaryrefslogtreecommitdiffstats
path: root/docs/release/calipso-model.rst
diff options
context:
space:
mode:
Diffstat (limited to 'docs/release/calipso-model.rst')
-rw-r--r--docs/release/calipso-model.rst926
1 files changed, 926 insertions, 0 deletions
diff --git a/docs/release/calipso-model.rst b/docs/release/calipso-model.rst
new file mode 100644
index 0000000..d7ee8f7
--- /dev/null
+++ b/docs/release/calipso-model.rst
@@ -0,0 +1,926 @@
+###############################################################################
+# Copyright (c) 2017 Koren Lev (Cisco Systems), Yaron Yogev (Cisco Systems) #
+# and others #
+# #
+# All rights reserved. This program and the accompanying materials #
+# are made available under the terms of the Apache License, Version 2.0 #
+# which accompanies this distribution, and is available at #
+# http://www.apache.org/licenses/LICENSE-2.0 #
+###############################################################################
+| Calipso.io
+| Objects Model
+
+|image0|
+
+Project “Calipso” tries to illuminate complex virtual networking with
+real time operational state visibility for large and highly distributed
+Virtual Infrastructure Management (VIM).
+
+Calipso provides visible insights using smart discovery and virtual
+topological representation in graphs, with monitoring per object in the
+graph inventory to reduce error vectors and troubleshooting, maintenance
+cycles for VIM operators and administrators.
+
+Calipso model, described in this document, was *built for
+multi-environment and many VIM variances*, the model was tested
+successfully (as of Aug 27\ :sup:`th`) against 60 different VIM
+variances (Distributions, Versions, Networking Drivers and Types).
+
+Table of Contents
+
+Calipso.io Objects Model 1
+
+1 Environments config 4
+
+2 Inventory objects 6
+
+2.1 Host 6
+
+2.2 physical NIC (pNIC) 7
+
+2.3 Bond 7
+
+2.4 Instance 7
+
+2.5 virtual Service (vService) 7
+
+2.6 Network 7
+
+2.7 virtual NIC (vNIC) 7
+
+2.8 Port 8
+
+2.9 virtual Connector (vConnector) 8
+
+2.10 virtual Edge (vEdge) 8
+
+2.11 Overlay-Tunnel-Endpoint (OTEP) 8
+
+2.12 Network\_segment 8
+
+2.13 Network\_Agent 8
+
+2.14 Looking up Calipso objects details 9
+
+3 Link Objects 10
+
+3.1 Link types 11
+
+4 Clique objects 11
+
+4.1 Clique types 11
+
+5 Supported Environments 12
+
+6 System collections 14
+
+6.1 Attributes\_for\_hover\_on\_data 14
+
+6.2 Clique\_constraints 14
+
+6.3 Connection\_tests 14
+
+6.4 Messages 14
+
+6.5 Network\_agent\_types 14
+
+6.6 Roles, Users 15
+
+6.7 Statistics 15
+
+6.8 Constants 15
+
+6.9 Constants-env\_types 15
+
+6.10 Constants-log\_levels 15
+
+6.11 Constants-mechanism\_drivers 15
+
+6.12 Constants-type\_drivers 15
+
+6.13 Constants-environment\_monitoring\_types 15
+
+6.14 Constants-link\_states 15
+
+6.15 Constants-environment\_provision\_types 15
+
+6.16 Constants-environment\_operational\_status 16
+
+6.17 Constants-link\_types 16
+
+6.18 Constants-monitoring\_sides 16
+
+6.19 Constants-object\_types 16
+
+6.20 Constants-scans\_statuses 16
+
+6.21 Constants-distributions 16
+
+6.22 Constants-distribution\_versions 16
+
+6.23 Constants-message\_source\_systems 16
+
+6.24 Constants-object\_types\_for\_links 16
+
+6.25 Constants-scan\_object\_types 17
+
+Environments config
+===================
+
+ Environment is defined as a certain type of Virtual Infrastructure
+ facility the runs under a single unified Management (like an
+ OpenStack facility).
+
+ Everything in Calipso application rely on environments config, this
+ is maintained in the **“environments\_config”** collection in the
+ mongo Calipso DB.
+
+ Environment configs are pushed down to Calipso DB either through UI
+ or API (and only in OPNFV case Calipso provides an automated program
+ to build all needed environments\_config parameters for an ‘Apex’
+ distribution automatically).
+
+ When scanning and discovering items Calipso uses this configuration
+ document for successful scanning results, here is an example of an
+ environment config document:
+
+ **{ **
+
+ **"name": "DEMO-ENVIRONMENT-SCHEME", **
+
+ **"enable\_monitoring": true, **
+
+ **"last\_scanned": "filled-by-scanning", **
+
+ **"app\_path": "/home/scan/calipso\_prod/app", **
+
+ **"type": "environment", **
+
+ **"distribution": "Mirantis", **
+
+ **"distribution\_version": "8.0”, **
+
+ **"mechanism\_drivers": ["OVS”], **
+
+ **"type\_drivers": "vxlan"**
+
+ **"operational": "stopped", **
+
+ **"listen": true, **
+
+ **"scanned": false, **
+
+ **"configuration": [**
+
+ **{**
+
+ **"name": "OpenStack", **
+
+ **"port":”5000”, **
+
+ **"user": "adminuser", **
+
+ **"pwd": "dummy\_pwd", **
+
+ **"host": "10.0.0.1", **
+
+ **"admin\_token": "dummy\_token"**
+
+ **}, **
+
+ **{**
+
+ **"name": "mysql", **
+
+ **"pwd": "dummy\_pwd", **
+
+ **"host": "10.0.0.1", **
+
+ **"port": “3307”, **
+
+ **"user": "mysqluser"**
+
+ **}, **
+
+ **{**
+
+ **"name": "CLI", **
+
+ **"user": "sshuser", **
+
+ **"host": "10.0.0.1", **
+
+ **"pwd": "dummy\_pwd"**
+
+ **}, **
+
+ **{**
+
+ **"name": "AMQP", **
+
+ **"pwd": "dummy\_pwd", **
+
+ **"host": "10.0.0.1", **
+
+ **"port": “5673”, **
+
+ **"user": "rabbitmquser"**
+
+ **}, **
+
+ **{**
+
+ **"name": "Monitoring", **
+
+ **"ssh\_user": "root", **
+
+ **"server\_ip": "10.0.0.1", **
+
+ **"ssh\_password": "dummy\_pwd", **
+
+ **"rabbitmq\_pass": "dummy\_pwd", **
+
+ **"rabbitmq\_user": "sensu", **
+
+ **"rabbitmq\_port": “5671”, **
+
+ **"provision": "None", **
+
+ **"env\_type": "production", **
+
+ **"ssh\_port": “20022”, **
+
+ **"config\_folder": "/local\_dir/sensu\_config", **
+
+ **"server\_name": "sensu\_server", **
+
+ **"type": "Sensu", **
+
+ **"api\_port": NumberInt(4567)**
+
+ **}, **
+
+ **{**
+
+ **"name": "ACI", **
+
+ **"user": "admin", **
+
+ **"host": "10.1.1.104", **
+
+ **"pwd": "dummy\_pwd"**
+
+ **}**
+
+ **], **
+
+ **"user": "wNLeBJxNDyw8G7Ssg", **
+
+ **"auth": {**
+
+ **"view-env": [**
+
+ **"wNLeBJxNDyw8G7Ssg"**
+
+ **], **
+
+ **"edit-env": [**
+
+ **"wNLeBJxNDyw8G7Ssg"**
+
+ **]**
+
+ **}, **
+
+ **}**
+
+ Here is a brief explanation of the purpose of major keys in this
+ environment configuration doc:
+
+ **Distribution**: captures type of VIM, used for scanning of
+ objects, links and cliques.
+
+ **Distribution\_version**: captures version of VIM distribution,
+ used for scanning of objects, links and cliques.
+
+ **Mechanism\_driver**: captures virtual switch type used by the VIM,
+ used for scanning of objects, links and cliques.
+
+ **Type\_driver**: captures virtual switch tunneling type used by the
+ switch, used for scanning of objects, links and cliques.
+
+ **Listen**: defines whether or not to use Calipso listener against
+ the VIM BUS for updating inventory in real-time from VIM events.
+
+ **Scanned**: defines whether or not Calipso ran a full and a
+ successful scan against this environment.
+
+ **Last\_scanned**: end time of last scan.
+
+ **Operational**: defines whether or not VIM environment endpoints
+ are up and running.
+
+ **Enable\_monitoring**: defines whether or not Calipso should deploy
+ monitoring of the inventory objects running inside all environment
+ hosts.
+
+ **Configuration-OpenStack**: defines credentials for OpenStack API
+ endpoints access.
+
+ **Configuration-mysql**: defines credentials for OpenStack DB
+ access.
+
+ **Configuration-CLI**: defines credentials for servers CLI access.
+
+ **Configuration-AMQP**: defines credentials for OpenStack BUS
+ access.
+
+ **Configuration-Monitoring**: defines credentials and setup for
+ Calipso sensu server (see monitoring-guide for details).
+
+ **Configuration-ACI**: defines credentials for ACI switched
+ management API, if exists.
+
+ **User and auth**: used for UI authorizations to view and edit this
+ environment.
+
+ **App-path**: defines the root directory of the scanning
+ application.
+
+Inventory objects
+=================
+
+ Calipso’s success in scanning, discovering and analyzing many (60 as
+ of 27\ :sup:`th` Aug 2017) variances of virtual infrastructures lies
+ with its objects model and relationship definitions (model was
+ tested even against a vSphere VMware environment).
+
+ Those objects are the real-time processes and systems that are built
+ by workers and agents on the virtual infrastructure servers.
+
+ All Calipso objects are maintained in the **“inventory”**
+ collection.
+
+ Here are the major objects defined in Calipso inventory in order to
+ capture the real-time state of networking:
+
+Host
+----
+
+ It’s the physical server that runs all virtual objects, typically a
+ hypervisor or a containers hosting machine.
+
+ It’s typically a bare-metal server, in some cases it might be
+ virtual (running “nesting” VMs as second virtualization layer inside
+ it).
+
+physical NIC (pNIC)
+-------------------
+
+ It’s the physical Ethernet Network Interface Card attached to the
+ Host, typically several of those are available on a host, in some
+ cases few of those are grouped (bundled) together into etherchannel
+ bond interfaces.
+
+ For capturing data from real infrastructure devices Calipso created
+ 2 types of pNICs: host\_pnic (pNICs on the servers) and switch\_pnic
+ (pNICs on the physical switches). Calipso currently discovers host
+ to switch physical connections only in some types of switches (Cisco
+ ACI as of Aug 27\ :sup:`th` 2017).
+
+Bond
+----
+
+ It’s a logical Network Interface using etherchannel standard
+ protovcols to form a group of pNICs providing enhanced throughput
+ for communications to/from the host.
+
+ Calipso currently maintains bond details inside a host\_pnic object.
+
+Instance
+--------
+
+ It’s the virtual server created for running a certain application or
+ function. Typically it’s a Virtual Machine, sometimes it’s a
+ Container.
+
+virtual Service (vService)
+--------------------------
+
+ It’s a process/system that provides some type of networking service
+ to instances running on networks, some might be deployed as
+ namespaces and some might deploy as VM or Container, for example:
+ DHCP server, Router, Firewall, Load-Balancer, VPN service and
+ others. Calipso categorized vServices accordingly.
+
+Network
+-------
+
+ It’s an abstracted object, illustrating and representing all the
+ components (see below) that builds and provides communication
+ services for several instances and vServices.
+
+virtual NIC (vNIC)
+------------------
+
+ There are 2 types - instance vNIC and vService vNIC:
+
+- Instance vNIC: It’s the virtual Network Interface Card attached to
+ the Instance and used by it for communications from/to that instance.
+
+- vService vNIC: It’s the virtual Network Interface Card attached to
+ the vService used by it for communications from/to that vService.
+
+Port
+----
+
+ It’s an abstracted object representing the attachment point for an
+ instance or a vService into the network, in reality it’s fulfilled
+ by deployment of vNICs on hosts.
+
+virtual Connector (vConnector)
+------------------------------
+
+ It’s a process/system that provides layer 2 isolation for a specific
+ network inside the host (isolating traffic from other networks).
+ Examples: Linux Bridge, Bridge-group, port-group etc.
+
+virtual Edge (vEdge)
+--------------------
+
+ It’s a process/system that provides switching and routing services
+ for instances and/or vServices running on a specific host. It
+ function as an edge device between virtual components running on
+ that host and the pNICs on that host, making sure traffic is
+ maintained and still isolated across different networks.
+
+ Examples: Open Virtual Switch, Midonet, VPP.
+
+Overlay-Tunnel-Endpoint (OTEP)
+------------------------------
+
+ It’s an abstracted object representing the end-point on the host
+ that runs a certain tunneling technology to provide isolation across
+ networks and hosts for packets leaving and entering the pNICs of a
+ specific host. Examples: VXLAN tunnels endpoints, GRE tunnels
+ endpoints etc.
+
+Network\_segment
+----------------
+
+ It’s the specific segment used inside the “overlay tunnel” to
+ represent traffic from a specific network, this depends on the
+ specific type (encapsulation) of the OTEP.
+
+ Calipso currently maintains segments details inside a network
+ object.
+
+Network\_Agent
+--------------
+
+ It’s a controlling software running on the hosts for orchestrating
+ the lifecycle of the above virtual components. Examples: DHCP agent,
+ L3 agent, OVS agent, Metadata agent etc.
+
+Looking up Calipso objects details
+----------------------------------
+
+ As explained in more details in Calipso admin-guide, the underlying
+ database used is mongoDB. All major objects discovered by Calipso
+ scanning module are maintained in the “inventory” collection and
+ those document includes detailed attributes captured from the
+ infrastructure about those objects, here are the main objects
+ quarries to use for grabbing each of the above object types from
+ Calipso’s inventory:
+
+ **{type:"vnic"}**
+
+ **{type:"vservice"}**
+
+ **{type:"instance"}**
+
+ **{type:"host\_pnic"}**
+
+ **{type:"switch\_pnic"}**
+
+ **{type:"vconnector"}**
+
+ **{type:"vedge"}**
+
+ **{type:"network"}**
+
+ **{type:"network\_agent"}**
+
+ **{type:"otep"}**
+
+ **{type:"host"}**
+
+ **{type:"port"}**
+
+ All Calipso modules (visualization, monitoring and analysis) rely on
+ those objects as baseline inventory items for any further
+ computation.
+
+ Here is an example of a query made using mongo Chef Client
+ application:
+
+ |image1|
+
+ \* See Calipso API-guide for details on looking up those objects
+ through the Calipso API.
+
+ The following simplified UML illustrates the way Calipso objects
+ relationships are maintained in a VIM of type OpenStack:
+
+|image2|
+
+Link Objects
+============
+
+ Calipso analyzes all objects in its inventory for relationships,
+ finding in real-time, which object is attached to which object and
+ then creates a link object representing this relationship. This
+ analysis finds a link that is “single hop away” - a connection from
+ certain object to certain object that is attached to it directly.
+
+ Derived relationships (A to B and B to C = A to C) is maintained as
+ ‘cliques’.
+
+ Links objects are maintained in the **“links”** collection.
+
+Link types
+----------
+
+ Based on the specific VIM distribution, distribution version,
+ mechanism driver and type driver a set of links are discovered
+ automatically by Calipso scanning module. Each link type is
+ bi-directional, it means that if a connection is discovered from A
+ to B, a connection also exists from B to A.
+
+ Here is the list of link types that might be discovered from a
+ certain environment in the current release:
+
+ **{"link\_type": "instance-vnic"}**
+
+ **{"link\_type": "vnic-vconnector"}**
+
+ **{"link\_type": "vconnector-vedge"}**
+
+ **{"link\_type": "vedge-host\_pnic"}**
+
+ **{"link\_type: "host\_pnic-network"}**
+
+ **{"link\_type": "vedge-otep"}**
+
+ **{"link\_type": "otep-vconnector"}**
+
+ **{"link\_type": "otep-host\_pnic"}**
+
+ **{"link\_type": "vconnector-host\_pnic"}**
+
+ **{"link\_type": "vnic-vedge"}**
+
+ **{"link\_type": "vservice-vnic"}**
+
+ **{"link\_type": "switch\_pnic-host\_pnic"}**
+
+ **{"link\_type": "switch\_pnic-switch\_pnic"}**
+
+ **{"link\_type": "switch\_pnic-switch"}**
+
+ A successful completion of scanning and discovery means that all
+ inventory objects, link objects and clique objects (see below) are
+ found and accurately representing real-time state of the virtual
+ networking on the specific environment.
+
+Clique objects
+==============
+
+ Cliques are lists of links. Clique represent a certain path in the
+ virtual networking infrastructure that an administrator is
+ interested in, this is made to allow easier searching and finding of
+ certain points of interest (“focal point”).
+
+Clique types
+------------
+
+ Based on the specific VIM distribution, distribution version,
+ mechanism driver and type driver variance, Calipso scanning module
+ search for specific cliques using a model that is pre-populated in
+ its **“clique\_types”** collection, and it depends on the
+ environment variance, here is an example of a clique\_type:
+
+ **{ **
+
+ **"environment" : "Apex-Euphrates", **
+
+ **"link\_types" : [**
+
+ **"instance-vnic", **
+
+ **"vnic-vconnector", **
+
+ **"vconnector-vedge", **
+
+ **"vedge-otep", **
+
+ **"otep-host\_pnic", **
+
+ **"host\_pnic-network"**
+
+ **], **
+
+ **"name": "instance\_clique\_for\_opnfv", **
+
+ **"focal\_point\_type": "instance"**
+
+ **}**
+
+ The above model instruct the Calipso scanner to create cliques with
+ the above list of link types for a “focal\_point” that is an
+ “instance” type of object. We believe this is a highly customized
+ model for analysis of dependencies for many use cases. We have
+ included several clique types, common across variances supported in
+ this release.
+
+ The cliques themselves are then maintained in the **“cliques**\ ”
+ collection.
+
+ To clarify this concept, here is an example for an implementation
+ use case in the Calipso UI module:
+
+ When the user of the UI clicks on a certain object of type=instance,
+ he expresses he’s wish to see a graph representing the path taken by
+ traffic from that specific instance (as the root source of traffic,
+ on that specific network) all the way down to the host pNIC and the
+ (abstracted) network itself.
+
+ A successful completion of scanning and discovery means that all
+ inventory objects, link objects and clique objects (based on the
+ environment clique types) are found and accurately representing
+ real-time state of the virtual networking on the specific
+ environment.
+
+Supported Environments
+======================
+
+ As of Aug 27\ :sup:`th` 2017, Calipso application supports 60
+ different VIM environment variances and with each release the
+ purpose of the application is to maintain support and add more
+ variances per the VIM development cycles. The latest supported
+ variance and the specific functions of Calipso available for that
+ specific variance is captured in the **“supported\_environments”**
+ collection, here are two examples of that ‘supported’ model:
+
+ **1.**
+
+ **{ **
+
+ **"environment" : {**
+
+ **"distribution" : "Apex", **
+
+ **"distribution\_version" : ["Euphrates"], **
+
+ **"mechanism\_drivers" : "OVS", **
+
+ **"type\_drivers" : "vxlan"**
+
+ **}, **
+
+ **"features" : {**
+
+ **"listening" : true, **
+
+ **"scanning" : true, **
+
+ **"monitoring" : false**
+
+ **}**
+
+ **}**
+
+ **2.**
+
+ **{ **
+
+ **"environment" : {**
+
+ **"distribution" : "Mirantis", **
+
+ **"distribution\_version": ["6.0", "7.0", "8.0", "9.0", "9.1",
+ "10.0"], **
+
+ **"mechanism\_drivers" : "OVS", **
+
+ **"type\_drivers" : "vxlan"**
+
+ **}, **
+
+ **"features" : {**
+
+ **"listening" : true, **
+
+ **"scanning" : true, **
+
+ **"monitoring" : true**
+
+ **}**
+
+ **}**
+
+ The examples above defines for Calipso application that:
+
+1. For an ‘Apex’ environment of version ‘Euphrates’ using OVS and vxlan,
+ Calipso can scan/discover all details (objects, links, cliques) but
+ is not yet monitoring those discovered objects.
+
+2. For a “Mirantis” environment of versions 6.0 to 10.0 using OVS and
+ vxlan, Calipso can scan/discover all details (objects, links,
+ cliques) and also monitor those discovered objects.
+
+With each calipso release more “supported\_environments” should be
+added.
+
+System collections
+==================
+
+Calipso uses other system collections to maintain its data for scanning,
+event handling, monitoring and for helping to operate the API and UI
+modules, here is the recent list of collections not covered yet in other
+written guides:
+
+Attributes\_for\_hover\_on\_data
+--------------------------------
+
+This collection maintains a list of documents describing what will be
+presented on the UI popup screen when the use hover-on a specific object
+type, it details which parameters or attributed from the object’s data
+will be shown on the screen, making this popup fully customized.
+
+Clique\_constraints
+-------------------
+
+Defines the logic on which cliques are built, currently network is the
+main focus of the UI (central point of connection for all cliques in the
+system), but this is customizable.
+
+When building a clique graph, Calipso defaults to traversing all nodes
+edges (links) in the graph.
+
+In some cases we want to limit the graph so it will not expand too much
+(nor forever).
+
+For example: when we build the graph for a specific instance, we limit
+the graph to only take objects from the network on which this instance
+resides - otherwise the graph will show objects related to other
+instances.
+
+The constraint validation is done by checking value V from the focal
+point F on the links.
+
+For example, if an n instance has network X, we check that each link we
+use either has network X (attribute “network” has value X), or does not
+have the “network” attribute.
+
+Connection\_tests
+-----------------
+
+This collection keeps requests from the UI to test the different
+adapters (API, DB, CLI etc) connections to the underlying VIM, making
+sure dynamic and real-time data is maintained.
+
+Messages
+--------
+
+Aggregates all loggings from the different systems, source\_system of
+logs currently defined as “OpenStack” (the VIM), “Sensu” (the Monitoring
+module) and “Calipso” (logs of the application itself. Messages have 6
+levels of severity and can be browsed in the UI and through Calipso API.
+
+Network\_agent\_types
+---------------------
+
+Lists the types of networking agents supported on the VIM (per
+distribution and version).
+
+Roles, Users
+------------
+
+Basic RBAC facility to authorize calispo UI users for certain calipso
+functionalities on the UI.
+
+Statistics
+----------
+
+Built for detailed analysis and future functionalities, used today for
+traffic analysis (capturing samples of throughputs per session on VPP
+based environments).
+
+Constants
+---------
+
+This is an aggregated collection for many types of documents that are
+required mostly by the UI and basic functionality on some scanning
+classes (‘fetchers’).
+
+Constants-env\_types
+--------------------
+
+Type of environments to allow for configuration on sensu monitoring
+framework.
+
+Constants-log\_levels
+---------------------
+
+Severity levels for messages generated.
+
+Constants-mechanism\_drivers
+----------------------------
+
+Mechanism-drivers allowed for UI users.
+
+Constants-type\_drivers
+-----------------------
+
+Type-drivers allowed for UI users.
+
+Constants-environment\_monitoring\_types
+----------------------------------------
+
+Currently only “Sensu” is available, might be used for other monitoring
+systems integrations.
+
+Constants-link\_states
+----------------------
+
+Provides statuses for link objects, based on monitoring results.
+
+Constants-environment\_provision\_types
+---------------------------------------
+
+The types of deployment options available for monitoring (see
+monitoring-guide for details).
+
+Constants-environment\_operational\_status
+------------------------------------------
+
+Captures the overall (aggregated) status of a curtained environment.
+
+Constants-link\_types
+---------------------
+
+Lists the connections and relationships options for objects in the
+inventory.
+
+Constants-monitoring\_sides
+---------------------------
+
+Used for monitoring auto configurations of clients and servers.
+
+Constants-object\_types
+-----------------------
+
+Lists the type of objects supported through scanning (inventory
+objects).
+
+Constants-scans\_statuses
+-------------------------
+
+During scans, several statuses are shown on the UI, based on the
+specific stage and results.
+
+Constants-distributions
+-----------------------
+
+Lists the VIM distributions.
+
+Constants-distribution\_versions
+--------------------------------
+
+Lists the VIM different versions of different distributions.
+
+Constants-message\_source\_systems
+----------------------------------
+
+The list of systems that can generate logs and messages.
+
+Constants-object\_types\_for\_links
+-----------------------------------
+
+Object\_types used only for link popups on UI.
+
+Constants-scan\_object\_types
+-----------------------------
+
+Object\_types used during scanning, see development-guide for details.
+
+.. |image0| image:: media/image1.png
+ :width: 6.50000in
+ :height: 4.27153in
+.. |image1| image:: media/image8.png
+ :width: 6.50000in
+ :height: 2.43750in
+.. |image2| image:: media/image9.png
+ :width: 7.02325in
+ :height: 5.22917in