aboutsummaryrefslogtreecommitdiffstats
path: root/docs/release/developer-guide.rst
diff options
context:
space:
mode:
Diffstat (limited to 'docs/release/developer-guide.rst')
-rw-r--r--docs/release/developer-guide.rst1338
1 files changed, 1338 insertions, 0 deletions
diff --git a/docs/release/developer-guide.rst b/docs/release/developer-guide.rst
new file mode 100644
index 0000000..0de3f57
--- /dev/null
+++ b/docs/release/developer-guide.rst
@@ -0,0 +1,1338 @@
+| Calipso
+| Developer Guide
+
+|image0|
+
+Project “Calipso” tries to illuminate complex virtual networking with
+real time operational state visibility for large and highly distributed
+Virtual Infrastructure Management (VIM).
+
+We believe that Stability is driven by accurate Visibility.
+
+Calipso provides visible insights using smart discovery and virtual
+topological representation in graphs, with monitoring per object in the
+graph inventory to reduce error vectors and troubleshooting, maintenance
+cycles for VIM operators and administrators.
+
+Project architecture
+====================
+
+Calipso comprises two major parts: application and UI. We’ll focus on
+the former in this developer guide.
+
+Current project structure is as follows:
+
+- root/
+
+ - app/
+
+ - api/
+
+ - responders/
+
+ - auth/
+
+ - resource/
+
+ - *server.py*
+
+ - config/
+
+ - *events.json*
+
+ - *scanners.json*
+
+ - discover/
+
+ - events/
+
+ - listeners/
+
+ - *default\_listener.py*
+
+ - *listener\_base.py*
+
+ - handlers/
+
+ - *event\_base.py*
+
+ - *event\_\*.py*
+
+ - fetchers/
+
+ - aci/
+
+ - api/
+
+ - cli/
+
+ - db/
+
+ - *event\_manager.py*
+
+ - *scan.py*
+
+ - *scan\_manager.py*
+
+ - monitoring/
+
+ - checks/
+
+ - handlers/
+
+ - *monitor.py*
+
+ - setup/
+
+ - *monitoring\_setup\_manager.py*
+
+ - test/
+
+ - api/
+
+ - event\_based\_scan/
+
+ - fetch/
+
+ - scan/
+
+ - utils/
+
+ - ui/
+
+Application structure
+---------------------
+
+‘API’ package
+~~~~~~~~~~~~~
+
+Calipso API is designed to be used by native and third-party
+applications that are planning to use Calipso discovery application.
+
+***api/responders***
+
+This package contains all exposed API endpoint handlers:
+
+*auth* package contains token management handlers,
+
+*resource* package contains resource handlers.
+
+***server.py***
+
+API server startup script. In order for it to work correctly, connection
+arguments for a Mongo database used by a Calipso application instance
+are required:
+
+-m [MONGO\_CONFIG], --mongo\_config [MONGO\_CONFIG]
+
+name of config file with mongo access details
+
+--ldap\_config [LDAP\_CONFIG]
+
+name of the config file with ldap server config
+
+details
+
+-l [LOGLEVEL], --loglevel [LOGLEVEL]
+
+logging level (default: 'INFO')
+
+-b [BIND], --bind [BIND]
+
+binding address of the API server (default
+
+127.0.0.1:8000)
+
+-y [INVENTORY], --inventory [INVENTORY]
+
+name of inventory collection (default: 'inventory')
+
+-t [TOKEN\_LIFETIME], --token-lifetime [TOKEN\_LIFETIME]
+
+lifetime of the token
+
+For detailed reference and endpoints guide, see the API Guide document.
+
+‘Discover’ package
+~~~~~~~~~~~~~~~~~~
+
+‘Discover’ package contains the core Calipso functionality which
+involves:
+
+- scanning a network topology using a defined suite of scanners (see
+ `Scanning concepts <#scanning-concepts>`__, `Scanners configuration
+ file structure <#the-scanners-configuration-file-structure>`__) that
+ use fetchers to get all needed data on objects of the topology;
+
+- tracking live events that modifies the topology in any way (by adding
+ new object, updating existing or deleting them) using a suite of
+ event handlers and event listeners;
+
+- managing the aforementioned suites using specialized manager scripts
+ (*scan\_manager.py* and *event\_manager.py*)
+
+‘Tests’ package
+~~~~~~~~~~~~~~~
+
+‘Tests’ package contains unit tests for main Calipso components: API,
+event handlers, fetchers, scanners and utils.
+
+Other packages
+~~~~~~~~~~~~~~
+
+***Install***
+
+Installation and deployment scripts (with initial data for Calipso
+database).
+
+***Monitoring***
+
+Monitoring configurations, checks and handlers (see
+`Monitoring <#monitoring>`__ section and Monitoring Guide document).
+
+***Utils***
+
+Utility modules for app-wide use (inventory manager, mongo access,
+loggers, etc.).
+
+Scanning Guide
+==============
+
+ Introduction to scanning
+-------------------------
+
+Architecture overview
+~~~~~~~~~~~~~~~~~~~~~
+
+Calipso backend will scan any OpenStack environment to discover the
+objects that it is made of, and place the objects it discovered in a
+MongoDB database.
+
+Following discovery of objects, Calipso will:
+
+| Find what links exist between these objects, and save these links to
+ MongoDB as well.
+| For example, it will create a pnic-network link from a pNIC (physical
+ NIC) and the network it is connected to.
+
+Based on user definitions, it will create a 'clique' for each object
+using the links it previously found. These cliques are later used to
+present graphs for objects being viewed in the Calipso UI. This is not a
+clique by graph theory definition, but more like the social definition
+of clique: a graph of related, interconnected nodes.
+
+ 
+
+OpenStack Scanning is done using the following methods, in order of
+preference:
+
+1. OpenStack API
+
+2. MySQL DB - fetch any extra detail we can from the infrastructure
+ MySQL DB used by OpenStack
+
+3. CLI - connect by SSH to the hosts in the OpenStack environment to run
+ commands, e.g. ifconfig, that will provide the most in-depth details.
+
+ 
+
+| *Note*: 'environment' in Calipso means a single deployment of
+ OpenStack, possibly containing multiple tenants (projects), hosts and
+ instances (VMs). A single Calipso instance can handle multiple
+ OpenStack environments. 
+| However, we expect that typically Calipso will reside inside an
+ OpenStack control node and will handle just that node's OpenStack
+ environment.
+
+ 
+
+***Environment***
+
+| The Calipso scan script, written in Python, is called scan.py.
+| It uses Python 3, along with the following libraries:
+
+- pymongo - for MongoDB access
+
+- mysql-connector - For MySQL DB access
+
+- paramiko - for SSH access
+
+- requests - For handling HTTP requests and responses to the OpenStack
+ API
+
+- xmltodict - for handling XML output of CLI commands
+
+- cryptography - used by Paramiko
+
+See Calipso installation guide for environment setup instructions.
+
+***Configuration***
+
+The configuration for accessing the OpenStack environment, by API, DB or
+SSH, is saved in the Calipso MongoDB *“environments\_config”*
+collection.
+
+Calipso can work with a remote MongoDB instance, the details of which
+are read from a configuration file (default: */etc/calipso/mongo.conf*).
+
+| The first column is the configuration key while the second is the
+ configuration value, in the case the value is the server host name or
+ IP address.
+| Other possible keys for MongoDB access:
+
+- port: IP port number
+
+- Other parameters for the PyMongo MongoClient class constructor
+
+Alternate file location can be specified using the CLI -m parameter.
+
+Scanning concepts
+~~~~~~~~~~~~~~~~~
+
+***DB Schema***
+
+Objects are stored in the inventory collection, named *“inventory”* by
+default, along with the accompanying collections, named by
+default: \ *"links", "cliques", "clique\_types" and
+"clique\_constraints"*. For development, separate sets of collections
+can be defined per environment (collection names are created by
+appending the default collection name to the alternative inventory
+collection name).
+
+The inventory, links and cliques collections are all designed to work
+with a multi-environment scenario, so documents are marked with an
+*"environment"* attribute.
+
+The clique\_types collection allows Calipso users (typically
+administrators) to define how the "clique" graphs are to be defined. 
+
+It defines a set of link types to be traversed when an object such as an
+instance is clicked in the UI (therefore referred to as the focal
+point). See "Clique Scanning" below. This definition can differ between
+environments.
+
+Example: for focal point type "instance", the link types are often set
+to
+
+- instance-vnic
+
+- vnic-vconnector
+
+- vconnector-vedge
+
+- vedge-pnic
+
+- pnic-network
+
+| The clique\_constraints collection defines a constraint on links
+ traversed for a specific clique when starting from a given focal
+ point. 
+| For example: instance cliques are constrained to a specific
+ network. If we wouldn't have this constraint, the resulting graph
+ would stretch to include objects from neighboring networks that are
+ not really related to the instance.
+
+\ ***Hierarchy of Scanning***
+
+The initial scanning is done hierarchically, starting from the
+environment level and discovering lower levels in turn.
+
+Examples:
+
+- Under environment we scan for regions and projects (tenants).
+
+- Under availability zone we have hosts, and under hosts we have
+ instances and host services
+
+The actual scanning order is not always same as the logical hierarchical
+order of objects, to improve scanning performance.
+
+Some objects are referenced multiple times in the hierarchy. For
+example, hosts are always in an availability zone, but can also be part
+of a host aggregate. Such extra references are saved as references to
+the main object.
+
+***Clique Scanning***
+
+| For creating cliques based on the discovered objects and links, clique
+ types need to be defined for the given environment.
+| A clique type specifies the list of link types used in building a
+ clique for a specific focal point object type.
+
+For example, it can define that for instance objects we want to have the
+following link types:
+
+- instance-vnic
+
+- vnic-vconnector
+
+- vconnector-vedge
+
+- vedge-pnic
+
+- pnic-network
+
+ 
+
+As in many cases the same clique types are used, default clique types
+will be provided with a new Calipso deployment.
+
+\ ***Clique creation algorithm***
+
+- For each clique type CT:
+
+ - For each focal point object F of the type specified as the clique
+ type focal point type:
+
+ - Create a new clique C
+
+ - Add F to the list of objects included in the clique
+
+ - For each link type X-Y of the link types in CT:
+
+ - Find all the source objects of type x that are already
+ included in the clique
+
+ - For each such source object S:
+
+ - for all links L of type X-Y that have S as their source
+
+ - Add the object T of type Y that is the target in L to the
+ list of objects included in the clique
+
+ - Add L to the list of links in the clique C
+
+How to run scans
+----------------
+
+For running environment scans Calipso uses a specialized daemon script
+called *scan manager*. If Calipso application is deployed in docker
+containers, scan manager will run inside the *calipso-scan* container.
+
+Scan manager uses MongoDB connection to fetch requests for environment
+scans and execute them by running a *scan* script. It also performs
+extra checks and procedures connected to scan failure/completion, such
+as marking *environment* as scanned and reporting errors (see
+`details <#scan-manager>`__).
+
+Scan script workflow:
+
+1. Loads specific scanners definitions from a predefined metadata file
+ (which can be extended in order to support scanning of new object
+ types).
+
+2. Runs the root scanner and then children scanners recursively (see
+ `Hierarchy of scanning <#Hierarchy_of_scanning>`__)
+
+ a. Scanners do all necessary work to insert objects in *inventory*.
+
+3. Finalizes the scan and publishes successful scan completion.
+
+Scan manager
+~~~~~~~~~~~~
+
+Scan manager is a script which purpose is to manage the full lifecycle
+of scans requested through API. It runs indefinitely while:
+
+1. Polling the database (*scans* and *scheduled\_scans* collections) for
+ new and scheduled scan requests;
+
+2. Parsing their configurations;
+
+3. Running the scans;
+
+4. Logging the results.
+
+Scan manager can be run in a separate container provided that it has
+connection to the database and the topology source system.
+
+Monitoring
+----------
+
+***Monitoring Subsystem Overview***
+
+Calipso monitoring uses Sensu to remotely track actual state of hosts.
+
+A Sensu server is installed as a Docker image along with the other
+Calipso components.
+
+ 
+
+Remote hosts send check events to the Sensu server. 
+
+We use a filtering of events such that the first occurrence of a check
+is always used, after that cases where status is unchanged are ignored.
+
+When handling a check event, the Calipso Sensu handlers will find the
+matching Calipso object, and update its status.
+
+We also keep the timestamp of the last status update, along with the
+full check output.
+
+Setup of checks and handlers code on the server and the remote hosts can
+be done by Calipso. It is also possible to have this done using another
+tool, e.g. Ansible or Puppet.
+
+More info is available in Monitoring Guide document.
+
+ 
+
+***Package Structure***
+
+Monitoring package is divided like this:
+
+1.      Checks: these are the actual check scripts that will be run on
+the hosts;
+
+2.      Handlers: the code that does handling of check events;
+
+3.      Setup: code for setting up handlers and checks.
+
+Events Guide
+============
+
+Introduction
+------------
+
+Events
+~~~~~~
+
+Events in general sense are any changes to the monitored topology
+objects that are trackable by Calipso. We currently support subscription
+to Neutron notification queues for several OpenStack distributions as a
+source of events.
+
+The two core concepts of working with events are *listening to events*
+and *event handling*, so the main module groups in play are the *event
+listener* and *event handlers*.
+
+Event listeners
+~~~~~~~~~~~~~~~
+
+An event listener is a module that handles connection to the event
+source, listening to the new events and routing them to respective event
+handlers.
+
+An event listener class should be designed to run indefinitely in
+foreground or background (daemon) while maintaining a connection to the
+source of events (generally a message queue like RabbitMQ or Apache
+Kafka). Each incoming event is examined and, if it has the correct
+format, is routed to the corresponding event handler class. The routing
+can be accomplished through a dedicated event router class using a
+metadata file and a metadata parser (see `Metadata
+parsers <#metadata-parsers>`__).
+
+Event handlers
+~~~~~~~~~~~~~~
+
+An event handler is a specific class that parses the incoming event
+payload and performs a certain CUD (Create/Update/Delete) operation on
+zero or more database objects. Event handler should be independent of
+the event listener implementation.
+
+Event manager
+~~~~~~~~~~~~~
+
+Event manager is a script which purpose is to manage event listeners. It
+runs indefinitely and performs the following operations:
+
+1. Starts a process for each valid entry in *environments\_config*
+ collection that is scanned (*scanned == true*) and has the *listen*
+ flag set to *true*;
+
+2. Checks the *operational* statuses of event listeners and updating
+ them in *environments\_config* collection;
+
+3. Stops the event listeners that no longer qualify for listening (see
+ step 1);
+
+4. Restarts the event listeners that quit unexpectedly;
+
+5. Repeats steps 1-5
+
+Event manager can be run in a separate container provided that it has
+connection to the database and to all events source systems that event
+listeners use.
+
+Contribution
+~~~~~~~~~~~~
+
+You can contribute to Calipso *events* system in several ways:
+
+- create custom event handlers for an existing listener;
+
+- create custom event listeners and reuse existing handlers;
+
+- create custom event handlers and listeners.
+
+See `Creating new event handlers <#creating-new-event-handlers>`__ and
+`Creating new event listeners <#creating-new-event-listeners>`__ for the
+respective guides.
+
+Contribution
+============
+
+This section covers the designed approach to contribution to Calipso.
+
+The main scenario of contribution consists of introducing a new *object*
+type to the discovery engine, defining *links* that connect this new
+object to existing ones, and describing a *clique* (or cliques) that
+makes use of the object and its links. Below we describe how this
+scenario should be implemented, step-by-step.
+
+*Note*: Before writing any new code, you need to create your own
+environment using UI (see User Guide document) or API (see the API guide
+doc). Creating an entry directly in *“environments\_config”* collection
+is not recommended.
+
+Creating new object types
+-------------------------
+
+Before you proceed with creation of new object type, you need to make
+sure the following requirements are met:
+
+- New object type has a unique name and purpose
+
+- New object type has an existing parent object type
+
+First of all, you need to create a fetcher that will take care of
+getting info on objects of the new type, processing it and adding new
+entries in Calipso database.
+
+Creating new fetchers
+~~~~~~~~~~~~~~~~~~~~~
+
+A fetcher is a common name for a class that handles fetching of all
+objects of a certain type that have a common parent object. The source
+of this data may be already implemented in Calipso (like OpenStack API,
+CLI and DB sources) or you may create one yourself.
+
+***Common fetchers***
+
+Fetchers package structure should adhere to the following pattern (where
+*%source\_name%* is a short prefix like *api, cli, db*):
+
+- app
+
+ - discover
+
+ - fetchers
+
+ - *%source\_name%*
+
+ - *%source\_name%*\ \_\ *%fetcher\_name%.*\ py
+
+If you reuse the existing data source, your new fetcher should subclass
+the class located in *%source\_name%\_access* module inside the
+*%source\_name%* directory.
+
+Fetcher class name should repeat the module name, except in CamelCase
+instead of snake\_case.
+
+Example: if you are adding a new cli fetcher, you should subclass
+*CliAccess* class found by *app/discover/fetchers/cli/cli\_access.py*
+path. If the module is named *cli\_fetch\_new\_objects.py*, fetcher
+class should be named *CliFetchNewObjects*.
+
+If you are creating a fetcher that uses new data source, you may
+consider adding an “access” class for this data source to store
+convenience methods. In this case, the “access” class should subclass
+the base Fetcher class (found in *app/discover/fetcher.py*) and the
+fetcher class should subclass the “access” class.
+
+All business logic of a fetcher should be defined inside the overridden
+method from base Fetcher class *get(self, parent\_id)*. You should use
+the second argument that is automatically passed by parent scanner to
+get the parent entity from database and any data you may need. This
+method has to return a list of new objects (dicts) that need to be
+inserted in Calipso database. Their parent object should be passed along
+other fields (see example).
+
+*Note*: types of returned objects should match the one their fetcher is
+designed for.
+
+***Example***:
+
+**app/discover/fetchers/cli/cli\_fetch\_new\_objects.py**
+
+ | **from** discover.fetchers.cli.cli\_access **import** CliAccess
+ | **from** utils.inventory\_mgr **import** InventoryMgr
+ | **class** CliFetchNewObjects(CliAccess):
+ | **def** \_\_init\_\_(self):
+ | super().\_\_init\_\_()
+ | self.inv = InventoryMgr()
+ | **def** get(self, parent\_id):
+ | parent = self.inv.get\_by\_id(self.env, parent\_id)
+ | *# do something
+ *\ objects = [{**"type"**: **"new\_type"**, **"id"**: **"1234"**,
+ **"parent"**: parent},
+ | {**"type"**: **"new\_type"**, **"id"**: **"2345"**, **"parent"**:
+ parent}]
+ | **return** objects
+
+This is an example of a fetcher that deals with the objects of type
+*“new\_type”*. It uses the parent id to fetch the parent object, then
+performs some operations in order to fetch the new objects and
+ultimately returns the objects list, at which point it has gathered all
+required information.
+
+\ ***Folder fetcher***
+
+A special type of fetchers is the folder fetcher. It serves as a dummy
+object used to aggregate objects in a specific point in objects
+hierarchy. If you would like to logically separate children objects from
+parent, you may use folder fetcher found at
+*app/discover/fetchers/folder\_fetcher.py*.
+
+Usage is described `here <#Folder_scanner>`__.
+
+The scanners configuration file structure
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**Scanners.json** (full path *app/config/scanners.json*) is an essential
+configuration file that defines scanners hierarchy. It has a forest
+structure, meaning that it is a set of trees, where each tree has a
+*root* scanner, potentially many levels of *children* scanners and
+pointers from parent scanners to children scanners. Scanning hierarchy
+is described `here <#Hierarchy_of_scanning>`__.
+
+A scanner is essentially a list of fetchers with configuration (we’ll
+call those **Fetch types**). Fetch types can be **Simple** and
+**Folder**, described below.
+
+***Simple fetch type***
+
+A simple fetch type looks like this:
+
+ | {
+ | **"type"**: **"project"**,
+ | **"fetcher"**: **"ApiFetchProjects"**,
+ | **"object\_id\_to\_use\_in\_child"**: **"name"**,
+
+ | **"environment\_condition"**: {
+ | **"mechanism\_drivers"**: **"VPP"
+ ** },
+ | **"children\_scanner"**: **"ScanProject"
+ **}
+
+Supported fields include:
+
+- *“fetcher”* – class name of fetcher that the scanner uses;
+
+- *“type”* – object type that the fetcher works with;
+
+- *“children\_scanner”* – (optional) full name of a scanner that should
+ run after current one finishes;
+
+- *“environment\_condition”* – (optional) specific constraints that
+ should be checked against the environment in *environments\_config*
+ collection before execution;
+
+- *“object\_id\_to\_use\_in\_child*\ ” – (optional) which parent field
+ should be passed as parent id to the fetcher (default: “id”).
+
+ \ ***Folder fetch type***
+
+Folder fetch types deal with folder fetchers (described
+`here <#Folder_fetcher>`__) and have a slightly different structure:
+
+ | {
+ | **"type"**: **"aggregates\_folder"**,
+ | **"fetcher"**: {
+ | **"folder"**: **true**,
+ | **"types\_name"**: **"aggregates"**,
+ | **"parent\_type"**: **"region"
+ **},
+
+ **"object\_id\_to\_use\_in\_child"**: **"name"**,
+
+ | **"environment\_condition"**: {
+ | **"mechanism\_drivers"**: **"VPP"
+ ** },
+ | **"children\_scanner"**: **"ScanAggregatesRoot"
+ **}
+
+The only difference is that *“fetcher”* field is now a dictionary with
+the following fields:
+
+- *“folder”* – should always be **true**;
+
+- *“types\_name”* – type name in plural (with added ‘s’) of objects
+ that serve as folder’s children
+
+- *“parent\_type”* – folder’s parent type (basically the parent type of
+ folder’s objects).
+
+Updating scanners
+~~~~~~~~~~~~~~~~~
+
+After creating a new fetcher, you should integrate it into scanners
+hierarchy. There are several possible courses of action:
+
+***Add new scanner as a child of an existing one***
+
+If the parent type of your newly added object type already has a
+scanner, you can add your new scanner as a child of an existing one.
+There are two ways to do that:
+
+1. Add new scanner as a *“children\_scanner”* field to parent scanner
+
+ ***Example***
+
+ Before:
+
+ | **"ScanHost"**: [
+ | {
+ | **"type"**: **"host"**,
+ | **"fetcher"**: **"ApiFetchProjectHosts"**,
+ | }
+ | ],
+
+ After:
+
+ | **"ScanHost"**: [
+ | {
+ | **"type"**: **"host"**,
+ | **"fetcher"**: **"ApiFetchProjectHosts"**,
+ | **"children\_scanner"**: **"NewTypeScanner"
+ **}
+ | ],
+ | **"NewTypeScanner"**: [
+ | {
+ | **"type"**: **"new\_type"**,
+ | **"fetcher"**: **"CliFetchNewType"
+ **}
+ | ]
+
+1. Add new fetch type to parent scanner (in case if children scanner
+ already exists)
+
+ ***Example***
+
+ Before:
+
+ | **"ScanHost"**: [
+ | {
+ | **"type"**: **"host"**,
+ | **"fetcher"**: **"ApiFetchProjectHosts"**,
+ | **"children\_scanner"**: **"ScanHostPnic"
+ **}
+ | ],
+
+ After:
+
+ | **"ScanHost"**: [
+ | {
+ | **"type"**: **"host"**,
+ | **"fetcher"**: **"ApiFetchProjectHosts"**,
+ | **"children\_scanner"**: **"ScanHostPnic"
+ **},
+ | {
+ | **"type"**: **"new\_type"**,
+ | **"fetcher"**: **"CliFetchNewType"
+ **}
+ | ],
+
+***Add new scanner and set an existing one as a child***
+
+***Example***
+
+ Before:
+
+ | **"ScanHost"**: [
+ | {
+ | **"type"**: **"host"**,
+ | **"fetcher"**: **"ApiFetchProjectHosts"**,
+ | **"children\_scanner"**: **"ScanHostPnic"
+ **}
+ | ],
+
+ After:
+
+ | **"NewTypeScanner"**: [
+ | {
+ | **"type"**: **"new\_type"**,
+ | **"fetcher"**: **"CliFetchNewType"**,
+ | **"children\_scanner"**: **"ScanHost"
+ **}
+ | ]
+
+ | **"ScanHost"**: [
+ | {
+ | **"type"**: **"host"**,
+ | **"fetcher"**: **"ApiFetchProjectHosts"**,
+ | **"children\_scanner"**: **"ScanHostPnic"
+ **}
+ | ],
+
+***Other cases***
+
+You may choose to combine approaches or use none of them and create an
+isolated scanner if needed.
+
+Updating constants collection
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Before testing your new scanner and fetcher you need to add the newly
+created object type to *“constants”* collection in Calipso database:
+
+1. **constants.object\_types** document
+
+ Append a *{“value”: “new\_type”, “label”: “new\_type”}* object to
+ **data** list.
+
+1. **constants.scan\_object\_types** document
+
+ Append a *{“value”: “new\_type”, “label”: “new\_type”}* object to
+ **data** list.
+
+1. **constants.object\_types\_for\_links** document
+
+ If you’re planning to build links using this object type (you
+ probably are), append a *{“value”: “new\_type”, “label”:
+ “new\_type”}* object to **data** list.
+
+Setting up monitoring
+~~~~~~~~~~~~~~~~~~~~~
+
+In order to setup monitoring for the new object type you have defined,
+you’ll need to add a Sensu check:
+
+1. Add a check script in app/monitoring/checks:
+
+ a. | Checks should return the following values:
+ | 0: **OK**
+ | 1: **Warning**
+ | 2: **Error**
+
+ b. Checks can print the underlying query results to stdout. Do so
+ within reason, as this output is later stored in the DB, so avoid
+ giving too much output;
+
+ c. Test your script on a remote host:
+
+ i. Write it in */etc/sensu/plugins* directory;
+
+ ii. Update the Sensu configuration on the remote host to run this
+ check;
+
+ iii. Add the check in the “checks” section of
+ */etc/sensu/conf.d/client.json*;
+
+ iv. The name under which you save the check will be used by the
+ handler to determine the DB object that it relates to;
+
+ v. Restart the client with the command: *sudo service
+ sensu-client restart*;
+
+ vi. Check the client log file to see the check is run and
+ produces the expected output (in */var/log/sensu* directory).
+
+ d. Add the script to the source directory (*app/monitoring/checks*).
+
+2. Add a handler in app/monitoring/handlers:
+
+ a. If you use a standard check naming scheme and check an object, the
+ *BasicCheckHandler* can take care of this, but add the object type
+ in *basic\_handling\_types* list in *get\_handler()*;
+
+ b. If you have a more complex naming scheme, override
+ MonitoringCheckHandler. See HandleOtep for example.
+
+3. If you deploy monitoring using Calipso:
+
+ a. Add the check in the *monitoring\_config\_templates* collection.
+
+*Check Naming*
+
+The check name should start with the type of the related object,
+followed by an underscore (“\_”). For example, the name for a check
+related to an OTEP (type “otep”)  will start with “otep\_“. It should
+then be followed by the object ID.
+
+ 
+
+For checks related to links, the check name will have this
+format:
link\_<link type>\_<from\_id>\_<to\_id>
+
+Creating new link types
+-----------------------
+
+After you’ve added a new object type you may consider adding new link
+types to connect objects of new type to existing objects in topology.
+Your new object type may serve as a *source* and/or *target* type for
+the new link type.
+
+The process of new link type creation includes several steps:
+
+1. Write a link finder class;
+
+2. Add the link finder class to the link finders configuration file;
+
+3. Update *“constants”* collection with the new link types.
+
+Writing link finder classes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A new link finder class should:
+
+1. Subclass *app.discover.link\_finders.FindLinks* class;
+
+2. Be located in the *app.discover.link\_finders* package;
+
+3. Define an instance method called *add\_links(self)* with no
+ additional arguments. This method is the only entry point for link
+ finder classes.
+
+*FindLinks* class provides access to inventory manager to its subclasses
+which they should use to their advantage. It also provides a convenience
+method *create\_links(self, …)* for saving links to database. It is
+reasonable to call this method at the end of *add\_links* method.
+
+You may opt to add more than one link type at a time in a single link
+finder.
+
+***Example***
+
+ | **from** discover.find\_links **import** FindLinks
+ | **class** FindLinksForNewType(FindLinks):
+ | **def** add\_links(self):
+ | new\_objects = self.inv.find\_items({\ **"environment"**:
+ self.get\_env(),
+ | **"type"**: **"new\_type"**})
+ | **for** new\_object **in** new\_objects:
+ | old\_object = self.inv.get\_by\_id(environment=self.get\_env(),
+ | item\_id=new\_object[**"old\_object\_id"**])
+ | link\_type = **"old\_type-new\_type"
+ **\ link\_name = **"{}-{}"**.format(old\_object[**"name"**],
+ new\_object[**"name"**])
+ | state = **"up"** *# TBD
+ *\ link\_weight = 0 *# TBD
+ *\ self.create\_link(env=self.get\_env(),
+ | source=old\_object[**"\_id"**],
+ | source\_id=old\_object[**"id"**],
+ | target=new\_object[**"\_id"**],
+ | target\_id=new\_object[**"id"**],
+ | link\_type=link\_type,
+ | link\_name=link\_name,
+ | state=state,
+ | link\_weight=link\_weight)
+
+Updating the link finders configuration file
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Default link finders configuration file can be found at
+*/app/config/link\_finders.json* and has the following structure:
+
+ | {
+ | **"finders\_package"**: **"discover.link\_finders"**,
+ | **"base\_finder"**: **"FindLinks"**,
+ | **"link\_finders"**: [
+ | **"FindLinksForInstanceVnics"**,
+ | **"FindLinksForOteps"**,
+ | **"FindLinksForPnics"**,
+ | **"FindLinksForVconnectors"**,
+ | **"FindLinksForVedges"**,
+ | **"FindLinksForVserviceVnics"
+ **]
+ | }
+
+File contents:
+
+- *finders\_package* – python path to the package that contains link
+ finders (relative to $PYTHONPATH environment variable);
+
+- *base\_finder* – base link finder class name;
+
+- *link\_finders* – class names of actual link finders.
+
+If your new fetcher meets the requirements described in `Writing link
+finder classes <#writing-link-finder-classes>`__ section, you can append
+its name to the *“link\_finders”* list in *link\_finders.json* file.
+
+Updating constants collection
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Before testing your new links finder, you need to add the newly created
+link types to *“constants”* collection in Calipso database:
+
+1. **constants.link\_types** document
+
+ Append a *{“value”: “source\_type-target\_type”, “label”:
+ “source\_type-target\_type”}* object to **data** list for each new
+ link type.
+
+Creating custom link finders configuration file
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you consider writing a custom list finders configuration file, you
+should also follow the guidelines from 4.2.1-4.2.3 while designing link
+finder classes and including them in the new link finders source file.
+
+The general approach is the following:
+
+1. Custom configuration file should have the same key structure with the
+ basic one;
+
+2. You should create a *base\_finder* class that subclasses the basic
+ FindLinks class (see `Writing link finder
+ classes <#writing-link-finder-classes>`__);
+
+3. Your link finder classes should be located in the same package with
+ your *base\_finder* class;
+
+4. Your link finder classes should subclass your *base\_finder* class
+ and override the *add\_links(self)* method.
+
+Creating new clique types
+-------------------------
+
+Two steps in creating new clique types and including them in clique
+finder are:
+
+1. Designing new clique types
+
+2. Updating clique types collection
+
+Designing new clique types
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A clique type is basically a list of links that will be traversed during
+clique scans (see `Clique creation algorithm <#clique_creation>`__). The
+process of coming up with clique types involves general networking
+concepts knowledge as well as expertise in monitored system details
+(e.g. OpenStack distribution specifics). In a nutshell, it is not a
+trivial process, so the clique design should be considered carefully.
+
+The predefined clique types (in *clique\_types* collection) may give you
+some idea about the rationale behind clique design.
+
+Updating clique types collection
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+After designing the new clique type, you need to update the
+*clique\_types* collection in order for the clique finder to use it. For
+this purpose, you should add a document of the following structure:
+
+ { 
+
+     "environment": "ANY", 
+
+     "link\_types": [
+
+         "instance-vnic", 
+
+         "vnic-vconnector", 
+
+         "vconnector-vedge", 
+
+         "vedge-otep", 
+
+         "otep-vconnector", 
+
+         "vconnector-host\_pnic", 
+
+         "host\_pnic-network"
+
+     ], 
+
+     "name": "instance", 
+
+     "focal\_point\_type": "instance"
+
+ }
+
+Document fields are:
+
+- *environment* – can either hold the environment name, for which the
+ new clique type is designed, or **“ANY”** if the new clique type
+ should be added to all environments;
+
+- *name* – display name for the new clique type;
+
+- *focal\_point\_type* – the aggregate object type for the new clique
+ type to use as a starting point;
+
+- *link\_types* – a list of links that constitute the new clique type.
+
+Creating new event handlers
+---------------------------
+
+There are three steps to creating a new event handler:
+
+1. Determining *event types* that will be handled by the new handler;
+
+2. Writing the new handler module and class;
+
+3. Adding the (event type -> handler) mapping to the event handlers
+ configuration file.
+
+Writing custom handler classes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each event handler should adhere to the following design:
+
+1. Event handler class should subclass
+ the app.discover.events.event\_base.EventBase class;
+
+2. Event handler class should override handle method of EventBase.
+ Business logic of the event handler should be placed inside
+ the handle method;
+
+ a. Handle method accepts two arguments: environment name (str) and
+ notification contents (dict). No other event data will be provided
+ to the method;
+
+ b. Handle method returns an EventResult object, which accepts the
+ following arguments in its constructor:
+
+ i. *result* (mandatory) - determines whether the event handling
+ was successful;
+
+ ii. *retry* (optional) - determines whether the message should be
+ put back in the queue in order to be processed later. This
+ argument is checked only if result was set to False;
+
+ iii. *message* (optional) - (Currently unused) a string comment on
+ handling status;
+
+ iv. *related\_object* (optional) – id of the object related to
+ the handled event;
+
+ v. *display\_context* (optional) – (Calipso UI requirement).
+
+3. Module containing event handler class should have the same name as
+ the relevant handler class except translated
+ from UpperCamelCase to snake\_case.
+
+ ***Example:***
+
+ **app/discover/events/event\_new\_object\_add.py**
+
+ | **from** discover.events.event\_base **import** EventBase,
+ EventResult
+ | **class** EventNewObjectAdd(EventBase):
+ | **def** handle(self, env: str, notification: dict) -> EventResult:
+ | obj\_id = notification[**'payload'**][**'new\_object'**][**'id'**]
+ | obj = {
+ | **'id'**: obj\_id,
+ | **'type'**: **'new\_object'
+ **}
+ | self.inv.set(obj)
+ | **return** EventResult(result=\ **True**)
+
+ Modifications in *events.json*:
+
+ <...>
+
+ | **"event\_handlers"**: {
+ | <...>
+ | **"new\_object.create"**: **"EventNewObjectAdd"**,
+ | <...>**
+ **}
+
+ <...>
+
+After these changes are implemented, any event of type
+new\_object.create will be consumed by the event listener and the
+payload will be passed to EventNewObjectAdd handler which will insert a
+new document in the database.
+
+Event handlers configuration file structure
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**Events.json** (full path *app/config/events.json*) is a configuration
+file that contains information about events and event handlers,
+including:
+
+- Event subscription details (queues and exchanges for Neutron
+ notifications);
+
+- Location of event handlers package;
+
+- Mappings between event types and respective event handlers.
+
+The structure of *events.json* is as following:
+
+ | {
+ | **"handlers\_package"**: **"discover.events"**,
+ | **"queues"**: [
+ | {
+ | **"queue"**: **"notifications.nova"**,
+ | **"exchange"**: **"nova"
+ **},
+ | <…>
+ | ],
+ | **"event\_handlers"**: {
+ | **"compute.instance.create.end"**: **"EventInstanceAdd"**,
+ | **"compute.instance.update"**: **"EventInstanceUpdate"**,
+ | **"compute.instance.delete.end"**: **"EventInstanceDelete"**,
+ | **"network.create.end"**: **"EventNetworkAdd"**,
+
+ | <…>**
+ **}
+ | }
+
+The root object contains the following fields:
+
+- **handlers\_package** - python path to the package that contains
+ event handlers (relative to $PYTHONPATH environment variable)
+
+- **queues –** RabbitMQ queues and exchanges to consume messages from
+ (for Neutron notifications case)
+
+- **event\_handlers** – mappings of event types to the respective
+ handlers. The structure suggests that any event can have only one
+ handler.
+
+In order to add a new event handler to the configuration file, you
+should add another mapping to the event\_handlers object, where key is
+the event type being handled and value is the handler class name (module
+name will be determined automatically).
+
+If your event is being published to a queue and/or exchange that the
+listener is not subscribed to, you should add another entry to the
+queues list.
+
+Creating new event listeners
+----------------------------
+
+At the moment, the only guideline for creation of new event listeners is
+that they should subclass the *ListenerBase* class (full path
+*app/discover/events/listeners/listener\_base.py*) and override the
+*listen(self)* method that listens to incoming events indefinitely
+(until terminated by a signal).
+
+In future versions, a comprehensive guide to listeners structure is
+planned.
+
+Metadata parsers
+----------------
+
+Metadata parsers are specialized classes that are designed to verify
+metadata files (found in *app/*\ config directory), use data from them
+to load instances of implementation classes (e.g. scanners, event
+handlers, link finders) in memory, and supply them by request. Scanners
+and link finders configuration files are used in scanner, event handlers
+configuration file – in event listener.
+
+In order to create a new metadata parser, you should consider
+subclassing *MetadataParser* class (found in
+*app/utils/metadata\_parser.py*). *MetadataParser* supports parsing and
+validating of json files out of the box. Entry point for the class is
+the *parse\_metadata\_file* method, which requires the abstract
+*get\_required\_fields* method to be overridden in subclasses. This
+method should return a list of keys that the metadata file is required
+to contain.
+
+For different levels of customization you may consider:
+
+1. Overriding *validate\_metadata* method to provide more precise
+ validation of metadata;
+
+2. Overriding *parse\_metadata\_file* to provide custom metadata
+ handling required by your use case.
+
+.. |image0| image:: media/image1.png
+ :width: 6.50000in
+ :height: 4.27153in