| Calipso | Developer Guide |image0| Project “Calipso” tries to illuminate complex virtual networking with real time operational state visibility for large and highly distributed Virtual Infrastructure Management (VIM). We believe that Stability is driven by accurate Visibility. Calipso provides visible insights using smart discovery and virtual topological representation in graphs, with monitoring per object in the graph inventory to reduce error vectors and troubleshooting, maintenance cycles for VIM operators and administrators. Project architecture ==================== Calipso comprises two major parts: application and UI. We’ll focus on the former in this developer guide. Current project structure is as follows: - root/ - app/ - api/ - responders/ - auth/ - resource/ - *server.py* - config/ - *events.json* - *scanners.json* - discover/ - events/ - listeners/ - *default\_listener.py* - *listener\_base.py* - handlers/ - *event\_base.py* - *event\_\*.py* - fetchers/ - aci/ - api/ - cli/ - db/ - *event\_manager.py* - *scan.py* - *scan\_manager.py* - monitoring/ - checks/ - handlers/ - *monitor.py* - setup/ - *monitoring\_setup\_manager.py* - test/ - api/ - event\_based\_scan/ - fetch/ - scan/ - utils/ - ui/ Application structure --------------------- ‘API’ package ~~~~~~~~~~~~~ Calipso API is designed to be used by native and third-party applications that are planning to use Calipso discovery application. ***api/responders*** This package contains all exposed API endpoint handlers: *auth* package contains token management handlers, *resource* package contains resource handlers. ***server.py*** API server startup script. In order for it to work correctly, connection arguments for a Mongo database used by a Calipso application instance are required: -m [MONGO\_CONFIG], --mongo\_config [MONGO\_CONFIG] name of config file with mongo access details --ldap\_config [LDAP\_CONFIG] name of the config file with ldap server config details -l [LOGLEVEL], --loglevel [LOGLEVEL] logging level (default: 'INFO') -b [BIND], --bind [BIND] binding address of the API server (default 127.0.0.1:8000) -y [INVENTORY], --inventory [INVENTORY] name of inventory collection (default: 'inventory') -t [TOKEN\_LIFETIME], --token-lifetime [TOKEN\_LIFETIME] lifetime of the token For detailed reference and endpoints guide, see the API Guide document. ‘Discover’ package ~~~~~~~~~~~~~~~~~~ ‘Discover’ package contains the core Calipso functionality which involves: - scanning a network topology using a defined suite of scanners (see `Scanning concepts <#scanning-concepts>`__, `Scanners configuration file structure <#the-scanners-configuration-file-structure>`__) that use fetchers to get all needed data on objects of the topology; - tracking live events that modifies the topology in any way (by adding new object, updating existing or deleting them) using a suite of event handlers and event listeners; - managing the aforementioned suites using specialized manager scripts (*scan\_manager.py* and *event\_manager.py*) ‘Tests’ package ~~~~~~~~~~~~~~~ ‘Tests’ package contains unit tests for main Calipso components: API, event handlers, fetchers, scanners and utils. Other packages ~~~~~~~~~~~~~~ ***Install*** Installation and deployment scripts (with initial data for Calipso database). ***Monitoring*** Monitoring configurations, checks and handlers (see `Monitoring <#monitoring>`__ section and Monitoring Guide document). ***Utils*** Utility modules for app-wide use (inventory manager, mongo access, loggers, etc.). Scanning Guide ============== Introduction to scanning ------------------------- Architecture overview ~~~~~~~~~~~~~~~~~~~~~ Calipso backend will scan any OpenStack environment to discover the objects that it is made of, and place the objects it discovered in a MongoDB database. Following discovery of objects, Calipso will: | Find what links exist between these objects, and save these links to MongoDB as well. | For example, it will create a pnic-network link from a pNIC (physical NIC) and the network it is connected to. Based on user definitions, it will create a 'clique' for each object using the links it previously found. These cliques are later used to present graphs for objects being viewed in the Calipso UI. This is not a clique by graph theory definition, but more like the social definition of clique: a graph of related, interconnected nodes.   OpenStack Scanning is done using the following methods, in order of preference: 1. OpenStack API 2. MySQL DB - fetch any extra detail we can from the infrastructure MySQL DB used by OpenStack 3. CLI - connect by SSH to the hosts in the OpenStack environment to run commands, e.g. ifconfig, that will provide the most in-depth details.   | *Note*: 'environment' in Calipso means a single deployment of OpenStack, possibly containing multiple tenants (projects), hosts and instances (VMs). A single Calipso instance can handle multiple OpenStack environments.  | However, we expect that typically Calipso will reside inside an OpenStack control node and will handle just that node's OpenStack environment.   ***Environment*** | The Calipso scan script, written in Python, is called scan.py. | It uses Python 3, along with the following libraries: - pymongo - for MongoDB access - mysql-connector - For MySQL DB access - paramiko - for SSH access - requests - For handling HTTP requests and responses to the OpenStack API - xmltodict - for handling XML output of CLI commands - cryptography - used by Paramiko See Calipso installation guide for environment setup instructions. ***Configuration*** The configuration for accessing the OpenStack environment, by API, DB or SSH, is saved in the Calipso MongoDB *“environments\_config”* collection. Calipso can work with a remote MongoDB instance, the details of which are read from a configuration file (default: */etc/calipso/mongo.conf*). | The first column is the configuration key while the second is the configuration value, in the case the value is the server host name or IP address. | Other possible keys for MongoDB access: - port: IP port number - Other parameters for the PyMongo MongoClient class constructor Alternate file location can be specified using the CLI -m parameter. Scanning concepts ~~~~~~~~~~~~~~~~~ ***DB Schema*** Objects are stored in the inventory collection, named *“inventory”* by default, along with the accompanying collections, named by default: \ *"links", "cliques", "clique\_types" and "clique\_constraints"*. For development, separate sets of collections can be defined per environment (collection names are created by appending the default collection name to the alternative inventory collection name). The inventory, links and cliques collections are all designed to work with a multi-environment scenario, so documents are marked with an *"environment"* attribute. The clique\_types collection allows Calipso users (typically administrators) to define how the "clique" graphs are to be defined.  It defines a set of link types to be traversed when an object such as an instance is clicked in the UI (therefore referred to as the focal point). See "Clique Scanning" below. This definition can differ between environments. Example: for focal point type "instance", the link types are often set to - instance-vnic - vnic-vconnector - vconnector-vedge - vedge-pnic - pnic-network | The clique\_constraints collection defines a constraint on links traversed for a specific clique when starting from a given focal point.  | For example: instance cliques are constrained to a specific network. If we wouldn't have this constraint, the resulting graph would stretch to include objects from neighboring networks that are not really related to the instance. \ ***Hierarchy of Scanning*** The initial scanning is done hierarchically, starting from the environment level and discovering lower levels in turn. Examples: - Under environment we scan for regions and projects (tenants). - Under availability zone we have hosts, and under hosts we have instances and host services The actual scanning order is not always same as the logical hierarchical order of objects, to improve scanning performance. Some objects are referenced multiple times in the hierarchy. For example, hosts are always in an availability zone, but can also be part of a host aggregate. Such extra references are saved as references to the main object. ***Clique Scanning*** | For creating cliques based on the discovered objects and links, clique types need to be defined for the given environment. | A clique type specifies the list of link types used in building a clique for a specific focal point object type. For example, it can define that for instance objects we want to have the following link types: - instance-vnic - vnic-vconnector - vconnector-vedge - vedge-pnic - pnic-network   As in many cases the same clique types are used, default clique types will be provided with a new Calipso deployment. \ ***Clique creation algorithm*** - For each clique type CT: - For each focal point object F of the type specified as the clique type focal point type: - Create a new clique C - Add F to the list of objects included in the clique - For each link type X-Y of the link types in CT: - Find all the source objects of type x that are already included in the clique - For each such source object S: - for all links L of type X-Y that have S as their source - Add the object T of type Y that is the target in L to the list of objects included in the clique - Add L to the list of links in the clique C How to run scans ---------------- For running environment scans Calipso uses a specialized daemon script called *scan manager*. If Calipso application is deployed in docker containers, scan manager will run inside the *calipso-scan* container. Scan manager uses MongoDB connection to fetch requests for environment scans and execute them by running a *scan* script. It also performs extra checks and procedures connected to scan failure/completion, such as marking *environment* as scanned and reporting errors (see `details <#scan-manager>`__). Scan script workflow: 1. Loads specific scanners definitions from a predefined metadata file (which can be extended in order to support scanning of new object types). 2. Runs the root scanner and then children scanners recursively (see `Hierarchy of scanning <#Hierarchy_of_scanning>`__) a. Scanners do all necessary work to insert objects in *inventory*. 3. Finalizes the scan and publishes successful scan completion. Scan manager ~~~~~~~~~~~~ Scan manager is a script which purpose is to manage the full lifecycle of scans requested through API. It runs indefinitely while: 1. Polling the database (*scans* and *scheduled\_scans* collections) for new and scheduled scan requests; 2. Parsing their configurations; 3. Running the scans; 4. Logging the results. Scan manager can be run in a separate container provided that it has connection to the database and the topology source system. Monitoring ---------- ***Monitoring Subsystem Overview*** Calipso monitoring uses Sensu to remotely track actual state of hosts. A Sensu server is installed as a Docker image along with the other Calipso components.   Remote hosts send check events to the Sensu server.  We use a filtering of events such that the first occurrence of a check is always used, after that cases where status is unchanged are ignored. When handling a check event, the Calipso Sensu handlers will find the matching Calipso object, and update its status. We also keep the timestamp of the last status update, along with the full check output. Setup of checks and handlers code on the server and the remote hosts can be done by Calipso. It is also possible to have this done using another tool, e.g. Ansible or Puppet. More info is available in Monitoring Guide document.   ***Package Structure*** Monitoring package is divided like this: 1.      Checks: these are the actual check scripts that will be run on the hosts; 2.      Handlers: the code that does handling of check events; 3.      Setup: code for setting up handlers and checks. Events Guide ============ Introduction ------------ Events ~~~~~~ Events in general sense are any changes to the monitored topology objects that are trackable by Calipso. We currently support subscription to Neutron notification queues for several OpenStack distributions as a source of events. The two core concepts of working with events are *listening to events* and *event handling*, so the main module groups in play are the *event listener* and *event handlers*. Event listeners ~~~~~~~~~~~~~~~ An event listener is a module that handles connection to the event source, listening to the new events and routing them to respective event handlers. An event listener class should be designed to run indefinitely in foreground or background (daemon) while maintaining a connection to the source of events (generally a message queue like RabbitMQ or Apache Kafka). Each incoming event is examined and, if it has the correct format, is routed to the corresponding event handler class. The routing can be accomplished through a dedicated event router class using a metadata file and a metadata parser (see `Metadata parsers <#metadata-parsers>`__). Event handlers ~~~~~~~~~~~~~~ An event handler is a specific class that parses the incoming event payload and performs a certain CUD (Create/Update/Delete) operation on zero or more database objects. Event handler should be independent of the event listener implementation. Event manager ~~~~~~~~~~~~~ Event manager is a script which purpose is to manage event listeners. It runs indefinitely and performs the following operations: 1. Starts a process for each valid entry in *environments\_config* collection that is scanned (*scanned == true*) and has the *listen* flag set to *true*; 2. Checks the *operational* statuses of event listeners and updating them in *environments\_config* collection; 3. Stops the event listeners that no longer qualify for listening (see step 1); 4. Restarts the event listeners that quit unexpectedly; 5. Repeats steps 1-5 Event manager can be run in a separate container provided that it has connection to the database and to all events source systems that event listeners use. Contribution ~~~~~~~~~~~~ You can contribute to Calipso *events* system in several ways: - create custom event handlers for an existing listener; - create custom event listeners and reuse existing handlers; - create custom event handlers and listeners. See `Creating new event handlers <#creating-new-event-handlers>`__ and `Creating new event listeners <#creating-new-event-listeners>`__ for the respective guides. Contribution ============ This section covers the designed approach to contribution to Calipso. The main scenario of contribution consists of introducing a new *object* type to the discovery engine, defining *links* that connect this new object to existing ones, and describing a *clique* (or cliques) that makes use of the object and its links. Below we describe how this scenario should be implemented, step-by-step. *Note*: Before writing any new code, you need to create your own environment using UI (see User Guide document) or API (see the API guide doc). Creating an entry directly in *“environments\_config”* collection is not recommended. Creating new object types ------------------------- Before you proceed with creation of new object type, you need to make sure the following requirements are met: - New object type has a unique name and purpose - New object type has an existing parent object type First of all, you need to create a fetcher that will take care of getting info on objects of the new type, processing it and adding new entries in Calipso database. Creating new fetchers ~~~~~~~~~~~~~~~~~~~~~ A fetcher is a common name for a class that handles fetching of all objects of a certain type that have a common parent object. The source of this data may be already implemented in Calipso (like OpenStack API, CLI and DB sources) or you may create one yourself. ***Common fetchers*** Fetchers package structure should adhere to the following pattern (where *%source\_name%* is a short prefix like *api, cli, db*): - app - discover - fetchers - *%source\_name%* - *%source\_name%*\ \_\ *%fetcher\_name%.*\ py If you reuse the existing data source, your new fetcher should subclass the class located in *%source\_name%\_access* module inside the *%source\_name%* directory. Fetcher class name should repeat the module name, except in CamelCase instead of snake\_case. Example: if you are adding a new cli fetcher, you should subclass *CliAccess* class found by *app/discover/fetchers/cli/cli\_access.py* path. If the module is named *cli\_fetch\_new\_objects.py*, fetcher class should be named *CliFetchNewObjects*. If you are creating a fetcher that uses new data source, you may consider adding an “access” class for this data source to store convenience methods. In this case, the “access” class should subclass the base Fetcher class (found in *app/discover/fetcher.py*) and the fetcher class should subclass the “access” class. All business logic of a fetcher should be defined inside the overridden method from base Fetcher class *get(self, parent\_id)*. You should use the second argument that is automatically passed by parent scanner to get the parent entity from database and any data you may need. This method has to return a list of new objects (dicts) that need to be inserted in Calipso database. Their parent object should be passed along other fields (see example). *Note*: types of returned objects should match the one their fetcher is designed for. ***Example***: **app/discover/fetchers/cli/cli\_fetch\_new\_objects.py** | **from** discover.fetchers.cli.cli\_access **import** CliAccess | **from** utils.inventory\_mgr **import** InventoryMgr | **class** CliFetchNewObjects(CliAccess): | **def** \_\_init\_\_(self): | super().\_\_init\_\_() | self.inv = InventoryMgr() | **def** get(self, parent\_id): | parent = self.inv.get\_by\_id(self.env, parent\_id) | *# do something *\ objects = [{**"type"**: **"new\_type"**, **"id"**: **"1234"**, **"parent"**: parent}, | {**"type"**: **"new\_type"**, **"id"**: **"2345"**, **"parent"**: parent}] | **return** objects This is an example of a fetcher that deals with the objects of type *“new\_type”*. It uses the parent id to fetch the parent object, then performs some operations in order to fetch the new objects and ultimately returns the objects list, at which point it has gathered all required information. \ ***Folder fetcher*** A special type of fetchers is the folder fetcher. It serves as a dummy object used to aggregate objects in a specific point in objects hierarchy. If you would like to logically separate children objects from parent, you may use folder fetcher found at *app/discover/fetchers/folder\_fetcher.py*. Usage is described `here <#Folder_scanner>`__. The scanners configuration file structure ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Scanners.json** (full path *app/config/scanners.json*) is an essential configuration file that defines scanners hierarchy. It has a forest structure, meaning that it is a set of trees, where each tree has a *root* scanner, potentially many levels of *children* scanners and pointers from parent scanners to children scanners. Scanning hierarchy is described `here <#Hierarchy_of_scanning>`__. A scanner is essentially a list of fetchers with configuration (we’ll call those **Fetch types**). Fetch types can be **Simple** and **Folder**, described below. ***Simple fetch type*** A simple fetch type looks like this: | { | **"type"**: **"project"**, | **"fetcher"**: **"ApiFetchProjects"**, | **"object\_id\_to\_use\_in\_child"**: **"name"**, | **"environment\_condition"**: { | **"mechanism\_drivers"**: **"VPP" ** }, | **"children\_scanner"**: **"ScanProject" **} Supported fields include: - *“fetcher”* – class name of fetcher that the scanner uses; - *“type”* – object type that the fetcher works with; - *“children\_scanner”* – (optional) full name of a scanner that should run after current one finishes; - *“environment\_condition”* – (optional) specific constraints that should be checked against the environment in *environments\_config* collection before execution; - *“object\_id\_to\_use\_in\_child*\ ” – (optional) which parent field should be passed as parent id to the fetcher (default: “id”). \ ***Folder fetch type*** Folder fetch types deal with folder fetchers (described `here <#Folder_fetcher>`__) and have a slightly different structure: | { | **"type"**: **"aggregates\_folder"**, | **"fetcher"**: { | **"folder"**: **true**, | **"types\_name"**: **"aggregates"**, | **"parent\_type"**: **"region" **}, **"object\_id\_to\_use\_in\_child"**: **"name"**, | **"environment\_condition"**: { | **"mechanism\_drivers"**: **"VPP" ** }, | **"children\_scanner"**: **"ScanAggregatesRoot" **} The only difference is that *“fetcher”* field is now a dictionary with the following fields: - *“folder”* – should always be **true**; - *“types\_name”* – type name in plural (with added ‘s’) of objects that serve as folder’s children - *“parent\_type”* – folder’s parent type (basically the parent type of folder’s objects). Updating scanners ~~~~~~~~~~~~~~~~~ After creating a new fetcher, you should integrate it into scanners hierarchy. There are several possible courses of action: ***Add new scanner as a child of an existing one*** If the parent type of your newly added object type already has a scanner, you can add your new scanner as a child of an existing one. There are two ways to do that: 1. Add new scanner as a *“children\_scanner”* field to parent scanner ***Example*** Before: | **"ScanHost"**: [ | { | **"type"**: **"host"**, | **"fetcher"**: **"ApiFetchProjectHosts"**, | } | ], After: | **"ScanHost"**: [ | { | **"type"**: **"host"**, | **"fetcher"**: **"ApiFetchProjectHosts"**, | **"children\_scanner"**: **"NewTypeScanner" **} | ], | **"NewTypeScanner"**: [ | { | **"type"**: **"new\_type"**, | **"fetcher"**: **"CliFetchNewType" **} | ] 1. Add new fetch type to parent scanner (in case if children scanner already exists) ***Example*** Before: | **"ScanHost"**: [ | { | **"type"**: **"host"**, | **"fetcher"**: **"ApiFetchProjectHosts"**, | **"children\_scanner"**: **"ScanHostPnic" **} | ], After: | **"ScanHost"**: [ | { | **"type"**: **"host"**, | **"fetcher"**: **"ApiFetchProjectHosts"**, | **"children\_scanner"**: **"ScanHostPnic" **}, | { | **"type"**: **"new\_type"**, | **"fetcher"**: **"CliFetchNewType" **} | ], ***Add new scanner and set an existing one as a child*** ***Example*** Before: | **"ScanHost"**: [ | { | **"type"**: **"host"**, | **"fetcher"**: **"ApiFetchProjectHosts"**, | **"children\_scanner"**: **"ScanHostPnic" **} | ], After: | **"NewTypeScanner"**: [ | { | **"type"**: **"new\_type"**, | **"fetcher"**: **"CliFetchNewType"**, | **"children\_scanner"**: **"ScanHost" **} | ] | **"ScanHost"**: [ | { | **"type"**: **"host"**, | **"fetcher"**: **"ApiFetchProjectHosts"**, | **"children\_scanner"**: **"ScanHostPnic" **} | ], ***Other cases*** You may choose to combine approaches or use none of them and create an isolated scanner if needed. Updating constants collection ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Before testing your new scanner and fetcher you need to add the newly created object type to *“constants”* collection in Calipso database: 1. **constants.object\_types** document Append a *{“value”: “new\_type”, “label”: “new\_type”}* object to **data** list. 1. **constants.scan\_object\_types** document Append a *{“value”: “new\_type”, “label”: “new\_type”}* object to **data** list. 1. **constants.object\_types\_for\_links** document If you’re planning to build links using this object type (you probably are), append a *{“value”: “new\_type”, “label”: “new\_type”}* object to **data** list. Setting up monitoring ~~~~~~~~~~~~~~~~~~~~~ In order to setup monitoring for the new object type you have defined, you’ll need to add a Sensu check: 1. Add a check script in app/monitoring/checks: a. | Checks should return the following values: | 0: **OK** | 1: **Warning** | 2: **Error** b. Checks can print the underlying query results to stdout. Do so within reason, as this output is later stored in the DB, so avoid giving too much output; c. Test your script on a remote host: i. Write it in */etc/sensu/plugins* directory; ii. Update the Sensu configuration on the remote host to run this check; iii. Add the check in the “checks” section of */etc/sensu/conf.d/client.json*; iv. The name under which you save the check will be used by the handler to determine the DB object that it relates to; v. Restart the client with the command: *sudo service sensu-client restart*; vi. Check the client log file to see the check is run and produces the expected output (in */var/log/sensu* directory). d. Add the script to the source directory (*app/monitoring/checks*). 2. Add a handler in app/monitoring/handlers: a. If you use a standard check naming scheme and check an object, the *BasicCheckHandler* can take care of this, but add the object type in *basic\_handling\_types* list in *get\_handler()*; b. If you have a more complex naming scheme, override MonitoringCheckHandler. See HandleOtep for example. 3. If you deploy monitoring using Calipso: a. Add the check in the *monitoring\_config\_templates* collection. *Check Naming* The check name should start with the type of the related object, followed by an underscore (“\_”). For example, the name for a check related to an OTEP (type “otep”)  will start with “otep\_“. It should then be followed by the object ID.   For checks related to links, the check name will have this format:
link\_\_\_ Creating new link types ----------------------- After you’ve added a new object type you may consider adding new link types to connect objects of new type to existing objects in topology. Your new object type may serve as a *source* and/or *target* type for the new link type. The process of new link type creation includes several steps: 1. Write a link finder class; 2. Add the link finder class to the link finders configuration file; 3. Update *“constants”* collection with the new link types. Writing link finder classes ~~~~~~~~~~~~~~~~~~~~~~~~~~~ A new link finder class should: 1. Subclass *app.discover.link\_finders.FindLinks* class; 2. Be located in the *app.discover.link\_finders* package; 3. Define an instance method called *add\_links(self)* with no additional arguments. This method is the only entry point for link finder classes. *FindLinks* class provides access to inventory manager to its subclasses which they should use to their advantage. It also provides a convenience method *create\_links(self, …)* for saving links to database. It is reasonable to call this method at the end of *add\_links* method. You may opt to add more than one link type at a time in a single link finder. ***Example*** | **from** discover.find\_links **import** FindLinks | **class** FindLinksForNewType(FindLinks): | **def** add\_links(self): | new\_objects = self.inv.find\_items({\ **"environment"**: self.get\_env(), | **"type"**: **"new\_type"**}) | **for** new\_object **in** new\_objects: | old\_object = self.inv.get\_by\_id(environment=self.get\_env(), | item\_id=new\_object[**"old\_object\_id"**]) | link\_type = **"old\_type-new\_type" **\ link\_name = **"{}-{}"**.format(old\_object[**"name"**], new\_object[**"name"**]) | state = **"up"** *# TBD *\ link\_weight = 0 *# TBD *\ self.create\_link(env=self.get\_env(), | source=old\_object[**"\_id"**], | source\_id=old\_object[**"id"**], | target=new\_object[**"\_id"**], | target\_id=new\_object[**"id"**], | link\_type=link\_type, | link\_name=link\_name, | state=state, | link\_weight=link\_weight) Updating the link finders configuration file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Default link finders configuration file can be found at */app/config/link\_finders.json* and has the following structure: | { | **"finders\_package"**: **"discover.link\_finders"**, | **"base\_finder"**: **"FindLinks"**, | **"link\_finders"**: [ | **"FindLinksForInstanceVnics"**, | **"FindLinksForOteps"**, | **"FindLinksForPnics"**, | **"FindLinksForVconnectors"**, | **"FindLinksForVedges"**, | **"FindLinksForVserviceVnics" **] | } File contents: - *finders\_package* – python path to the package that contains link finders (relative to $PYTHONPATH environment variable); - *base\_finder* – base link finder class name; - *link\_finders* – class names of actual link finders. If your new fetcher meets the requirements described in `Writing link finder classes <#writing-link-finder-classes>`__ section, you can append its name to the *“link\_finders”* list in *link\_finders.json* file. Updating constants collection ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Before testing your new links finder, you need to add the newly created link types to *“constants”* collection in Calipso database: 1. **constants.link\_types** document Append a *{“value”: “source\_type-target\_type”, “label”: “source\_type-target\_type”}* object to **data** list for each new link type. Creating custom link finders configuration file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you consider writing a custom list finders configuration file, you should also follow the guidelines from 4.2.1-4.2.3 while designing link finder classes and including them in the new link finders source file. The general approach is the following: 1. Custom configuration file should have the same key structure with the basic one; 2. You should create a *base\_finder* class that subclasses the basic FindLinks class (see `Writing link finder classes <#writing-link-finder-classes>`__); 3. Your link finder classes should be located in the same package with your *base\_finder* class; 4. Your link finder classes should subclass your *base\_finder* class and override the *add\_links(self)* method. Creating new clique types ------------------------- Two steps in creating new clique types and including them in clique finder are: 1. Designing new clique types 2. Updating clique types collection Designing new clique types ~~~~~~~~~~~~~~~~~~~~~~~~~~ A clique type is basically a list of links that will be traversed during clique scans (see `Clique creation algorithm <#clique_creation>`__). The process of coming up with clique types involves general networking concepts knowledge as well as expertise in monitored system details (e.g. OpenStack distribution specifics). In a nutshell, it is not a trivial process, so the clique design should be considered carefully. The predefined clique types (in *clique\_types* collection) may give you some idea about the rationale behind clique design. Updating clique types collection ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ After designing the new clique type, you need to update the *clique\_types* collection in order for the clique finder to use it. For this purpose, you should add a document of the following structure: {      "environment": "ANY",      "link\_types": [         "instance-vnic",          "vnic-vconnector",          "vconnector-vedge",          "vedge-otep",          "otep-vconnector",          "vconnector-host\_pnic",          "host\_pnic-network"     ],      "name": "instance",      "focal\_point\_type": "instance" } Document fields are: - *environment* – can either hold the environment name, for which the new clique type is designed, or **“ANY”** if the new clique type should be added to all environments; - *name* – display name for the new clique type; - *focal\_point\_type* – the aggregate object type for the new clique type to use as a starting point; - *link\_types* – a list of links that constitute the new clique type. Creating new event handlers --------------------------- There are three steps to creating a new event handler: 1. Determining *event types* that will be handled by the new handler; 2. Writing the new handler module and class; 3. Adding the (event type -> handler) mapping to the event handlers configuration file. Writing custom handler classes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each event handler should adhere to the following design: 1. Event handler class should subclass the app.discover.events.event\_base.EventBase class; 2. Event handler class should override handle method of EventBase. Business logic of the event handler should be placed inside the handle method; a. Handle method accepts two arguments: environment name (str) and notification contents (dict). No other event data will be provided to the method; b. Handle method returns an EventResult object, which accepts the following arguments in its constructor: i. *result* (mandatory) - determines whether the event handling was successful; ii. *retry* (optional) - determines whether the message should be put back in the queue in order to be processed later. This argument is checked only if result was set to False; iii. *message* (optional) - (Currently unused) a string comment on handling status; iv. *related\_object* (optional) – id of the object related to the handled event; v. *display\_context* (optional) – (Calipso UI requirement). 3. Module containing event handler class should have the same name as the relevant handler class except translated from UpperCamelCase to snake\_case. ***Example:*** **app/discover/events/event\_new\_object\_add.py** | **from** discover.events.event\_base **import** EventBase, EventResult | **class** EventNewObjectAdd(EventBase): | **def** handle(self, env: str, notification: dict) -> EventResult: | obj\_id = notification[**'payload'**][**'new\_object'**][**'id'**] | obj = { | **'id'**: obj\_id, | **'type'**: **'new\_object' **} | self.inv.set(obj) | **return** EventResult(result=\ **True**) Modifications in *events.json*: <...> | **"event\_handlers"**: { | <...> | **"new\_object.create"**: **"EventNewObjectAdd"**, | <...>** **} <...> After these changes are implemented, any event of type new\_object.create will be consumed by the event listener and the payload will be passed to EventNewObjectAdd handler which will insert a new document in the database. Event handlers configuration file structure ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Events.json** (full path *app/config/events.json*) is a configuration file that contains information about events and event handlers, including: - Event subscription details (queues and exchanges for Neutron notifications); - Location of event handlers package; - Mappings between event types and respective event handlers. The structure of *events.json* is as following: | { | **"handlers\_package"**: **"discover.events"**, | **"queues"**: [ | { | **"queue"**: **"notifications.nova"**, | **"exchange"**: **"nova" **}, | <…> | ], | **"event\_handlers"**: { | **"compute.instance.create.end"**: **"EventInstanceAdd"**, | **"compute.instance.update"**: **"EventInstanceUpdate"**, | **"compute.instance.delete.end"**: **"EventInstanceDelete"**, | **"network.create.end"**: **"EventNetworkAdd"**, | <…>** **} | } The root object contains the following fields: - **handlers\_package** - python path to the package that contains event handlers (relative to $PYTHONPATH environment variable) - **queues –** RabbitMQ queues and exchanges to consume messages from (for Neutron notifications case) - **event\_handlers** – mappings of event types to the respective handlers. The structure suggests that any event can have only one handler. In order to add a new event handler to the configuration file, you should add another mapping to the event\_handlers object, where key is the event type being handled and value is the handler class name (module name will be determined automatically). If your event is being published to a queue and/or exchange that the listener is not subscribed to, you should add another entry to the queues list. Creating new event listeners ---------------------------- At the moment, the only guideline for creation of new event listeners is that they should subclass the *ListenerBase* class (full path *app/discover/events/listeners/listener\_base.py*) and override the *listen(self)* method that listens to incoming events indefinitely (until terminated by a signal). In future versions, a comprehensive guide to listeners structure is planned. Metadata parsers ---------------- Metadata parsers are specialized classes that are designed to verify metadata files (found in *app/*\ config directory), use data from them to load instances of implementation classes (e.g. scanners, event handlers, link finders) in memory, and supply them by request. Scanners and link finders configuration files are used in scanner, event handlers configuration file – in event listener. In order to create a new metadata parser, you should consider subclassing *MetadataParser* class (found in *app/utils/metadata\_parser.py*). *MetadataParser* supports parsing and validating of json files out of the box. Entry point for the class is the *parse\_metadata\_file* method, which requires the abstract *get\_required\_fields* method to be overridden in subclasses. This method should return a list of keys that the metadata file is required to contain. For different levels of customization you may consider: 1. Overriding *validate\_metadata* method to provide more precise validation of metadata; 2. Overriding *parse\_metadata\_file* to provide custom metadata handling required by your use case. .. |image0| image:: media/image1.png :width: 6.50000in :height: 4.27153in