aboutsummaryrefslogtreecommitdiffstats
path: root/docs/release/developer-guide.rst
blob: 0de3f57199214b7b43f14b5369a76acb8525a7b4 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
| Calipso
| Developer Guide

|image0|

Project “Calipso” tries to illuminate complex virtual networking with
real time operational state visibility for large and highly distributed
Virtual Infrastructure Management (VIM).

We believe that Stability is driven by accurate Visibility.

Calipso provides visible insights using smart discovery and virtual
topological representation in graphs, with monitoring per object in the
graph inventory to reduce error vectors and troubleshooting, maintenance
cycles for VIM operators and administrators.

Project architecture
====================

Calipso comprises two major parts: application and UI. We’ll focus on
the former in this developer guide.

Current project structure is as follows:

-  root/

   -  app/

      -  api/

         -  responders/

            -  auth/

            -  resource/

         -  *server.py*

      -  config/

         -  *events.json*

         -  *scanners.json*

      -  discover/

         -  events/

            -  listeners/

               -  *default\_listener.py*

               -  *listener\_base.py*

            -  handlers/

               -  *event\_base.py*

               -  *event\_\*.py*

         -  fetchers/

            -  aci/

            -  api/

            -  cli/

            -  db/

         -  *event\_manager.py*

         -  *scan.py*

         -  *scan\_manager.py*

      -  monitoring/

         -  checks/

         -  handlers/

            -  *monitor.py*

         -  setup/

            -  *monitoring\_setup\_manager.py*

      -  test/

         -  api/

         -  event\_based\_scan/

         -  fetch/

         -  scan/

      -  utils/

   -  ui/

Application structure
---------------------

‘API’ package
~~~~~~~~~~~~~

Calipso API is designed to be used by native and third-party
applications that are planning to use Calipso discovery application.

***api/responders***

This package contains all exposed API endpoint handlers:

*auth* package contains token management handlers,

*resource* package contains resource handlers.

***server.py***

API server startup script. In order for it to work correctly, connection
arguments for a Mongo database used by a Calipso application instance
are required:

-m [MONGO\_CONFIG], --mongo\_config [MONGO\_CONFIG]

name of config file with mongo access details

--ldap\_config [LDAP\_CONFIG]

name of the config file with ldap server config

details

-l [LOGLEVEL], --loglevel [LOGLEVEL]

logging level (default: 'INFO')

-b [BIND], --bind [BIND]

binding address of the API server (default

127.0.0.1:8000)

-y [INVENTORY], --inventory [INVENTORY]

name of inventory collection (default: 'inventory')

-t [TOKEN\_LIFETIME], --token-lifetime [TOKEN\_LIFETIME]

lifetime of the token

For detailed reference and endpoints guide, see the API Guide document.

‘Discover’ package
~~~~~~~~~~~~~~~~~~

‘Discover’ package contains the core Calipso functionality which
involves:

-  scanning a network topology using a defined suite of scanners (see
   `Scanning concepts <#scanning-concepts>`__, `Scanners configuration
   file structure <#the-scanners-configuration-file-structure>`__) that
   use fetchers to get all needed data on objects of the topology;

-  tracking live events that modifies the topology in any way (by adding
   new object, updating existing or deleting them) using a suite of
   event handlers and event listeners;

-  managing the aforementioned suites using specialized manager scripts
   (*scan\_manager.py* and *event\_manager.py*)

‘Tests’ package
~~~~~~~~~~~~~~~

‘Tests’ package contains unit tests for main Calipso components: API,
event handlers, fetchers, scanners and utils.

Other packages
~~~~~~~~~~~~~~

***Install***

Installation and deployment scripts (with initial data for Calipso
database).

***Monitoring***

Monitoring configurations, checks and handlers (see
`Monitoring <#monitoring>`__ section and Monitoring Guide document).

***Utils***

Utility modules for app-wide use (inventory manager, mongo access,
loggers, etc.).

Scanning Guide
==============

 Introduction to scanning
-------------------------

Architecture overview
~~~~~~~~~~~~~~~~~~~~~

Calipso backend will scan any OpenStack environment to discover the
objects that it is made of, and place the objects it discovered in a
MongoDB database.

Following discovery of objects, Calipso will:

| Find what links exist between these objects, and save these links to
  MongoDB as well.
| For example, it will create a pnic-network link from a pNIC (physical
  NIC) and the network it is connected to.

Based on user definitions, it will create a 'clique' for each object
using the links it previously found. These cliques are later used to
present graphs for objects being viewed in the Calipso UI. This is not a
clique by graph theory definition, but more like the social definition
of clique: a graph of related, interconnected nodes.

 

OpenStack Scanning is done using the following methods, in order of
preference:

1. OpenStack API

2. MySQL DB - fetch any extra detail we can from the infrastructure
   MySQL DB used by OpenStack

3. CLI - connect by SSH to the hosts in the OpenStack environment to run
   commands, e.g. ifconfig, that will provide the most in-depth details.

 

| *Note*: 'environment' in Calipso means a single deployment of
  OpenStack, possibly containing multiple tenants (projects), hosts and
  instances (VMs). A single Calipso instance can handle multiple
  OpenStack environments. 
| However, we expect that typically Calipso will reside inside an
  OpenStack control node and will handle just that node's OpenStack
  environment.

 

***Environment***

| The Calipso scan script, written in Python, is called scan.py.
| It uses Python 3, along with the following libraries:

-  pymongo - for MongoDB access

-  mysql-connector - For MySQL DB access

-  paramiko - for SSH access

-  requests - For handling HTTP requests and responses to the OpenStack
   API

-  xmltodict - for handling XML output of CLI commands

-  cryptography - used by Paramiko

See Calipso installation guide for environment setup instructions.

***Configuration***

The configuration for accessing the OpenStack environment, by API, DB or
SSH, is saved in the Calipso MongoDB *“environments\_config”*
collection.

Calipso can work with a remote MongoDB instance, the details of which
are read from a configuration file (default: */etc/calipso/mongo.conf*).

| The first column is the configuration key while the second is the
  configuration value, in the case the value is the server host name or
  IP address.
| Other possible keys for MongoDB access:

-  port: IP port number

-  Other parameters for the PyMongo MongoClient class constructor

Alternate file location can be specified using the CLI -m parameter.

Scanning concepts
~~~~~~~~~~~~~~~~~

***DB Schema***

Objects are stored in the inventory collection, named *“inventory”* by
default, along with the accompanying collections, named by
default: \ *"links", "cliques", "clique\_types" and
"clique\_constraints"*. For development, separate sets of collections
can be defined per environment (collection names are created by
appending the default collection name to the alternative inventory
collection name).

The inventory, links and cliques collections are all designed to work
with a multi-environment scenario, so documents are marked with an
*"environment"* attribute.

The clique\_types collection allows Calipso users (typically
administrators) to define how the "clique" graphs are to be defined. 

It defines a set of link types to be traversed when an object such as an
instance is clicked in the UI (therefore referred to as the focal
point). See "Clique Scanning" below. This definition can differ between
environments.

Example: for focal point type "instance", the link types are often set
to

-  instance-vnic

-  vnic-vconnector

-  vconnector-vedge

-  vedge-pnic

-  pnic-network

| The clique\_constraints collection defines a constraint on links
  traversed for a specific clique when starting from a given focal
  point. 
| For example: instance cliques are constrained to a specific
  network. If we wouldn't have this constraint, the resulting graph
  would stretch to include objects from neighboring networks that are
  not really related to the instance.

\ ***Hierarchy of Scanning***

The initial scanning is done hierarchically, starting from the
environment level and discovering lower levels in turn.

Examples:

-  Under environment we scan for regions and projects (tenants).

-  Under availability zone we have hosts, and under hosts we have
   instances and host services

The actual scanning order is not always same as the logical hierarchical
order of objects, to improve scanning performance.

Some objects are referenced multiple times in the hierarchy. For
example, hosts are always in an availability zone, but can also be part
of a host aggregate. Such extra references are saved as references to
the main object.

***Clique Scanning***

| For creating cliques based on the discovered objects and links, clique
  types need to be defined for the given environment.
| A clique type specifies the list of link types used in building a
  clique for a specific focal point object type.

For example, it can define that for instance objects we want to have the
following link types:

-  instance-vnic

-  vnic-vconnector

-  vconnector-vedge

-  vedge-pnic

-  pnic-network

 

As in many cases the same clique types are used, default clique types
will be provided with a new Calipso deployment.

\ ***Clique creation algorithm***

-  For each clique type CT:

   -  For each focal point object F of the type specified as the clique
      type focal point type:

      -  Create a new clique C

      -  Add F to the list of objects included in the clique

      -  For each link type X-Y of the link types in CT:

         -  Find all the source objects of type x that are already
            included in the clique

         -  For each such source object S:

         -  for all links L of type X-Y that have S as their source

            -  Add the object T of type Y that is the target in L to the
               list of objects included in the clique

            -  Add L to the list of links in the clique C

How to run scans
----------------

For running environment scans Calipso uses a specialized daemon script
called *scan manager*. If Calipso application is deployed in docker
containers, scan manager will run inside the *calipso-scan* container.

Scan manager uses MongoDB connection to fetch requests for environment
scans and execute them by running a *scan* script. It also performs
extra checks and procedures connected to scan failure/completion, such
as marking *environment* as scanned and reporting errors (see
`details <#scan-manager>`__).

Scan script workflow:

1. Loads specific scanners definitions from a predefined metadata file
   (which can be extended in order to support scanning of new object
   types).

2. Runs the root scanner and then children scanners recursively (see
   `Hierarchy of scanning <#Hierarchy_of_scanning>`__)

   a. Scanners do all necessary work to insert objects in *inventory*.

3. Finalizes the scan and publishes successful scan completion.

Scan manager
~~~~~~~~~~~~

Scan manager is a script which purpose is to manage the full lifecycle
of scans requested through API. It runs indefinitely while:

1. Polling the database (*scans* and *scheduled\_scans* collections) for
   new and scheduled scan requests;

2. Parsing their configurations;

3. Running the scans;

4. Logging the results.

Scan manager can be run in a separate container provided that it has
connection to the database and the topology source system.

Monitoring
----------

***Monitoring Subsystem Overview***

Calipso monitoring uses Sensu to remotely track actual state of hosts.

A Sensu server is installed as a Docker image along with the other
Calipso components.

 

Remote hosts send check events to the Sensu server. 

We use a filtering of events such that the first occurrence of a check
is always used, after that cases where status is unchanged are ignored.

When handling a check event, the Calipso Sensu handlers will find the
matching Calipso object, and update its status.

We also keep the timestamp of the last status update, along with the
full check output.

Setup of checks and handlers code on the server and the remote hosts can
be done by Calipso. It is also possible to have this done using another
tool, e.g. Ansible or Puppet.

More info is available in Monitoring Guide document.

 

***Package Structure***

Monitoring package is divided like this:

1.      Checks: these are the actual check scripts that will be run on
the hosts;

2.      Handlers: the code that does handling of check events;

3.      Setup: code for setting up handlers and checks.

Events Guide
============

Introduction
------------

Events
~~~~~~

Events in general sense are any changes to the monitored topology
objects that are trackable by Calipso. We currently support subscription
to Neutron notification queues for several OpenStack distributions as a
source of events.

The two core concepts of working with events are *listening to events*
and *event handling*, so the main module groups in play are the *event
listener* and *event handlers*.

Event listeners
~~~~~~~~~~~~~~~

An event listener is a module that handles connection to the event
source, listening to the new events and routing them to respective event
handlers.

An event listener class should be designed to run indefinitely in
foreground or background (daemon) while maintaining a connection to the
source of events (generally a message queue like RabbitMQ or Apache
Kafka). Each incoming event is examined and, if it has the correct
format, is routed to the corresponding event handler class. The routing
can be accomplished through a dedicated event router class using a
metadata file and a metadata parser (see `Metadata
parsers <#metadata-parsers>`__).

Event handlers
~~~~~~~~~~~~~~

An event handler is a specific class that parses the incoming event
payload and performs a certain CUD (Create/Update/Delete) operation on
zero or more database objects. Event handler should be independent of
the event listener implementation.

Event manager
~~~~~~~~~~~~~

Event manager is a script which purpose is to manage event listeners. It
runs indefinitely and performs the following operations:

1. Starts a process for each valid entry in *environments\_config*
   collection that is scanned (*scanned == true*) and has the *listen*
   flag set to *true*;

2. Checks the *operational* statuses of event listeners and updating
   them in *environments\_config* collection;

3. Stops the event listeners that no longer qualify for listening (see
   step 1);

4. Restarts the event listeners that quit unexpectedly;

5. Repeats steps 1-5

Event manager can be run in a separate container provided that it has
connection to the database and to all events source systems that event
listeners use.

Contribution
~~~~~~~~~~~~

You can contribute to Calipso *events* system in several ways:

-  create custom event handlers for an existing listener;

-  create custom event listeners and reuse existing handlers;

-  create custom event handlers and listeners.

See `Creating new event handlers <#creating-new-event-handlers>`__ and
`Creating new event listeners <#creating-new-event-listeners>`__ for the
respective guides.

Contribution
============

This section covers the designed approach to contribution to Calipso.

The main scenario of contribution consists of introducing a new *object*
type to the discovery engine, defining *links* that connect this new
object to existing ones, and describing a *clique* (or cliques) that
makes use of the object and its links. Below we describe how this
scenario should be implemented, step-by-step.

*Note*: Before writing any new code, you need to create your own
environment using UI (see User Guide document) or API (see the API guide
doc). Creating an entry directly in *“environments\_config”* collection
is not recommended.

Creating new object types
-------------------------

Before you proceed with creation of new object type, you need to make
sure the following requirements are met:

-  New object type has a unique name and purpose

-  New object type has an existing parent object type

First of all, you need to create a fetcher that will take care of
getting info on objects of the new type, processing it and adding new
entries in Calipso database.

Creating new fetchers
~~~~~~~~~~~~~~~~~~~~~

A fetcher is a common name for a class that handles fetching of all
objects of a certain type that have a common parent object. The source
of this data may be already implemented in Calipso (like OpenStack API,
CLI and DB sources) or you may create one yourself.

***Common fetchers***

Fetchers package structure should adhere to the following pattern (where
*%source\_name%* is a short prefix like *api, cli, db*):

-  app

   -  discover

      -  fetchers

         -  *%source\_name%*

            -  *%source\_name%*\ \_\ *%fetcher\_name%.*\ py

If you reuse the existing data source, your new fetcher should subclass
the class located in *%source\_name%\_access* module inside the
*%source\_name%* directory.

Fetcher class name should repeat the module name, except in CamelCase
instead of snake\_case.

Example: if you are adding a new cli fetcher, you should subclass
*CliAccess* class found by *app/discover/fetchers/cli/cli\_access.py*
path. If the module is named *cli\_fetch\_new\_objects.py*, fetcher
class should be named *CliFetchNewObjects*.

If you are creating a fetcher that uses new data source, you may
consider adding an “access” class for this data source to store
convenience methods. In this case, the “access” class should subclass
the base Fetcher class (found in *app/discover/fetcher.py*) and the
fetcher class should subclass the “access” class.

All business logic of a fetcher should be defined inside the overridden
method from base Fetcher class *get(self, parent\_id)*. You should use
the second argument that is automatically passed by parent scanner to
get the parent entity from database and any data you may need. This
method has to return a list of new objects (dicts) that need to be
inserted in Calipso database. Their parent object should be passed along
other fields (see example).

*Note*: types of returned objects should match the one their fetcher is
designed for.

***Example***:

**app/discover/fetchers/cli/cli\_fetch\_new\_objects.py**

    | **from** discover.fetchers.cli.cli\_access **import** CliAccess
    | **from** utils.inventory\_mgr **import** InventoryMgr
    | **class** CliFetchNewObjects(CliAccess):
    | **def** \_\_init\_\_(self):
    | super().\_\_init\_\_()
    | self.inv = InventoryMgr()
    | **def** get(self, parent\_id):
    | parent = self.inv.get\_by\_id(self.env, parent\_id)
    | *# do something
      *\ objects = [{**"type"**: **"new\_type"**, **"id"**: **"1234"**,
      **"parent"**: parent},
    | {**"type"**: **"new\_type"**, **"id"**: **"2345"**, **"parent"**:
      parent}]
    | **return** objects

This is an example of a fetcher that deals with the objects of type
*“new\_type”*. It uses the parent id to fetch the parent object, then
performs some operations in order to fetch the new objects and
ultimately returns the objects list, at which point it has gathered all
required information.

\ ***Folder fetcher***

A special type of fetchers is the folder fetcher. It serves as a dummy
object used to aggregate objects in a specific point in objects
hierarchy. If you would like to logically separate children objects from
parent, you may use folder fetcher found at
*app/discover/fetchers/folder\_fetcher.py*.

Usage is described `here <#Folder_scanner>`__.

The scanners configuration file structure
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Scanners.json** (full path *app/config/scanners.json*) is an essential
configuration file that defines scanners hierarchy. It has a forest
structure, meaning that it is a set of trees, where each tree has a
*root* scanner, potentially many levels of *children* scanners and
pointers from parent scanners to children scanners. Scanning hierarchy
is described `here <#Hierarchy_of_scanning>`__.

A scanner is essentially a list of fetchers with configuration (we’ll
call those **Fetch types**). Fetch types can be **Simple** and
**Folder**, described below.

***Simple fetch type***

A simple fetch type looks like this:

    | {
    | **"type"**: **"project"**,
    | **"fetcher"**: **"ApiFetchProjects"**,
    | **"object\_id\_to\_use\_in\_child"**: **"name"**,

    | **"environment\_condition"**: {
    | **"mechanism\_drivers"**: **"VPP"
      ** },
    | **"children\_scanner"**: **"ScanProject"
      **}

Supported fields include:

-  *“fetcher”* – class name of fetcher that the scanner uses;

-  *“type”* – object type that the fetcher works with;

-  *“children\_scanner”* – (optional) full name of a scanner that should
   run after current one finishes;

-  *“environment\_condition”* – (optional) specific constraints that
   should be checked against the environment in *environments\_config*
   collection before execution;

-  *“object\_id\_to\_use\_in\_child*\ ” – (optional) which parent field
   should be passed as parent id to the fetcher (default: “id”).

    \ ***Folder fetch type***

Folder fetch types deal with folder fetchers (described
`here <#Folder_fetcher>`__) and have a slightly different structure:

    | {
    | **"type"**: **"aggregates\_folder"**,
    | **"fetcher"**: {
    | **"folder"**: **true**,
    | **"types\_name"**: **"aggregates"**,
    | **"parent\_type"**: **"region"
      **},

    **"object\_id\_to\_use\_in\_child"**: **"name"**,

    | **"environment\_condition"**: {
    | **"mechanism\_drivers"**: **"VPP"
      ** },
    | **"children\_scanner"**: **"ScanAggregatesRoot"
      **}

The only difference is that *“fetcher”* field is now a dictionary with
the following fields:

-  *“folder”* – should always be **true**;

-  *“types\_name”* – type name in plural (with added ‘s’) of objects
   that serve as folder’s children

-  *“parent\_type”* – folder’s parent type (basically the parent type of
   folder’s objects).

Updating scanners
~~~~~~~~~~~~~~~~~

After creating a new fetcher, you should integrate it into scanners
hierarchy. There are several possible courses of action:

***Add new scanner as a child of an existing one***

If the parent type of your newly added object type already has a
scanner, you can add your new scanner as a child of an existing one.
There are two ways to do that:

1. Add new scanner as a *“children\_scanner”* field to parent scanner

    ***Example***

    Before:

    | **"ScanHost"**: [
    | {
    | **"type"**: **"host"**,
    | **"fetcher"**: **"ApiFetchProjectHosts"**,
    | }
    | ],

    After:

    | **"ScanHost"**: [
    | {
    | **"type"**: **"host"**,
    | **"fetcher"**: **"ApiFetchProjectHosts"**,
    | **"children\_scanner"**: **"NewTypeScanner"
      **}
    | ],
    | **"NewTypeScanner"**: [
    | {
    | **"type"**: **"new\_type"**,
    | **"fetcher"**: **"CliFetchNewType"
      **}
    | ]

1. Add new fetch type to parent scanner (in case if children scanner
   already exists)

    ***Example***

    Before:

    | **"ScanHost"**: [
    | {
    | **"type"**: **"host"**,
    | **"fetcher"**: **"ApiFetchProjectHosts"**,
    | **"children\_scanner"**: **"ScanHostPnic"
      **}
    | ],

    After:

    | **"ScanHost"**: [
    | {
    | **"type"**: **"host"**,
    | **"fetcher"**: **"ApiFetchProjectHosts"**,
    | **"children\_scanner"**: **"ScanHostPnic"
      **},
    | {
    | **"type"**: **"new\_type"**,
    | **"fetcher"**: **"CliFetchNewType"
      **}
    | ],

***Add new scanner and set an existing one as a child***

***Example***

    Before:

    | **"ScanHost"**: [
    | {
    | **"type"**: **"host"**,
    | **"fetcher"**: **"ApiFetchProjectHosts"**,
    | **"children\_scanner"**: **"ScanHostPnic"
      **}
    | ],

    After:

    | **"NewTypeScanner"**: [
    | {
    | **"type"**: **"new\_type"**,
    | **"fetcher"**: **"CliFetchNewType"**,
    | **"children\_scanner"**: **"ScanHost"
      **}
    | ]

    | **"ScanHost"**: [
    | {
    | **"type"**: **"host"**,
    | **"fetcher"**: **"ApiFetchProjectHosts"**,
    | **"children\_scanner"**: **"ScanHostPnic"
      **}
    | ],

***Other cases***

You may choose to combine approaches or use none of them and create an
isolated scanner if needed.

Updating constants collection
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Before testing your new scanner and fetcher you need to add the newly
created object type to *“constants”* collection in Calipso database:

1. **constants.object\_types** document

    Append a *{“value”: “new\_type”, “label”: “new\_type”}* object to
    **data** list.

1. **constants.scan\_object\_types** document

    Append a *{“value”: “new\_type”, “label”: “new\_type”}* object to
    **data** list.

1. **constants.object\_types\_for\_links** document

    If you’re planning to build links using this object type (you
    probably are), append a *{“value”: “new\_type”, “label”:
    “new\_type”}* object to **data** list.

Setting up monitoring
~~~~~~~~~~~~~~~~~~~~~

In order to setup monitoring for the new object type you have defined,
you’ll need to add a Sensu check:

1. Add a check script in app/monitoring/checks:

   a. | Checks should return the following values:
      | 0: **OK**
      | 1: **Warning**
      | 2: **Error**

   b. Checks can print the underlying query results to stdout. Do so
      within reason, as this output is later stored in the DB, so avoid
      giving too much output;

   c. Test your script on a remote host:

      i.   Write it in */etc/sensu/plugins* directory;

      ii.  Update the Sensu configuration on the remote host to run this
           check;

      iii. Add the check in the “checks” section of
           */etc/sensu/conf.d/client.json*;

      iv.  The name under which you save the check will be used by the
           handler to determine the DB object that it relates to;

      v.   Restart the client with the command: *sudo service
           sensu-client restart*;

      vi.  Check the client log file to see the check is run and
           produces the expected output (in */var/log/sensu* directory).

   d. Add the script to the source directory (*app/monitoring/checks*).

2. Add a handler in app/monitoring/handlers:

   a. If you use a standard check naming scheme and check an object, the
      *BasicCheckHandler* can take care of this, but add the object type
      in *basic\_handling\_types* list in *get\_handler()*;

   b. If you have a more complex naming scheme, override
      MonitoringCheckHandler. See HandleOtep for example.

3. If you deploy monitoring using Calipso:

   a. Add the check in the *monitoring\_config\_templates* collection.

*Check Naming*

The check name should start with the type of the related object,
followed by an underscore (“\_”). For example, the name for a check
related to an OTEP (type “otep”)  will start with “otep\_“. It should
then be followed by the object ID.

 

For checks related to links, the check name will have this
format:
link\_<link type>\_<from\_id>\_<to\_id>

Creating new link types
-----------------------

After you’ve added a new object type you may consider adding new link
types to connect objects of new type to existing objects in topology.
Your new object type may serve as a *source* and/or *target* type for
the new link type.

The process of new link type creation includes several steps:

1. Write a link finder class;

2. Add the link finder class to the link finders configuration file;

3. Update *“constants”* collection with the new link types.

Writing link finder classes
~~~~~~~~~~~~~~~~~~~~~~~~~~~

A new link finder class should:

1. Subclass *app.discover.link\_finders.FindLinks* class;

2. Be located in the *app.discover.link\_finders* package;

3. Define an instance method called *add\_links(self)* with no
   additional arguments. This method is the only entry point for link
   finder classes.

*FindLinks* class provides access to inventory manager to its subclasses
which they should use to their advantage. It also provides a convenience
method *create\_links(self, …)* for saving links to database. It is
reasonable to call this method at the end of *add\_links* method.

You may opt to add more than one link type at a time in a single link
finder.

***Example***

    | **from** discover.find\_links **import** FindLinks
    | **class** FindLinksForNewType(FindLinks):
    | **def** add\_links(self):
    | new\_objects = self.inv.find\_items({\ **"environment"**:
      self.get\_env(),
    | **"type"**: **"new\_type"**})
    | **for** new\_object **in** new\_objects:
    | old\_object = self.inv.get\_by\_id(environment=self.get\_env(),
    | item\_id=new\_object[**"old\_object\_id"**])
    | link\_type = **"old\_type-new\_type"
      **\ link\_name = **"{}-{}"**.format(old\_object[**"name"**],
      new\_object[**"name"**])
    | state = **"up"** *# TBD
      *\ link\_weight = 0 *# TBD
      *\ self.create\_link(env=self.get\_env(),
    | source=old\_object[**"\_id"**],
    | source\_id=old\_object[**"id"**],
    | target=new\_object[**"\_id"**],
    | target\_id=new\_object[**"id"**],
    | link\_type=link\_type,
    | link\_name=link\_name,
    | state=state,
    | link\_weight=link\_weight)

Updating the link finders configuration file
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Default link finders configuration file can be found at
*/app/config/link\_finders.json* and has the following structure:

    | {
    | **"finders\_package"**: **"discover.link\_finders"**,
    | **"base\_finder"**: **"FindLinks"**,
    | **"link\_finders"**: [
    | **"FindLinksForInstanceVnics"**,
    | **"FindLinksForOteps"**,
    | **"FindLinksForPnics"**,
    | **"FindLinksForVconnectors"**,
    | **"FindLinksForVedges"**,
    | **"FindLinksForVserviceVnics"
      **]
    | }

File contents:

-  *finders\_package* – python path to the package that contains link
   finders (relative to $PYTHONPATH environment variable);

-  *base\_finder* – base link finder class name;

-  *link\_finders* – class names of actual link finders.

If your new fetcher meets the requirements described in `Writing link
finder classes <#writing-link-finder-classes>`__ section, you can append
its name to the *“link\_finders”* list in *link\_finders.json* file.

Updating constants collection
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Before testing your new links finder, you need to add the newly created
link types to *“constants”* collection in Calipso database:

1. **constants.link\_types** document

    Append a *{“value”: “source\_type-target\_type”, “label”:
    “source\_type-target\_type”}* object to **data** list for each new
    link type.

Creating custom link finders configuration file
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you consider writing a custom list finders configuration file, you
should also follow the guidelines from 4.2.1-4.2.3 while designing link
finder classes and including them in the new link finders source file.

The general approach is the following:

1. Custom configuration file should have the same key structure with the
   basic one;

2. You should create a *base\_finder* class that subclasses the basic
   FindLinks class (see `Writing link finder
   classes <#writing-link-finder-classes>`__);

3. Your link finder classes should be located in the same package with
   your *base\_finder* class;

4. Your link finder classes should subclass your *base\_finder* class
   and override the *add\_links(self)* method.

Creating new clique types
-------------------------

Two steps in creating new clique types and including them in clique
finder are:

1. Designing new clique types

2. Updating clique types collection

Designing new clique types
~~~~~~~~~~~~~~~~~~~~~~~~~~

A clique type is basically a list of links that will be traversed during
clique scans (see `Clique creation algorithm <#clique_creation>`__). The
process of coming up with clique types involves general networking
concepts knowledge as well as expertise in monitored system details
(e.g. OpenStack distribution specifics). In a nutshell, it is not a
trivial process, so the clique design should be considered carefully.

The predefined clique types (in *clique\_types* collection) may give you
some idea about the rationale behind clique design.

Updating clique types collection
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

After designing the new clique type, you need to update the
*clique\_types* collection in order for the clique finder to use it. For
this purpose, you should add a document of the following structure:

    { 

        "environment": "ANY", 

        "link\_types": [

            "instance-vnic", 

            "vnic-vconnector", 

            "vconnector-vedge", 

            "vedge-otep", 

            "otep-vconnector", 

            "vconnector-host\_pnic", 

            "host\_pnic-network"

        ], 

        "name": "instance", 

        "focal\_point\_type": "instance"

    }

Document fields are:

-  *environment* – can either hold the environment name, for which the
   new clique type is designed, or **“ANY”** if the new clique type
   should be added to all environments;

-  *name* – display name for the new clique type;

-  *focal\_point\_type* – the aggregate object type for the new clique
   type to use as a starting point;

-  *link\_types* – a list of links that constitute the new clique type.

Creating new event handlers
---------------------------

There are three steps to creating a new event handler:

1. Determining *event types* that will be handled by the new handler;

2. Writing the new handler module and class;

3. Adding the (event type -> handler) mapping to the event handlers
   configuration file.

Writing custom handler classes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Each event handler should adhere to the following design:

1. Event handler class should subclass
   the app.discover.events.event\_base.EventBase class;

2. Event handler class should override handle method of EventBase.
   Business logic of the event handler should be placed inside
   the handle method;

   a. Handle method accepts two arguments: environment name (str) and
      notification contents (dict). No other event data will be provided
      to the method;

   b. Handle method returns an EventResult object, which accepts the
      following arguments in its constructor:

      i.   *result* (mandatory) - determines whether the event handling
           was successful;

      ii.  *retry* (optional) - determines whether the message should be
           put back in the queue in order to be processed later. This
           argument is checked only if result was set to False;

      iii. *message* (optional) - (Currently unused) a string comment on
           handling status;

      iv.  *related\_object* (optional) – id of the object related to
           the handled event;

      v.   *display\_context* (optional) – (Calipso UI requirement).

3. Module containing event handler class should have the same name as
   the relevant handler class except translated
   from UpperCamelCase to snake\_case.

    ***Example:***

    **app/discover/events/event\_new\_object\_add.py**

    | **from** discover.events.event\_base **import** EventBase,
      EventResult
    | **class** EventNewObjectAdd(EventBase):
    | **def** handle(self, env: str, notification: dict) -> EventResult:
    | obj\_id = notification[**'payload'**][**'new\_object'**][**'id'**]
    | obj = {
    | **'id'**: obj\_id,
    | **'type'**: **'new\_object'
      **}
    | self.inv.set(obj)
    | **return** EventResult(result=\ **True**)

    Modifications in *events.json*:

    <...>

    | **"event\_handlers"**: {
    | <...>
    | **"new\_object.create"**: **"EventNewObjectAdd"**,
    | <...>**
      **}

    <...>

After these changes are implemented, any event of type
new\_object.create will be consumed by the event listener and the
payload will be passed to EventNewObjectAdd handler which will insert a
new document in the database.

Event handlers configuration file structure
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Events.json** (full path *app/config/events.json*) is a configuration
file that contains information about events and event handlers,
including:

-  Event subscription details (queues and exchanges for Neutron
   notifications);

-  Location of event handlers package;

-  Mappings between event types and respective event handlers.

The structure of *events.json* is as following:

    | {
    | **"handlers\_package"**: **"discover.events"**,
    | **"queues"**: [
    | {
    | **"queue"**: **"notifications.nova"**,
    | **"exchange"**: **"nova"
      **},
    | <…>
    | ],
    | **"event\_handlers"**: {
    | **"compute.instance.create.end"**: **"EventInstanceAdd"**,
    | **"compute.instance.update"**: **"EventInstanceUpdate"**,
    | **"compute.instance.delete.end"**: **"EventInstanceDelete"**,
    | **"network.create.end"**: **"EventNetworkAdd"**,

    | <…>**
      **}
    | }

The root object contains the following fields:

-  **handlers\_package** - python path to the package that contains
   event handlers (relative to $PYTHONPATH environment variable)

-  **queues –** RabbitMQ queues and exchanges to consume messages from
   (for Neutron notifications case)

-  **event\_handlers** – mappings of event types to the respective
   handlers. The structure suggests that any event can have only one
   handler.

In order to add a new event handler to the configuration file, you
should add another mapping to the event\_handlers object, where key is
the event type being handled and value is the handler class name (module
name will be determined automatically).

If your event is being published to a queue and/or exchange that the
listener is not subscribed to, you should add another entry to the
queues list.

Creating new event listeners
----------------------------

At the moment, the only guideline for creation of new event listeners is
that they should subclass the *ListenerBase* class (full path
*app/discover/events/listeners/listener\_base.py*) and override the
*listen(self)* method that listens to incoming events indefinitely
(until terminated by a signal).

In future versions, a comprehensive guide to listeners structure is
planned.

Metadata parsers
----------------

Metadata parsers are specialized classes that are designed to verify
metadata files (found in *app/*\ config directory), use data from them
to load instances of implementation classes (e.g. scanners, event
handlers, link finders) in memory, and supply them by request. Scanners
and link finders configuration files are used in scanner, event handlers
configuration file – in event listener.

In order to create a new metadata parser, you should consider
subclassing *MetadataParser* class (found in
*app/utils/metadata\_parser.py*). *MetadataParser* supports parsing and
validating of json files out of the box. Entry point for the class is
the *parse\_metadata\_file* method, which requires the abstract
*get\_required\_fields* method to be overridden in subclasses. This
method should return a list of keys that the metadata file is required
to contain.

For different levels of customization you may consider:

1. Overriding *validate\_metadata* method to provide more precise
   validation of metadata;

2. Overriding *parse\_metadata\_file* to provide custom metadata
   handling required by your use case.

.. |image0| image:: media/image1.png
   :width: 6.50000in
   :height: 4.27153in