aboutsummaryrefslogtreecommitdiffstats
path: root/docs/release/monitoring-guide.rst
blob: 253284801e233eff9610f993628ee33d228218f6 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
###############################################################################
# Copyright (c) 2017 Koren Lev (Cisco Systems), Yaron Yogev (Cisco Systems)   #
# and others                                                                  #
#                                                                             #
# All rights reserved. This program and the accompanying materials            #
# are made available under the terms of the Apache License, Version 2.0       #
# which accompanies this distribution, and is available at                    #
# http://www.apache.org/licenses/LICENSE-2.0                                  #
###############################################################################
| Calipso.io
| Monitoring Guide

|image0|

Project “Calipso” tries to illuminate complex virtual networking with
real time operational state visibility for large and highly distributed
Virtual Infrastructure Management (VIM).

We believe that Stability is driven by accurate Visibility.

Calipso provides visible insights using smart discovery and virtual
topological representation in graphs, with monitoring per object in the
graph inventory to reduce error vectors and troubleshooting, maintenance
cycles for VIM operators and administrators.

Table of Contents

Calipso.io Monitoring Guide 1

1 Monitoring deployment options 3

1.1 Calipso monitoring provisioning 3

1.2 Calipso-sensu container 5

2 Monitoring configurations 5

2.1 Calipso monitoring configuration templates 6

2.2 Calipso Apex monitoring integration 11

Monitoring deployment options 
==============================

    Calipso uses sensu framework that includes a customized calipso
    sensu server and a set of customized clients, for details on the
    framework see:

    https://sensuapp.org/docs/latest/overview/what-is-sensu.html

    Calipso uses this framework but further configure and customizes the
    install, per:

    https://sensuapp.org/docs/1.0/installation/

    Server and clients are deployed with rabbitmq over SSL as the
    client-server transport:

    https://sensuapp.org/docs/1.0/reference/ssl.html

    Calipso then deploys customized checks and handlers as new types of
    plugins:

    http://sensu-plugins.io/plugins/

    In order to monitor all virtual components as described in
    Calipso-model document, Calipso offers two major functions:

1. Calipso scan module: customizes and automatically deploys sensu,
   config files and all needed plugins and scripts on all clients and
   the central server.

2. Calipso sensu module: listens for the results of the customized
   events and updates the inventories with state and statuses, while
   generating all related messages.

Calipso monitoring provisioning
-------------------------------

    Calipso administrator should instruct the main scanning engine
    whether or not to deploy monitoring across the VIM environment
    hosts. This is done first by the Calipso scan container that uses a
    set of “environment\_config” parameters (see calipso-model guide for
    details).

    A full scan of an environment is a mandatory pre-requisite for
    monitoring, because calipso monitoring provisioning uses inventory
    discovered data as values of several configuration items to be
    deployed onto the sensu framework.

    Here are the important parts of an “environment\_config” document
    that needs to be defined either through UI or API as a pre-requisite
    for monitoring provisioning:

    {

    **"enable\_monitoring": true**, // this must be “true” for any
    monitoring result

    **"name"** **: "my-env-with-monitoring",** // name is needed as
    usual

    **"distribution" :** // other details removed ….

    **"configuration" : [** // in configuration: a monitoring section is
    defined

    **{**

    **"name" : "Monitoring",** // name of this configuration section

    **"type" : "Sensu",** // instruct to use a sensu type of monitoring
    framework

    **"rabbitmq\_port" : 5671,** // the SSL port used by sensu clients
    to server

    **"rabbitmq\_pass" : "dummy\_pwd",** // clients-to-server rabbitmq
    password

    **"rabbitmq\_user" : "sensu",** // clients-to-server rabbitmq
    username

    **"server\_ip" : "10.0.0.1",** // the calipso sensu container ip
    address

    **"ssh\_user": "root",** // user used inside the sensu server
    (container)

    **"ssh\_password": "dummy\_pwd",** // root pwd used inside the sensu
    server

    **"ssh\_port" : 20022, //** ssh port where sensu container offers
    ssh access

    **"server\_name”: "sensu\_server", /**/ name to use for the sensu
    server api

    **"env\_type" : "production",** // a type of environment to setup
    (sensu’s ‘type’)

    **"config\_folder" : "/local\_dir/sensu\_config",** // location to
    place config files

    **"api\_port" : 4567** // sensu server api port to connect to

    **"provision" : "None" // \*\* see provision options described next
    ! \*\***

    **}, **

    **]**

    \*\* **Provision** options are:

1. **“None”:** does not deploy any monitoring configurations

2. **“DB”:** creates all necessary configuration files in the mongoDB,
   placed in “monitoring\_config” collection.

3. **“Files”:** creates all necessary configuration files in the
   mongoDB, placed in “monitoring\_config” collection and also in
   ‘config\_folder’ location on the server.

4. **“Deploy”:** Does what “Files” is doing, but also goes and deploys
   all configuration files and scripts on all servers on the
   environment.

    *This is a high level representation of the calipso monitoring
    provisioning logic:*

    |image1|

Calipso-sensu container
-----------------------

    Once sensu clients and all needed configurations and plugins are
    deployed properly, the sensu server should start receiving results
    and update the relevant inventory objects with their states and
    statuses.

    The calipso-sensu container is a pre-built sensu server customized
    for calipso design.

    The following services are exposed on the calipso-sensu container:

1. Calipso monitoring handling app running in:
   /home/scan/calipso\_prod/app/monitoring/handlers/monitor.py

2. Sensu-api service accessible on port 4567.

3. Calipso container bash access through ssh on port 20022 (see
   quickstart-guide).

4. Sensu-server service listening as rabbitmq over ssl on port 5671.

5. Rabbitmq management server on port 15672 (http://server-ip:15672/ to
   access).

6. Sensu uchiwa UI listening on port 3000 (http://server-ip:3000 to
   access).

    All the above services are maintained by calipso and customized for
    calipso’s virtual inventory monitoring design.

Monitoring configurations 
==========================

    You can access calipso-sensus container by ssh, using:

    **ssh scan@localhost -p 20022** with a default password “scan”.

    Inside the calipso-sensu container the calipso monitoring
    application is maintained at:

    **/home/scan/calipso\_prod/app/monitoring**

    SSL keys are shipped with the calipso-sensu container and maintained
    at:

    **/etc/sensu/ssl **

    Check scripts used to run a health check against monitored objects
    are maintained at:

    **/home/scan/calipso\_prod/app/monitoring/checks**

    Handling scripts to grab and handle checks results from clients are
    maintained at:

    **/home/scan/calipso\_prod/app/monitoring/handlers**

    Configuration deployment setup scripts are maintained at:

    **/home/scan/calipso\_prod/app/monitoring/setup**

Calipso monitoring configuration templates
------------------------------------------

    Calipso application maintained a customizable model for all the
    configuration files deployed onto the sensu server and clients at
    provisioning stage of its main scanning engine.

    The following configuration files are customizable:

    *Server side:*

    **redis.json**

    **transport,json**

    **rabbitmq.json**

    **api.json**

    **client.json**

    **filters.json**

    **handlers.json**

    *Clients side:*

    **transport,json**

    **rabbitmq.json**

    **client.json**

    **customized checks to integrate into client.json **

    We have defined a default set of configuration parameters in those
    files per our knowledge and best practices, so a typical user would
    not need to edit any of the above set of configuration files and
    their options. In case any of these needs customizations we offer a
    very granular model maintained in Calipso mongoDB under the
    **“monitoring\_config\_templates”** collection.

    Here is an example of such template, and its options:

    **"type" : "client.json",** // this will be used for building
    client.json files

    **"order" : "1",** // if several of these templates are configured,
    take a priority to use

    **"side" : "client", //**\ client.json file deployed onto the client
    side (env hosts)

    **"config" : {** // the actual configuration txt follows …

    **"client" : {**

    **"address" : "{client\_name}",** // take the name of host from scan
    inventory

    **"subscriptions" : [** //a future option

    **], **

    **"environment" : "{env\_name}",** // name of the VIM environment
    for logs

    **"name" : "{client\_name}" //** take the name of host from scan
    inventory

    **}, **

    **"api" : {**

    **"host" : "{server\_ip}", //** take the server\_ip of
    environment\_config

    **"port" : NumberInt(4567)** // use this port for interfacing with
    the server

    **}**

    **}, **

    **"monitoring\_system" : "sensu"**

    **}**

    The above is just a simple example, login to mongoDB and check
    “monitoring\_config\_templates” collection for recent information on
    deployment files and their configuration options.

    The results of the monitoring provisioning are placed, by default,
    in mongoDB at the collection – “\ **monitoring\_config**\ ”, locally
    on the calipso-scan container at **/local\_dir/sensu\_config** and
    finally on the server (calipso-sensu container) and on all the hosts
    (clients). Here is an example of the resulted client.json file on
    one of the environment hosts (example deployment from real VIM
    environment):

    **{**

    **"api": {**

    **"host": "korlev-calipso-dev.cisco.com",**

    **"port": 4567**

    **},**

    **"checks": {**

    **"host\_pnic\_eno16777728-00---..58..---50---..58..---56---..58..---ac---..58..---e8---..58..---97":
    {**

    **"command": "check\_pnic\_ovs.py eno16777728",**

    **"handlers": [**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **},**

    **"host\_pnic\_eno33554952-00---..58..---50---..58..---56---..58..---ac---..58..---c9---..58..---a2":
    {**

    **"command": "check\_pnic\_ovs.py eno33554952",**

    **"handlers": [**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **},**

    **"otep\_node-6.cisco.com-otep\_vxlan-c0a80201": {**

    **"command": "check\_ping.py -c 10 -i 0.5 -p 4f532d444e41 -w 10 -s
    256 -f 192.168.2.2 -t 192.168.2.1 -W 1%/301.11/600 -C
    10%/1020.12/2000",**

    **"handlers": [**

    **"default",**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **},**

    **"otep\_node-6.cisco.com-otep\_vxlan-c0a80203": {**

    **"command": "check\_ping.py -c 10 -i 0.5 -p 4f532d444e41 -w 10 -s
    256 -f 192.168.2.2 -t 192.168.2.3 -W 1%/301.11/600 -C
    10%/1020.12/2000",**

    **"handlers": [**

    **"default",**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **},**

    **"vedge\_bc865c43-3dc5-4940-af1d-b4be59df1bd0": {**

    **"command": "check\_vedge\_ovs.py",**

    **"handlers": [**

    **"default",**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **},**

    **"vservice\_qdhcp-6c5ddc76-fcd7-4bdd-bff4-1d08b88b96ca": {**

    **"command": "PYTHONPATH=/etc/sensu/plugins check\_vservice.py dhcp
    qdhcp-6c5ddc76-fcd7-4bdd-bff4-1d08b88b96ca",**

    **"handlers": [**

    **"default",**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **},**

    **"vservice\_qdhcp-721f9c95-3042-4840-b8a4-83968c1e92b6": {**

    **"command": "PYTHONPATH=/etc/sensu/plugins check\_vservice.py dhcp
    qdhcp-721f9c95-3042-4840-b8a4-83968c1e92b6",**

    **"handlers": [**

    **"default",**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **},**

    **"vservice\_qdhcp-cc7ea40b-bb11-4b51-8e51-1a3b7abd283d": {**

    **"command": "PYTHONPATH=/etc/sensu/plugins check\_vservice.py dhcp
    qdhcp-cc7ea40b-bb11-4b51-8e51-1a3b7abd283d",**

    **"handlers": [**

    **"default",**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **},**

    **"vservice\_qrouter-1833846f-573e-45ef-8c87-3f7df530cdbd": {**

    **"command": "PYTHONPATH=/etc/sensu/plugins check\_vservice.py
    router qrouter-1833846f-573e-45ef-8c87-3f7df530cdbd",**

    **"handlers": [**

    **"default",**

    **"file",**

    **"osdna-monitor"**

    **],**

    **"interval": 15,**

    **"standalone": true,**

    **"subscribers": [**

    **"base"**

    **],**

    **"type": "metric"**

    **}**

    **},**

    **"client": {**

    **"address": "Mirantis-Liberty-node-6.cisco.com",**

    **"environment": "Mirantis-Liberty",**

    **"name": "Mirantis-Liberty-node-6.cisco.com",**

    **"subscriptions": []**

    **}**

    **}**

    All sensu configuration files, keys and scripts are eventually
    deployed, both on clients and on server side at the following
    locations:

    *SSL keys:* **/etc/sensu/ssl**

    *Calipso monitoring checks*: **/etc/sensu/plugins **

    *Configuration files:* **/etc/sensu/conf.d **

    The calipso-scan container is in-charge of the actual deployment (in
    case environment\_config is configured with needed details and
    “provision” = “Deploy.

    Calipso-scan then uses the calipso-sensu as the target “sensu
    server” to deploy and all the environment pre-discovered hosts as
    “sensu clients” to deploy, all pointing their monitoring results
    back to the calipso-sensu container.

    Calipso-scan uses ssh to access all hosts (through the master-host,
    see admin-guide) and also to access calipso-sensu container on port
    20022 to upload all customized files and places them in the above
    locations.

Calipso Apex monitoring integration
-----------------------------------

    For OPNFV version ‘P’ of the calipso application, farther automation
    has been developed for ‘zero touch’ automation. Calipso has a
    built-in ‘apex-configurator’ that runs at apex install phase
    (current scenario: os-nosdn-calipso-noha) and deploys the sensu
    clients themselves with all needed configurations, per apex install
    parameters and customizes the calipso-sensu container accordingly…no
    manual UI or API steps are needed for end-to-end functionality.

.. |image0| image:: media/image1.png
   :width: 6.50000in
   :height: 4.27153in
.. |image1| image:: media/image10.png
   :width: 6.50000in
   :height: 3.62708in