1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
|
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
Doctor
^^^^^^
Platform overview
"""""""""""""""""
Doctor platform provides these features in `Colorado Release <https://wiki.opnfv.org/display/SWREL/Colorado>`_:
* Immediate Notification
* Consistent resource state awareness for compute host down
* Valid compute host status given to VM owner
These features enable high availability of Network Services on top of
the virtualized infrastructure. Immediate notification allows VNF managers
(VNFM) to process recovery actions promptly once a failure has occurred.
Consistency of resource state is necessary to execute recovery actions
properly in the VIM.
Ability to query host status gives VM owner the possibility to get
consistent state information through an API in case of a compute host
fault.
The Doctor platform consists of the following components:
* OpenStack Compute (Nova)
* OpenStack Telemetry (Ceilometer)
* OpenStack Alarming (Aodh)
* Doctor Inspector
* Doctor Monitor
.. note::
Doctor Inspector and Monitor are sample implementations for reference.
You can see an overview of the Doctor platform and how components interact in
:numref:`figure-p1`.
.. figure:: /platformoverview/images/figure-p1.png
:name: figure-p1
:width: 100%
Doctor platform and typical sequence (Colorado)
Detailed information on the Doctor architecture can be found in the Doctor
requirements documentation:
http://artifacts.opnfv.org/doctor/docs/requirements/05-implementation.html
Use case
""""""""
* A consumer of the NFVI wants to receive immediate notifications about faults
in the NFVI affecting the proper functioning of the virtual resources.
Therefore, such faults have to be detected as quickly as possible, and, when
a critical error is observed, the affected consumer is immediately informed
about the fault and can switch over to the STBY configuration.
The faults to be monitored (and at which detection rate) will be configured by
the consumer. Once a fault is detected, the Inspector in the Doctor
architecture will check the resource map maintained by the Controller, to find
out which virtual resources are affected and then update the resources state.
The Notifier will receive the failure event requests sent from the Controller,
and notify the consumer(s) of the affected resources according to the alarm
configuration.
Detailed workflow information is as follows:
* Consumer(VNFM): (step 0) creates resources (network, server/instance) and an
event alarm on state down notification of that server/instance
* Monitor: (step 1) periodically checks nodes, such as ping from/to each
dplane nic to/from gw of node, (step 2) once it fails to send out event
with "raw" fault event information to Inspector
* Inspector: when it receives an event, it will (step 3) mark the host down
("mark-host-down"), (step 4) map the PM to VM, and change the VM status to
down
* Controller: (step 5) sends out instance update event to Ceilometer
* Notifier: (step 6) Ceilometer transforms and passes the event to Aodh,
(step 7) Aodh will evaluate event with the registered alarm definitions,
then (step 8) it will fire the alarm to the "consumer" who owns the
instance
* Consumer(VNFM): (step 9) receives the event and (step 10) recreates a new
instance
Test case
"""""""""
Functest will call the "run.sh" script in Doctor to run the test job.
The "run.sh" script will execute the following steps.
Firstly, verify connectivity to target compute host according to different
installer and prepare image for booting VM. Currently, only 'Apex' and
'local' installer are supported.
Secondly, the Doctor components are started, and, based on the above
preparation, a test user (default as demo) will be created for the Doctor
tests.
Thirdly, the VM is booted, and an alarm event is created in Ceilometer.
After sleeping for 1 minute in order to wait for the VM launch to complete,
a failure is injected to the system, i.e. the network of compute host is
disabled for 3 minutes. To ensure the host is down, the status of the host
will be checked.
Finally, the notification time, i.e. the time between the execution of step 2
(Monitor detects failure) and step 9 (Consumer receives failure notification)
is calculated.
According to the Doctor requirements, the Doctor test is successful if the
notification time is below 1 second.
|
