summaryrefslogtreecommitdiffstats
path: root/Scenario_Seperate_Sections
diff options
context:
space:
mode:
Diffstat (limited to 'Scenario_Seperate_Sections')
-rw-r--r--Scenario_Seperate_Sections/Section_1_2_VNF UseCases.rst731
-rw-r--r--Scenario_Seperate_Sections/Section_3_Communication_Interfaces.rst80
-rw-r--r--Scenario_Seperate_Sections/Section_4_UseCases_for_Network_Nodes.rst157
-rw-r--r--Scenario_Seperate_Sections/Section_5_Storage-HA-Scenarios.rst442
-rw-r--r--Scenario_Seperate_Sections/Section_6_multi_site.rst45
-rw-r--r--Scenario_Seperate_Sections/StorageImages/FTCluster.pngbin0 -> 92027 bytes
-rw-r--r--Scenario_Seperate_Sections/StorageImages/HostStorageCluster.pngbin0 -> 78913 bytes
-rw-r--r--Scenario_Seperate_Sections/StorageImages/RedundantStoragePaths.pngbin0 -> 173416 bytes
-rw-r--r--Scenario_Seperate_Sections/images/Slide10.pngbin0 -> 119323 bytes
-rw-r--r--Scenario_Seperate_Sections/images/Slide11.pngbin0 -> 143237 bytes
-rw-r--r--Scenario_Seperate_Sections/images/Slide13.pngbin0 -> 118070 bytes
-rw-r--r--Scenario_Seperate_Sections/images/Slide14.pngbin0 -> 142022 bytes
-rw-r--r--Scenario_Seperate_Sections/images/Slide16.pngbin0 -> 113076 bytes
-rw-r--r--Scenario_Seperate_Sections/images/Slide17.pngbin0 -> 137640 bytes
-rw-r--r--Scenario_Seperate_Sections/images/Slide19.pngbin0 -> 115266 bytes
-rw-r--r--Scenario_Seperate_Sections/images/Slide20.pngbin0 -> 114247 bytes
-rw-r--r--Scenario_Seperate_Sections/images/Slide4.pngbin0 -> 118822 bytes
-rw-r--r--Scenario_Seperate_Sections/images/Slide6.pngbin0 -> 141097 bytes
-rw-r--r--Scenario_Seperate_Sections/images/StatefullVNF-VMfailure.pngbin0 -> 51353 bytes
-rw-r--r--Scenario_Seperate_Sections/images/StatefullVNF-VMfailureNoRed.pngbin0 -> 39956 bytes
-rw-r--r--Scenario_Seperate_Sections/images/StatefullVNF-VNFCfailure.pngbin0 -> 28838 bytes
-rw-r--r--Scenario_Seperate_Sections/images/StatefullVNF-VNFCfailureNoRed.pngbin0 -> 30368 bytes
-rw-r--r--Scenario_Seperate_Sections/images/StatelessVNF-VMfailure.pngbin0 -> 48843 bytes
-rw-r--r--Scenario_Seperate_Sections/images/StatelessVNF-VMfailureNoRed.pngbin0 -> 35560 bytes
-rw-r--r--Scenario_Seperate_Sections/images/StatelessVNF-VNFCfailure.pngbin0 -> 26125 bytes
-rw-r--r--Scenario_Seperate_Sections/images/StatelessVNF-VNFCfailureNoRed-Escalation.pngbin0 -> 41571 bytes
-rw-r--r--Scenario_Seperate_Sections/images/StatelessVNF-VNFCfailureNoRed.pngbin0 -> 18897 bytes
-rw-r--r--Scenario_Seperate_Sections/images_network_nodes/DHCP_deployment.pngbin0 -> 25005 bytes
-rw-r--r--Scenario_Seperate_Sections/images_network_nodes/DPCH_failure.pngbin0 -> 30573 bytes
-rw-r--r--Scenario_Seperate_Sections/images_network_nodes/L3_deployment.pngbin0 -> 28040 bytes
-rw-r--r--Scenario_Seperate_Sections/images_network_nodes/L3_failure.pngbin0 -> 28900 bytes
-rw-r--r--Scenario_Seperate_Sections/images_network_nodes/L3_ha_principle.pngbin0 -> 30089 bytes
-rw-r--r--Scenario_Seperate_Sections/images_network_nodes/L3_ha_workflow.pngbin0 -> 11844 bytes
-rw-r--r--Scenario_Seperate_Sections/images_network_nodes/LBaaS_deployment.pngbin0 -> 29490 bytes
-rw-r--r--Scenario_Seperate_Sections/images_network_nodes/LBaaS_failure.pngbin0 -> 31159 bytes
-rw-r--r--Scenario_Seperate_Sections/images_network_nodes/Network_nodes_deployment.pngbin0 -> 77307 bytes
36 files changed, 1455 insertions, 0 deletions
diff --git a/Scenario_Seperate_Sections/Section_1_2_VNF UseCases.rst b/Scenario_Seperate_Sections/Section_1_2_VNF UseCases.rst
new file mode 100644
index 0000000..57dfbc0
--- /dev/null
+++ b/Scenario_Seperate_Sections/Section_1_2_VNF UseCases.rst
@@ -0,0 +1,731 @@
+============
+HA Use Cases
+============
+
+**************
+1 Introduction
+**************
+
+This use case document outlines the model and failure modes for NFV systems. Its goal is along
+with the requirements documents and gap analysis help set context for engagement with various
+upstream projects. The OPNFV HA project team continuously evolving these documents, and in
+particular this use case document starting with a set of basic use cases.
+
+*****************
+2 Basic Use Cases
+*****************
+
+
+In this section we review some of the basic use cases related to service high availability,
+that is, the availability of the service or function provided by a VNF. The goal is to
+understand the different scenarios that need to be considered and the specific requirements
+to provide service high availability. More complex use cases will be discussed in
+other sections.
+
+With respect to service high availability we need to consider whether a VNF implementation is
+statefull or stateless and if it includes or not an HA manager which handles redundancy.
+For statefull VNFs we can also distinguish the cases when the state is maintained inside
+of the VNF or it is stored in an external shared storage making the VNF itself virtually
+stateless.
+
+Managing availability usually implies a fault detection mechanism, which triggers the
+actions necessary for fault isolation followed by the recovery from the fault.
+This recovery includes two parts:
+
+* the recovery of the service and
+* the repair of the failed entity.
+
+Very often the recovery of the service and the repair actions are perceived to be the same, for
+example, restarting a failed application repairs the application, which then provides the service again.
+Such a restart may take significant time causing service outage, for which redundancy is the solution.
+In cases when the service is protected by redundancy of the providing entities (e.g. application
+processes), the service is "failed over" to the standby or a spare entity, which replaces the
+failed entity while it is being repaired. E.g. when an application process providing the service fails,
+the standby application process takes over providing the service, while the failed one is restarted.
+Such a failover often allows for faster recovery of the service.
+
+We also need to distinguish between the failed and the faulty entities as a fault may or
+may not manifest in the entity containing the fault. Faults may propagate, i.e. cause other entities
+to fail or misbehave, i.e. an error, which in turn might be detected by a different failure or
+error detector entity each of which has its own scope. Similarly, the managers acting on these
+detected errors may have a limited scope. E.g. an HA manager contained in a VNF can only repair
+entities within the VNF. It cannot repair a failed VM, in fact due to the layered architecture
+in the VNF it cannot even know whether the VM failed, its hosting hypervisor, or the physical host.
+But its error detection mechanism will detect the result of such failures - a failure in the VNF -
+and the service can be recovered at the VNF level.
+On the other hand, the failure should be detected in the NFVI and the VIM should repair the failed
+entity (e.g. the VM). Accordingly a failure may be detected by different managers in different layers
+of the system, each of which may react to the event. This may cause interference.
+Thus, to resolve the problem in a consistent manner and completely recover from
+a failure the managers may need to collaborate and coordinate their actions.
+
+Considering all these issues the following basic use cases can be identified (see table 1.).
+These use cases assume that the failure is detected in the faulty entity (VNF component
+or the VM).
+
+
+*Table 1: VNF high availability use cases*
+
++---------+-------------------+----------------+-------------------+----------+
+| | VNF Statefullness | VNF Redundancy | Failure detection | Use Case |
++=========+===================+================+===================+==========+
+| VNF | yes | yes | VNF level only | UC1 |
+| | | +-------------------+----------+
+| | | | VNF & NFVI levels | UC2 |
+| | +----------------+-------------------+----------+
+| | | no | VNF level only | UC3 |
+| | | +-------------------+----------+
+| | | | VNF & NFVI levels | UC4 |
+| +-------------------+----------------+-------------------+----------+
+| | no | yes | VNF level only | UC5 |
+| | | +-------------------+----------+
+| | | | VNF & NFVI levels | UC6 |
+| | +----------------+-------------------+----------+
+| | | no | VNF level only | UC7 |
+| | | +-------------------+----------+
+| | | | VNF & NFVI levels | UC8 |
++---------+-------------------+----------------+-------------------+----------+
+
+As discussed, there is no guarantee that a fault manifests within the faulty entity. For
+example, a memory leak in one process may impact or even crash any other process running in
+the same execution environment. Accordingly, the repair of a failing entity (i.e. the crashed process)
+may not resolve the problem and soon the same or another process may fail within this execution
+environment indicating that the fault has remained in the system.
+Thus, there is a need for extrapolating the failure to a wider scope and perform the
+recovery at that level to get rid of the problem (at least temporarily till a patch is available
+for our leaking process).
+This requires the correlation of repeated failures in a wider scope and the escalation of the
+recovery action to this wider scope. In the layered architecture this means that the manager detecting the
+failure may not be the one in charge of the scope at which it can be resolved, so the escalation needs to
+be forwarded to the manager in charge of that scope, which brings us to an additional use case UC9.
+
+We need to consider for each of these use cases the events detected, their impact on other entities,
+and the actions triggered to recover the service provided by the VNF, and to repair the
+faulty entity.
+
+We are going to describe each of the listed use cases from this perspective to better
+understand how the problem of service high availability can be tackled the best.
+
+Before getting into the details it is worth mentioning the example end-to-end service recovery
+times provided in the ETSI NFV REL document [REL]_ (see table 2.). These values may change over time
+including lowering these thresholds.
+
+*Table 2: Service availability levels (SAL)*
+
++----+---------------+----------------------+------------------------------------+
+|SAL |Service |Customer Type | Recommendation |
+| |Recovery | | |
+| |Time | | |
+| |Threshold | | |
++====+===============+======================+====================================+
+|1 |5 - 6 seconds |Network Operator |Redundant resources to be |
+| | |Control Traffic |made available on-site to |
+| | | |ensure fastrecovery. |
+| | |Government/Regulatory | |
+| | |Emergency Services | |
++----+---------------+----------------------+------------------------------------+
+|2 |10 - 15 seconds|Enterprise and/or |Redundant resources to be available |
+| | |large scale customers |as a mix of on-site and off-site |
+| | | |as appropriate: On-site resources to|
+| | |Network Operators |be utilized for recovery of |
+| | |service traffic |real-time service; Off-site |
+| | | |resources to be utilized for |
+| | | |recovery of data services |
++----+---------------+----------------------+------------------------------------+
+|3 |20 - 25 seconds|General Consumer |Redundant resources to be mostly |
+| | |Public and ISP |available off-site. Real-time |
+| | |Traffic |services should be recovered before |
+| | | |data services |
++----+---------------+----------------------+------------------------------------+
+
+Note that even though SAL 1 of [REL]_ allows for 5-6 seconds of service recovery,
+for many services this is too long and such outage causes a service level reset or
+the loss of significant amount of data. Also the end-to-end service or network service
+may be served by multiple VNFs. Therefore for a single VNF the desired
+service recovery time is sub-second.
+
+Note that failing over the service to another provider entity implies the redirection of the traffic
+flow the VNF is handling. This could be achieved in different ways ranging from floating IP addresses
+to load balancers. The topic deserves its own investigation, therefore in these first set of
+use cases we assume that it is part of the solution without going into the details, which
+we will address as a complementary set of use cases.
+
+.. [REL] ETSI GS NFV-REL 001 V1.1.1 (2015-01)
+
+
+2.1 Use Case 1: VNFC failure in a statefull VNF with redundancy
+==============================================================
+
+Use case 1 represents a statefull VNF with redundancy managed by an HA manager,
+which is part of the VNF (Fig 1). The VNF consists of VNFC1, VNFC2 and the HA Manager.
+The latter managing the two VNFCs, e.g. the role they play in providing the service
+named "Provided NF" (Fig 2).
+
+The failure happens in one of the VNFCs and it is detected and handled by the HA manager.
+On practice the HA manager could be part of the VNFC implementations or it could
+be a separate entity in the VNF. The point is that the communication of these
+entities inside the VNF is not visible to the rest of the system. The observable
+events need to cross the boundary represented by the VNF box.
+
+
+.. figure:: images/Slide4.png
+ :alt: VNFC failure in a statefull VNF
+ :figclass: align-center
+
+ Fig 1. VNFC failure in a statefull VNF with built-in HA manager
+
+
+.. figure:: images/StatefullVNF-VNFCfailure.png
+ :alt: MSC of the VNFC failure in a statefull VNF
+ :figclass: align-center
+
+ Fig 2. Sequence of events for use case 1
+
+
+As shown in Fig 2. initially VNFC2 is active, i.e. provides the Provided NF and VNFC1
+is a standby. It is not shown, but it is expected that VNFC1 has some means to get the update
+of the state of the Provided NF from the active VNFC2, so that it is prepared to continue to
+provide the service in case VNFC2 fails.
+The sequence of events starts with the failure of VNFC2, which also interrupts the
+Provided NF. This failure is detected somehow and/or reported to the HA Manager, which
+in turn may report the failure to the VNFM and simultaneously it tries to isolate the
+fault by cleaning up VNFC2.
+
+Once the cleanup succeeds (i.e. the OK is received) it fails over the active role to
+VNFC1 by setting it active. This recovers the service, the Provided NF is indeed
+provided again. Thus this point marks the end of the outage caused by the failure
+that need to be considered from the perspective of service availability.
+
+The repair of the failed VNFC2, which might have started at the same time
+when VNFC1 was assigned the active state, may take longer but without further impact
+on the availability of the Provided NF service.
+If the HA Manager reported the interruption of the Provided NF to the VNFM, it should
+clear the error condition.
+
+The key points in this scenario are:
+
+* The failure of the VNFC2 is not detectable by any other part of the system except
+ the consumer of the Provided NF. The VNFM only
+ knows about the failure because of the error report, and only the information this
+ report provides. I.e. it may or may not include the information on what failed.
+* The Provided NF is resumed as soon as VNFC1 is assigned active regardless how long
+ it takes to repair VNFC2.
+* The HA manager could be part of the VNFM as well. This requires an interface to
+ detect the failures and to manage the VNFC life-cycle and the role assignments.
+
+2.2 Use Case 2: VM failure in a statefull VNF with redundacy
+============================================================
+
+Use case 2 also represents a statefull VNF with its redundancy managed by an HA manager,
+which is part of the VNF. The VNFCs of the VNF are hosted on the VMs provided by
+the NFVI (Fig 3).
+
+The VNF consists of VNFC1, VNFC2 and the HA Manager (Fig 4). The latter managing
+the role the VNFCs play in providing the service - Provided NF.
+The VMs provided by the NFVI are managed by the VIM.
+
+
+In this use case it is one of the VMs hosting the VNF fails. The failure is detected
+and handled at both the NFVI and the VNF levels simultaneously. The coordination occurs
+between the VIM and the VNFM.
+
+
+.. figure:: images/Slide6.png
+ :alt: VM failure in a statefull VNF
+ :figclass: align-center
+
+ Fig 3. VM failure in a statefull VNF with built-in HA manager
+
+
+.. figure:: images/StatefullVNF-VMfailure.png
+ :alt: MSC of the VM failure in a statefull VNF
+ :figclass: align-center
+
+ Fig 4. Sequence of events for use case 2
+
+
+Again initially VNFC2 is active and provides the Provided NF, while VNFC1 is the standby.
+It is not shown in Fig 4., but it is expected that VNFC1 has some means to learn the state
+of the Provided NF from the active VNFC2, so that it is able to continue providing the
+service if VNFC2 fails. VNFC1 is hosted on VM1, while VNFC2 is hosted on VM2 as indicated by
+the arrows between these objects in Fig 4.
+
+The sequence of events starts with the failure of VM2, which results in VNFC2 failing and
+interrupting the Provided NF. The HA Manager detects the failure of VNFC2 somehow
+and tries to handle it the same way as in use case 1. However because the VM is gone the
+clean up either not initiated at all or interrupted as soon as the failure of the VM is
+identified. In either case the faulty VNFC2 is considered as isolated.
+
+To recover the service the HA Manager fails over the active role to VNFC1 by setting it active.
+This recovers the Provided NF. Thus this point marks again the end of the outage caused
+by the VM failure that need to be considered from the perspective of service availability.
+If the HA Manager reported the interruption of the Provided NF to the VNFM, it should
+clear the error condition.
+
+On the other hand the failure of the VM is also detected in the NFVI and reported to the VIM.
+The VIM reports the VM failure to the VNFM, which passes on this information
+to the HA Manager of the VNF. This confirms for the VNF HA Manager the VM failure and that
+it needs to wait with the repair of the failed VNFC2 until the VM is provided again. The
+VNFM also confirms towards the VIM that it is safe to restart the VM.
+
+The repair of the failed VM may take some time, but since the service has been failed over
+to VNFC1 in the VNF, there is no further impact on the availability of Provided NF.
+
+When eventually VM2 is restarted the VIM reports this to the VNFM and
+the VNFC2 can be restored.
+
+The key points in this scenario are:
+
+* The failure of the VM2 is detectable at both levels VNF and NFVI, therefore both the HA
+ manager and the VIM reacts to it. It is essential that these reactions do not interfere,
+ e.g. if the VIM tries to protect the VM state at NFVI level that would conflict with the
+ service failover action at the VNF level.
+* While the failure detection happens at both NFVI and VNF levels, the time frame within
+ which the VIM and the HA manager detect and react may be very different. For service
+ availability the VNF level detection, i.e. by the HA manager is the critical one and expected
+ to be faster.
+* The Provided NF is resumed as soon as VNFC1 is assigned active regardless how long
+ it takes to repair VM2 and VNFC2.
+* The HA manager could be part of the VNFM as well.
+ This requires an interface to detect failures in/of the VNFC and to manage its life-cycle and
+ role assignments.
+* The VNFM may not know for sure that the VM failed until the VIM reports it, i.e. whether
+ the VM failure is due to host, hypervisor, host OS failure. Thus the VIM should report/alarm
+ and log VM, hypervisor, and physical host failures. The use cases for these failures
+ are similar with respect to the Provided NF.
+* The VM repair also should start with the fault isolation as appropriate for the actual
+ failed entity, e.g. if the VM failed due to a host failure a host may be fenced first.
+* The negotiation between the VNFM and the VIM may be replaced by configured repair actions.
+ E.g. on error restart VM in initial state, restart VM from last snapshot, or fail VM over to standby.
+
+
+2.3 Use Case 3: VNFC failure in a statefull VNF with no redundancy
+=================================================================
+
+Use case 3 also represents a statefull VNF, but it stores its state externally on a
+virtual disk provided by the NFVI. It has a single VNFC and it is managed by the VNFM
+(Fig 5).
+
+In this use case the VNFC fails and the failure is detected and handled by the VNFM.
+
+
+.. figure:: images/Slide10.png
+ :alt: VNFC failure in a statefull VNF No-Red
+ :figclass: align-center
+
+ Fig 5. VNFC failure in a statefull VNF with no redundancy
+
+
+.. figure:: images/StatefullVNF-VNFCfailureNoRed.png
+ :alt: MSC of the VNFC failure in a statefull VNF No-Red
+ :figclass: align-center
+
+ Fig 6. Sequence of events for use case 3
+
+
+The VNFC periodically checkpoints the state of the Provided NF to the external storage,
+so that in case of failure the Provided NF can be resumed (Fig 6).
+
+When the VNFC fails the Provided NF is interrupted. The failure is detected by the VNFM
+somehow, which to isolate the fault first cleans up the VNFC, then if the cleanup is
+successful it restarts the VNFC. When the VNFC starts up, first it reads the last checkpoint
+for the Provided NF, then resumes providing it. The service outage lasts from the VNFC failure
+till this moment.
+
+The key points in this scenario are:
+
+* The service state is saved in an external storage which should be highly available too to
+ protect the service.
+* The NFVI should provide this guarantee and also that storage and access network failures
+ are handled seemlessly from the VNF's perspective.
+* The VNFM has means to detect VNFC failures and manage its life-cycle appropriately. This is
+ not required if the VNF also provides its availability management.
+* The Provided NF can be resumed only after the VNFC is restarted and it has restored the
+ service state from the last checkpoint created before the failure.
+* Having a spare VNFC can speed up the service recovery. This requires that the VNFM coordinates
+ the role each VNFC takes with respect to the Provided NF. I.e. the VNFCs do not act on the
+ stored state simultaneously potentially interfering and corrupting it.
+
+
+
+2.4 Use Case 4: VM failure in a statefull VNF with no redundancy
+===============================================================
+
+Use case 4 also represents a statefull VNF without redundancy, which stores its state externally on a
+virtual disk provided by the NFVI. It has a single VNFC managed by the VNFM
+(Fig 7) as in use case 3.
+
+In this use case the VM hosting the VNFC fails and the failure is detected and handled by
+the VNFM and the VIM simultaneously.
+
+
+.. figure:: images/Slide11.png
+ :alt: VM failure in a statefull VNF No-Red
+ :figclass: align-center
+
+ Fig 7. VM failure in a statefull VNF with no redundancy
+
+.. figure:: images/StatefullVNF-VMfailureNoRed.png
+ :alt: MSC of the VM failure in a statefull VNF No-Red
+ :figclass: align-center
+
+ Fig 8. Sequence of events for use case 4
+
+Again, the VNFC regularly checkpoints the state of the Provided NF to the external storage,
+so that it can be resumed in case of a failure (Fig 8).
+
+When the VM hosting the VNFC fails the Provided NF is interrupted.
+
+On the one hand side, the failure is detected by the VNFM somehow, which to isolate the fault tries
+to clean the VNFC up which cannot be done because of the VM failure. When the absence of the VM has been
+determined the VNFM has to wait with restarting the VNFC until the hosting VM is restored. The VNFM
+may report the problem to the VIM, requesting a repair.
+
+On the other hand the failure is detected in the NFVI and reported to the VIM, which reports it
+to the VNFM, if the VNFM hasn't reported it yet.
+If the VNFM has requested the VM repair or if it acknowledges the repair, the VIM restarts the VM.
+Once the VM is up the VIM reports it to the VNFM, which in turn can restart the VNFC.
+
+When the VNFC restarts first it reads the last checkpoint for the Provided NF,
+to be able to resume it.
+The service outage last until this is recovery completed.
+
+The key points in this scenario are:
+
+
+* The service state is saved in external storage which should be highly available to
+ protect the service.
+* The NFVI should provide such a guarantee and also that storage and access network failures
+ are handled seemlessly from the perspective of the VNF.
+* The Provided NF can be resumed only after the VM and the VNFC are restarted and the VNFC
+ has restored the service state from the last checkpoint created before the failure.
+* The VNFM has means to detect VNFC failures and manage its life-cycle appropriately. Alternatively
+ the VNF may also provide its availability management.
+* The VNFM may not know for sure that the VM failed until the VIM reports this. It also cannot
+ distinguish host, hypervisor and host OS failures. Thus the VIM should report/alarm and log
+ VM, hypervisor, and physical host failures. The use cases for these failures are
+ similar with respect to the Provided NF.
+* The VM repair also should start with the fault isolation as appropriate for the actual
+ failed entity, e.g. if the VM failed due to a host failure a host may be fenced first.
+* The negotiation between the VNFM and the VIM may be replaced by configured repair actions.
+* VM level redundancy, i.e. running a standby or spare VM in the NFVI would allow faster service
+ recovery for this use case, but by itself it may not protect against VNFC level failures. I.e.
+ VNFC level error detection is still required.
+
+
+
+2.5 Use Case 5: VNFC failure in a stateless VNF with redundancy
+===============================================================
+
+Use case 5 represents a stateless VNF with redundancy, i.e. it is composed of VNFC1 and VNFC2.
+They are managed by an HA manager within the VNF. The HA manager assigns the active role to provide
+the Provided NF to one of the VNFCs while the other remains a spare meaning that it has no state
+information for the Provided NF (Fig 9) therefore it could replace any other VNFC capable of
+providing the Provided NF service.
+
+In this use case the VNFC fails and the failure is detected and handled by the HA manager.
+
+
+.. figure:: images/Slide13.png
+ :alt: VNFC failure in a stateless VNF with redundancy
+ :figclass: align-center
+
+ Fig 9. VNFC failure in a stateless VNF with redundancy
+
+
+.. figure:: images/StatelessVNF-VNFCfailure.png
+ :alt: MSC of the VNFC failure in a stateless VNF with redundancy
+ :figclass: align-center
+
+ Fig 10. Sequence of events for use case 5
+
+
+Initially VNFC2 provides the Provided NF while VNFC1 is idle or might not even been instantiated
+yet (Fig 10).
+
+When VNFC2 fails the Provided NF is interrupted. This failure is detected by the HA manager,
+which as a first reaction cleans up VNFC2 (fault isolation), then it assigns the active role to
+VNFC1. It may report an error to the VNFM as well.
+
+Since there is no state information to recover, VNFC1 can accept the active role right away
+and resume providing the Provided NF service. Thus the service outage is over. If the HA manager
+reported an error to the VNFM it should clear it at this point.
+
+The key points in this scenario are:
+
+* The spare VNFC may be instantiated only once the failure of active VNFC is detected.
+* As a result the HA manager's role might be limited to life-cycle management, i.e. no role
+ assignment is needed if the VNFCs provide the service as soon as they are started up.
+* Accordingly the HA management could be part of a generic VNFM provided it is capable of detecting
+ the VNFC failures. Besides the service users, the VNFC failure may not be detectable at any other
+ part of the system.
+* Also there could be multiple active VNFCs sharing the load of Provided NF and the spare/standby
+ may protect all of them.
+* Reporting the service failure to the VNFM is optional as the HA manager is in charge of recovering
+ the service and it is aware of the redundancy needed to do so.
+
+
+2.6 Use Case 6: VM failure in a stateless VNF with redundancy
+============================================================
+
+
+Similarly to use case 5, use case 6 represents a stateless VNF composed of VNFC1 and VNFC2,
+which are managed by an HA manager within the VNF. The HA manager assigns the active role to
+provide the Provided NF to one of the VNFCs while the other remains a spare meaning that it has
+no state information for the Provided NF (Fig 11) and it could replace any other VNFC capable
+of providing the Provided NF service.
+
+As opposed to use case 5 in this use case the VM hosting one of the VNFCs fails. This failure is
+detected and handled by the HA manager as well as the VIM.
+
+
+.. figure:: images/Slide14.png
+ :alt: VM failure in a stateless VNF with redundancy
+ :figclass: align-center
+
+ Fig 11. VM failure in a stateless VNF with redundancy
+
+
+.. figure:: images/StatelessVNF-VMfailure.png
+ :alt: MSC of the VM failure in a stateless VNF with redundancy
+ :figclass: align-center
+
+ Fig 12. Sequence of events for use case 6
+
+
+Initially VNFC2 provides the Provided NF while VNFC1 is idle or might not have been instantiated
+yet (Fig 12) as in use case 5.
+
+When VM2 fails VNFC2 fails with it and the Provided NF is interrupted. The failure is detected by
+the HA manager and by the VIM simultaneously and independently.
+
+The HA manager's first reaction is trying to clean up VNFC2 to isolate the fault. This is considered to
+be successful as soon as the disappearance of the VM is confirmed.
+After this the HA manager assigns the active role to VNFC1. It may report the error to the VNFM as well
+requesting a VM repair.
+
+Since there is no state information to recover, VNFC1 can accept the assignment right away
+and resume the Provided NF service. Thus the service outage is over. If the HA manager reported
+an error to the VNFM for the service it should clear it at this point.
+
+Simultaneously the VM failure is detected in the NFVI and reported to the VIM, which reports it
+to the VNFM, if the VNFM hasn't requested a repair yet. If the VNFM requested the VM repair or if
+it acknowledges the repair, the VIM restarts the VM.
+
+Once the VM is up the VIM reports it to the VNFM, which in turn may restart the VNFC if needed.
+
+
+The key points in this scenario are:
+
+* The spare VNFC may be instantiated only after the detection of the failure of the active VNFC.
+* As a result the HA manager's role might be limited to life-cycle management, i.e. no role
+ assignment is needed if the VNFC provides the service as soon as it is started up.
+* Accordingly the HA management could be part of a generic VNFM provided if it is capable of detecting
+ failures in/of the VNFC and managing its life-cycle.
+* Also there could be multiple active VNFCs sharing the load of Provided NF and the spare/standby
+ may protect all of them.
+* The VNFM may not know for sure that the VM failed until the VIM reports this. It also cannot
+ distinguish host, hypervisor and host OS failures. Thus the VIM should report/alarm and log
+ VM, hypervisor, and physical host failures. The use cases for these failures are
+ similar with respect to each Provided NF.
+* The VM repair also should start with the fault isolation as appropriate for the actual
+ failed entity, e.g. if the VM failed due to a host failure a host needs to be fenced first.
+* The negotiation between the VNFM and the VIM may be replaced by configured repair actions.
+* Reporting the service failure to the VNFM is optional as the HA manager is in charge recovering
+ the service and it is aware of the redundancy needed to do so.
+
+
+
+2.7 Use Case 7: VNFC failure in a stateless VNF with no redundancy
+==================================================================
+
+Use case 7 represents a stateless VNF composed of a single VNFC, i.e. with no redundancy.
+The VNF and in particular its VNFC is managed by the VNFM through managing its life-cycle (Fig 13).
+
+In this use case the VNFC fails. This failure is detected and handled by the VNFM. This use case
+requires that the VNFM can detect the failures in the VNF or they are reported to the VNFM.
+
+The failure is only detectable at the VNFM level and it is handled by the VNFM restarting the VNFC.
+
+
+.. figure:: images/Slide16.png
+ :alt: VNFC failure in a stateless VNF with no redundancy
+ :figclass: align-center
+
+ Fig 13. VNFC failure in a stateless VNF with no redundancy
+
+
+.. figure:: images/StatelessVNF-VNFCfailureNoRed.png
+ :alt: MSC of the VNFC failure in a stateless VNF with no redundancy
+ :figclass: align-center
+
+ Fig 14. Sequence of events for use case 7
+
+The VNFC is providing the Provided NF when it fails (Fig 14). This failure is detected or reported to
+the VNFM, which has to clean up the VNFC to isolate the fault. After cleanup success it can proceed
+with restarting the VNFC, which as soon as it is up it starts to provide the Provided NF
+as there is no state to recover.
+
+Thus the service outage is over, but it has included the entire time needed to restart the VNFC.
+Considering that the VNF is stateless this may not be significant still.
+
+
+The key points in this scenario are:
+
+* The VNFM has to have the means to detect VNFC failures and manage its life-cycle appropriately.
+ This is not required if the VNF comes with its availability management, but this is very unlikely
+ for such stateless VNFs.
+* The Provided NF can be resumed as soon as the VNFC is restarted, i.e. the restart time determines
+ the outage.
+* In case multiple VNFCs are used they should not interfere with one another, they should
+ operate independently.
+
+
+2.8 Use Case 8: VM failure in a stateless VNF with no redundancy
+================================================================
+
+Use case 8 represents the same stateless VNF composed of a single VNFC as use case 7, i.e. with
+no redundancy. The VNF and in particular its VNFC is managed by the VNFM through managing its
+life-cycle (Fig 15).
+
+In this use case the VM hosting the VNFC fails. This failure is detected and handled by the VNFM
+as well as by the VIM.
+
+
+.. figure:: images/Slide17.png
+ :alt: VM failure in a stateless VNF with no redundancy
+ :figclass: align-center
+
+ Fig 15. VM failure in a stateless VNF with no redundancy
+
+
+.. figure:: images/StatelessVNF-VMfailureNoRed.png
+ :alt: MSC of the VM failure in a stateless VNF with no redundancy
+ :figclass: align-center
+
+ Fig 16. Sequence of events for use case 8
+
+The VNFC is providing the Provided NF when the VM hosting the VNFC fails (Fig 16).
+
+This failure may be detected or reported to the VNFM as a failure of the VNFC. The VNFM may
+not be aware at this point that it is a VM failure. Accordingly its first reaction as in use case 7
+is to clean up the VNFC to isolate the fault. Since the VM is gone, this cannot succeed and the VNFM
+becomes aware of the VM failure through this or it is reported by the VIM. In either case it has to wait
+with the repair of the VMFC until the VM becomes available again.
+
+Meanwhile the VIM also detects the VM failure and reports it to the VNFM unless the VNFM has already
+requested the VM repair. After the VNFM confirming the VM repair the VIM restarts the VM and reports
+the successful repair to the VNFM, which in turn can start the VNFC hosted on it.
+
+
+Thus the recovery of the Provided NF includes the restart time of the VM and of the VNFC.
+
+The key points in this scenario are:
+
+* The VNFM has to have the means to detect VNFC failures and manage its life-cycle appropriately.
+ This is not required if the VNF comes with its availability management, but this is very unlikely
+ for such stateless VNFs.
+* The Provided NF can be resumed only after the VNFC is restarted on the repaired VM, i.e. the
+ restart time of the VM and the VNFC determines the outage.
+* In case multiple VNFCs are used they should not interfere with one another, they should
+ operate independently.
+* The VNFM may not know for sure that the VM failed until the VIM reports this. It also cannot
+ distinguish host, hypervisor and host OS failures. Thus the VIM should report/alarm and log
+ VM, hypervisor, and physical host failures. The use cases for these failures are
+ similar with respect to each Provided NF.
+* The VM repair also should start with the fault isolation as appropriate for the actual
+ failed entity, e.g. if the VM failed due to a host failure the host needs to be fenced first.
+* The repair negotiation between the VNFM and the VIM may be replaced by configured repair actions.
+* VM level redundancy, i.e. running a standby or spare VM in the NFVI would allow faster service
+ recovery for this use case, but by itself it may not protect against VNFC level failures. I.e.
+ VNFC level error detection is still required.
+
+2.9 Use Case 9: Repeated VNFC failure in a stateless VNF with no redundancy
+===========================================================================
+
+Finally use case 9 represents again a stateless VNF composed of a single VNFC as in use case 7, i.e.
+with no redundancy. The VNF and in particular its VNFC is managed by the VNFM through managing its
+life-cycle.
+
+In this use case the VNFC fails repeatedly. This failure is detected and handled by the VNFM,
+but results in no resolution of the fault (Fig 17) because the VNFC is manifesting a fault,
+which is not in its scope. I.e. the fault is propagating to the VNFC from a faulty VM or host,
+for example. Thus the VNFM cannot resolve the problem by itself.
+
+
+.. figure:: images/Slide19.png
+ :alt: Repeated VNFC failure in a stateless VNF with no redundancy
+ :figclass: align-center
+
+ Fig 17. VM failure in a stateless VNF with no redundancy
+
+
+To handle this case the failure handling needs to be escalated to the a bigger fault zone
+(or fault domain), i.e. a scope within which the faults may propagate and manifest. In case of the
+VNF the bigger fault zone is the VM and the facilities hosting it, all managed by the VIM.
+
+Thus the VNFM should request the repair from the VIM (Fig 18).
+
+Since the VNFM is only aware of the VM, it needs to report an error on the VM and it is the
+VIM's responsibility to sort out what might be the scope of the actual fault depending on other
+failures and error reports in its scope.
+
+
+.. figure:: images/Slide20.png
+ :alt: Escalation of repeated VNFC failure in a stateless VNF with no redundancy
+ :figclass: align-center
+
+ Fig 18. VM failure in a stateless VNF with no redundancy
+
+
+.. figure:: images/StatelessVNF-VNFCfailureNoRed-Escalation.png
+ :alt: MSC of the VM failure in a stateless VNF with no redundancy
+ :figclass: align-center
+
+ Fig 19. Sequence of events for use case 9
+
+
+This use case starts similarly to use case 7, i.e. the VNFC is providing the Provided NF when it fails
+(Fig 17).
+This failure is detected or reported to the VNFM, which cleans up the VNFC to isolate the fault.
+After successful cleanup the VNFM proceeds with restarting the VNFC, which as soon as it is up
+starts to provide the Provided NF again as in use case 7.
+
+However the VNFC failure occurs N times repeatedly within some Probation time for which the VNFM starts
+the timer when it detects the first failure of the VNFC. When the VNFC fails once more still within the
+probation time the Escalation counter maximum is exceeded and the VNFM reports an error to the VIM on
+the VM hosting the VNFC as obviously cleaning up and restarting the VNFC did not solve the problem.
+
+When the VIM receives the error report for the VM it has to isolate the fault by cleaning up at least
+the VM. After successful cleanup it can restart the VM and once it is up report the VM repair to the VNFM.
+At this point the VNFM can restart the VNFC, which in turn resumes the Provided VM.
+
+In this scenario the VIM needs to evaluate what may be the scope of the fault to determine what entity
+needs a repair. For example, if it has detected VM failures on that same host, or other VNFMs
+reported errors on VMs hosted on the same host, it should consider that the entire host needs a repair.
+
+
+The key points in this scenario are:
+
+* The VNFM has to have the means to detect VNFC failures and manage its life-cycle appropriately.
+ This is not required if the VNF comes with its availability management, but this is very unlikely
+ for such stateless VNFs.
+* The VNFM needs to correlate VNFC failures over time to be able to detect failure of a bigger fault zone.
+ One way to do so is through counting the failures within a probation time.
+* The VIM cannot detect all failures caused by faults in the entities under its control. It should be
+ able to receive error reports and correlate these error reports based on the dependencies
+ of the different entities.
+* The VNFM does not know the source of the failure, i.e. the faulty entity.
+* The VM repair should start with the fault isolation as appropriate for the actual
+ failed entity, e.g. if the VM failed due to a host failure the host needs to be fenced first.
+
+********************
+3 Concluding remarks
+********************
+
+This use case document outlined the model and some failure modes for NFV systems. These are an
+initial list. The OPNFV HA project team is continuing to grow the list of use cases and will
+issue additional documents going forward. The basic use cases and service availability considerations
+help define the key considerations for each use case taking into account the impact on the end service.
+The use case document along with the requirements documents and gap analysis help set context for
+engagement with various upstream projects.
diff --git a/Scenario_Seperate_Sections/Section_3_Communication_Interfaces.rst b/Scenario_Seperate_Sections/Section_3_Communication_Interfaces.rst
new file mode 100644
index 0000000..c97776b
--- /dev/null
+++ b/Scenario_Seperate_Sections/Section_3_Communication_Interfaces.rst
@@ -0,0 +1,80 @@
+3. Communication Interfaces for VNF HA schemes
+===========================================================
+
+This section will discuss some general issues about communication interfaces
+in the VNF HA schemes. In sections 2, the usecases of both stateful and
+stateless VNFs are discussed. While in this section, we would like to discuss
+some specific issues which are quite general for all the usecases proposed
+in the previous sections.
+
+3.1. VNF External Interfaces
+
+Regardless whether the VNF is stateful or stateless, all the VNFCs should act as
+a union from the perspective of the outside world. That means all the VNFCs should
+share a common interface where the outside modules (e.g., the other VNFs) can
+access the service from. There could be multiple solutions for this share of IP
+interface. However, all of this sharing and switching of IP address should be
+ignorant to the outside modules.
+
+There are several approaches for the VNFs to share the interfaces. A few of them
+are listed as follows and will be discussed in detail.
+
+1) IP address of VMs for active/stand-by VM.
+
+2) Load balancers for active/active use cases
+
+Note that combinition of these two approaches is also feasible.
+
+For active/standby VNFCs, there is a common IP address shared by the VMs hosting
+the active and standby VNFCs, so that they look as one instance from outside.
+The HA manager will manage the assignment of the IP address to the VMs.
+(The HA manager may not be aware of this, I.e. the address may be configured
+and the active/standby state management is linked to the possession of the IP
+address, i.e. the active VNFC claims it as part of becoming active.) Only the
+active one possesses the IP address. And when failover happens, the standby
+is set to be active and can take possession of the IP address to continue traffic
+process.
+
+
+For active/active VNFCs, a LB(Load Balancer) could be used. In such scenario, there
+could be two cases for the deployment and usage of LB.
+
+Case 1: LB used in front of a cluster of VNFCs to distribute the traffic flow.
+
+In such case, the LB is deployed in front of a cluster of multiple VNFCs. Such
+cluster can be managed by a seperate cluster manager, or can be managed just
+by the LB, which uses heartbeat to monitor each VNFC. When one of VNFCs fails,
+the cluster manager should first exclude the failed VNFC from the cluster so that
+the LB will re-route the traffic to the other VNFCs, and then the failed one should
+be recovered. In the case when the LB is acting as the cluster manager, it is
+the LB's responsibility to inform the VNFM to recover the failed VNFC if possible.
+
+
+Case 2: LB used in front of a cluster of VMs to distribute traffic flow.
+
+In this case, there exists a cluster manager(e.g. Pacemaker) to monitor and manage
+the VMs in the cluster. The LB sits in front of the VM cluster so as to distribute
+the traffic. When one of the VM fails, the cluster manager will detect that and will
+be in charge of the recovery. The cluster manager will also exclude the failed VM
+out of the cluster, so that the LB won't route traffic to the failed one.
+
+In both two cases, the HA of the LB should also be considered.
+
+
+3.2. Intra-VNF Communication
+
+For stateful VNFs, data synchronization is necessary between the active and standby VMs.
+The HA manager is responsible for handling VNFC failover, and do the assignment of the
+active/standby states between the VNFCs of the VNF. Data synchronization can be handled
+either by the HA manager or by the VNFC itself.
+
+The state synchronization can happen as
+
+- direct communication between the active and the standby VNFCs
+
+- based on the information received from the HA manager on channel or messages using a common queue,
+
+- it could be through a shared storage assigned to the whole VNF
+
+- through the checkpointing of state information via underlying memory and/or
+database checkpointing services to a separate VM and storage repository.
diff --git a/Scenario_Seperate_Sections/Section_4_UseCases_for_Network_Nodes.rst b/Scenario_Seperate_Sections/Section_4_UseCases_for_Network_Nodes.rst
new file mode 100644
index 0000000..bc9266a
--- /dev/null
+++ b/Scenario_Seperate_Sections/Section_4_UseCases_for_Network_Nodes.rst
@@ -0,0 +1,157 @@
+4 High Availability Scenarios for Network Nodes
+===============================================
+
+4.1 Network nodes and HA deployment
+-----------------------------------
+
+OpenStack network nodes contain: Neutron DHCP agent, Neutron L2 agent, Neutron L3 agent, Neutron LBaaS
+agent and Neutron Metadata agent. The DHCP agent provides DHCP services for virtual networks. The
+metadata agent provides configuration information such as credentials to instances. Note that the
+L2 agent cannot be distributed and highly available. Instead, it must be installed on each data
+forwarding node to control the virtual network drivers such as Open vSwitch or Linux Bridge. One L2
+agent runs per node and controls its virtual interfaces.
+
+A typical HA deployment of network nodes can be achieved in Fig 20. Here shows a two nodes cluster.
+The number of the nodes is decided by the size of the cluster. It can be 2 or more. More details can be
+achieved from each agent's part.
+
+
+.. figure:: images_network_nodes/Network_nodes_deployment.png
+ :alt: HA deployment of network nodes
+ :figclass: align-center
+
+ Fig 20. A typical HA deployment of network nodes
+
+
+4.2 DHCP agent
+--------------
+
+The DHCP agent can be natively highly available. Neutron has a scheduler which lets you run multiple
+agents across nodes. You can configure the dhcp_agents_per_network parameter in the neutron.conf file
+and set it to X (X >=2 for HA, default is 1).
+
+If the X is set to 2, as depicted in Fig 21 three tenant networks (there can be multiple tenant networks)
+are used as an example, six DHCP agents are deployed in two nodes for three networks, they are
+all active. Two dhcp1s serve one network, dhcp2s and dhcp3s serve other two different networks. In a
+network, all DHCP traffic is broadcast, DHCP servers race to offer IP. All the servers will update the
+lease tables. In Fig 22, when the agent(s) in Node1 doesn't work which can be caused by software
+failure or hardware failure, the dhcp agent(s) on Node2 will continue to offer IP for the network.
+
+
+.. figure:: images_network_nodes/DHCP_deployment.png
+ :alt: HA deployment of DHCP agents
+ :figclass: align-center
+
+ Fig 21. Natively HA deployment of DHCP agents
+
+
+.. figure:: images_network_nodes/DHCP_failure.png
+ :alt: Failure of DHCP agents
+ :figclass: align-center
+
+ Fig 22. Failure of DHCP agents
+
+
+4.3 L3 agent
+------------
+
+The L3 agent is also natively highly available. To achieve HA, it can be configured in the neutron.conf
+file.
+
+.. code-block:: bash
+
+ l3_ha = True # All routers are highly available by default
+
+ allow_automatic_l3agent_failover = True # Set automatic L3 agent failover for routers
+
+ max_l3_agents_per_router = 2 # Maximum number of network nodes to use for the HA router
+
+ min_l3_agents_per_router = 2 # Minimum number of network nodes to use for the HA router. A new router
+ can be created only if this number of network nodes are available.
+
+According to the neutron.conf file, the L3 agent scheduler supports Virtual Router Redundancy
+Protocol (VRRP) to distribute virtual routers across multiple nodes (e.g. 2). The scheduler will choose
+a number between the maximum and the minimum number according scheduling algorithm. VRRP is implemented
+by Keepalived.
+
+As depicted in Fig 23, both L3 agents in Node1 and Node2 host vRouter 1 and vRouter 2. In Node 1,
+vRouter 1 is active and vRouter 2 is standby (hot standby). In Node2, vRouter 1 is standby and
+vRouter 2 is active. For the purpose of reducing the load, two actives are deployed in two Nodes
+alternatively. In Fig 24, Keepalived will be used to manage the VIP interfaces. One instance of
+keepalived per virtual router, then one per namespace. 169.254.192.0/18 is a dedicated HA network
+which is created in order to isolate the administrative traffic from the tenant traffic, each vRouter
+will be connected to this dedicated network via an HA port. More details can be achieved from the
+Reference at the bottom.
+
+
+.. figure:: images_network_nodes/L3_deployment.png
+ :alt: HA deployment of L3 agents
+ :figclass: align-center
+
+ Fig 23. Natively HA deployment of L3 agents
+
+
+.. figure:: images_network_nodes/L3_ha_principle.png
+ :alt: HA principle of L3 agents
+ :figclass: align-center
+
+ Fig 24. Natively HA principle of L3 agents
+
+
+In Fig 25, when vRouter 1 in Node1 is down which can be caused by software failure or hardware failure,
+the Keepalived will detect the failure and the standby will take over to be active. In order to keep the
+TCP connection, Conntrackd is used to maintain the TCP sessions going through the router. One instance
+of conntrackd per virtual router, then one per namespace. After then, a rescheduling procedure will be
+triggered to respawn the failed virtual router to another l3 agent as standby. All the workflows is
+depicted in Fig 26.
+
+
+.. figure:: images_network_nodes/L3_failure.png
+ :alt: Failure of L3 agents
+ :figclass: align-center
+
+ Fig 25. Failure of L3 agents
+
+
+.. figure:: images_network_nodes/L3_ha_workflow.png
+ :alt: HA workflow of L3 agents
+ :figclass: align-center
+
+ Fig 26. HA workflow of L3 agents
+
+
+4.4 LBaaS agent and Metadata agent
+----------------------------------
+
+Currently, no native feature is provided to make the LBaaS agent highly available using the defaul
+plug-in HAProxy. A common way to make HAProxy highly available is to use Pacemaker.
+
+
+.. figure:: images_network_nodes/LBaaS_deployment.png
+ :alt: HA deployment of LBaaS agents
+ :figclass: align-center
+
+ Fig 27. HA deployment of LBaaS agents using Pacemaker
+
+
+As shown in Fig 27 HAProxy and pacemaker are deployed in both of the network nodes. The number of network
+nodes can be 2 or more. It depends on your cluster. HAProxy in Node 1 is the master and the VIP is in
+Node 1. Pacemaker monitors the liveness of HAProxy.
+
+
+.. figure:: images_network_nodes/LBaaS_failure.png
+ :alt: Failure of LBaaS agents
+ :figclass: align-center
+
+ Fig 28. Failure of LBaaS agents
+
+
+As shown in Fig 28 when HAProxy in Node1 falls down which can be caused by software failure or hardware
+failure, Pacemaker will fail over HAProxy and the VIP to Node 2.
+
+Note that the default plug-in HAProxy only supports TCP and HTTP.
+
+No native feature is available to make Metadata agent highly available. At this time, the Active/Passive
+solution exists to run the neutron metadata agent in failover mode with Pacemaker. The deployment and
+failover procedure can be the same as the case of LBaaS.
+
diff --git a/Scenario_Seperate_Sections/Section_5_Storage-HA-Scenarios.rst b/Scenario_Seperate_Sections/Section_5_Storage-HA-Scenarios.rst
new file mode 100644
index 0000000..b8b37a3
--- /dev/null
+++ b/Scenario_Seperate_Sections/Section_5_Storage-HA-Scenarios.rst
@@ -0,0 +1,442 @@
+Storage and High Availability Scenarios
+=======================================
+
+5.1 Elements of HA Storage Management and Delivery
+--------------------------------------------------
+
+Storage infrastructure, in any environment, can be broken down into two
+domains: Data Path and Control Path. Generally, High Availability of the
+storage infrastructure is measured by the occurence of Data
+Unavailability and Data Loss (DU/DL) events. While that meaning is
+obvious as it relates to the Data Path, it is also applicable to Control
+Path as well. The inability to attach a volume that has data to a host,
+for example, can be considered a Data Unavailability event. Likewise,
+the inability to create a volume to store data could be considered Data
+Loss since it may result in the inability to store critical data.
+
+Storage HA mechanisms are an integral part of most High Availability
+solutions today. In the first two sections below, we define the
+mechanisms of redundancy and protection required in the infrastructure
+for storage delivery in both the Data and Control Paths. Storage
+services that have these mechanisms can be used in HA environments that
+are based on a highly available storage infrastructure.
+
+In the third section below, we examine HA implementations that rely on
+highly available storage infrastructure. Note that the scope throughout this
+section is focused on local HA solutions. This does not address rapid remote
+Disaster Recovery scenarios that may be provided by storage, nor
+does it address metro active/active environments that implement stretched
+clusters of hosts across multiple sites for workload migration and availability.
+
+
+5.2 Storage Failure & Recovery Scenarios: Storage Data Path
+-----------------------------------------------------------
+
+In the failure and recovery scenarios described below, a redundant
+network infrastructure provides HA through network-related device
+failures, while a variety of strategies are used to reduce or minimize
+DU/DL events based on storage system failures. This starts with redundant
+storage network paths, as shown in Figure 29.
+
+.. figure:: StorageImages/RedundantStoragePaths.png
+ :alt: HA Storage Infrastructure
+ :figclass: align-center
+
+ Figure 29: Typical Highly Available Storage Infrastructure
+
+Storage implementations vary tremendously, and the recovery mechanisms
+for each implementation will vary. These scenarios described below are
+limited to 1) high level descriptions of the most common implementations
+since it is unpredictable as to
+which storage implementations may be used for NFVI; 2) HW- and
+SW-related failures (and recovery) of the storage data path, and not
+anything associated with user configuration and operational issues which
+typically create the most common storage failure scenarios; 3)
+non-LVM/DAS based storage implementations(managing failure and recovery
+in LVM-based storage for OpenStack is a very different scenario with
+less of a reliable track record); and 4) I will assume block storage
+only, and not object storage, which is often used for stateless
+applications (at a high level, object stores may include a
+subset of the block scenarios under the covers).
+
+To define the requirements for the data path, I will start at the
+compute node and work my way down the storage IO stack and touch on both
+HW and SW failure/recovery scenarios for HA along the way. I will use Figure 1 as a reference.
+
+1. Compute IO driver: Assuming iSCSI for connectivity between the
+compute and storage, an iSCSI initiator on the compute node maintains
+redundant connections to multiple iSCSI targets for the same storage
+service. These redundant connections may be aggregated for greater
+throughput, or run independently. This redundancy allows the iSCSI
+Initiator to handle failures in network connectivity from compute to
+storage infrastructure. (Fibre Channel works largely the same way, as do
+proprietary drivers that connect a host's IO stack to storage systems).
+
+2. Compute node network interface controller (NIC): This device may
+fail, and said failure reported via whatever means is in place for such
+reporting from the host.The redundant paths between iSCSI initiators and
+targets will allow connectivity from compute to storage to remain up,
+though operating at reduced capacity.
+
+3. Network Switch failure for storage network: Assuming there are
+redundant switches in place, and everything is properly configured so
+that two compute NICs go to two separate switches, which in turn go to
+two different storage controllers, then a switch may fail and the
+redundant paths between iSCSI initiators and targets allows connectivity
+from compute to storage to operational, though operating at reduced
+capacity.
+
+4. Storage system network interface failure: Assuming there are
+redundant storage system network interfaces (on separate storage
+controllers), then one may fail and the redundant paths between iSCSI
+initiators and targets allows connectivity from compute to storage to
+remain operational, though operating at reduced performance. The extent
+of the reduced performance is dependent upon the storage architecture.
+See 3.5 for more.
+
+5. Storage controller failure: A storage system can, at a very high
+level, be described as composed of network interfaces, one or more
+storage controllers that manage access to data, and a shared Data Path
+access to the HDD/SSD subsystem. The network interface failure is
+described in #4, and the HDD/SSD subsystem is described in #6. All
+modern storage architectures have either redundant or distributed
+storage controller architectures. In **dual storage controller
+architectures**, high availability is maintained through the ALUA
+protocol maintaining access to primary and secondary paths to iSCSI
+targets. Once a storage controller fails, the array operates in
+(potentially) degraded performance mode until the failed storage controller is
+replaced. The degree of reduced performance is dependent on the overall
+original load on the array. Dual storage controller arrays also remain at risk
+of a Data Unavailability event if the second storage controller should fail.
+This is rare, but should be accounted for in planning support and
+maintenance contracts.
+
+**Distributed storage controller architectures** are generally server-based,
+which may or may not operate on the compute servers in Converged
+Infrastructure environments. Hence the concept of “storage controller”
+is abstract in that it may involve a distribution of software components
+across multiple servers. Examples: Ceph and ScaleIO. In these environments,
+the data may be stored
+redundantly, and metadata for accessing the data in these redundant
+locations is available for whichever compute node needs the data (with
+authorization, of course). Data may also be stored using erasure coding
+(EC) for greater efficiency. The loss of a storage controller in this
+context leads to a discussion of impact caused by loss of a server in
+this distributed storage controller architecture. In the event of such a loss,
+if data is held in duplicate or triplicate on other servers, then access
+is simply redirected to maintain data availability. In the case of
+EC-based protection, then the data is simply re-built on the fly. The
+performance and increased risk impact in this case is dependent on the
+time required to rebalance storage distribution across other servers in
+the environment. Depending on configuration and implementation, it could
+impact storage access performance to VNFs as well.
+
+6. HDD/SSD subsystem: This subsystem contains any RAID controllers,
+spinning hard disk drives, and Solid State Drives. The failure of a RAID
+controller is equivalent to failure of a storage controller, as
+described in 5 above. The failure of one or more storage devices is
+protected by either RAID parity-based protection, Erasure Coding
+protection, or duplicate/triplicate storage of the data. RAID and
+Erasure Coding are typically more efficient in terms of space
+efficiency, but duplicate/triplicate provides better performance. This
+tradeoff is a common point of contention among implementations, and this
+will not go into greater detail than to assume that failed devices do
+not cause Data Loss events due to these protection algorithms. Multiple
+device failures can potentially cause Data Loss events, and the risk of
+each method must be taken into consideration for the HA requirements of
+the desired deployment.
+
+5.3 Storage Failure & Recovery Scenarios: Storage Control Path
+--------------------------------------------------------------
+
+As it relates to an NFVI environment, as proposed by OPNFV, there are
+two parts to the storage control path.
+
+* The storage system-specific control path to the storage controller
+
+* The OpenStack-specific cloud management framework for managing different
+storage elements
+
+
+5.3.1 Storage System Control Paths
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+High Availability of a storage controller is storage
+system-specific. Breaking it down to implementation variants is the best
+approach. However, both variants assume an IP-based management API in
+order to leverage network redundancy mechanisms for ubiquitous
+management access.
+
+An appliance style storage array with dual storage controllers must implement IP
+address failover for the management API's IP endpoint in either an
+active/active or active/passive configuration. Likewise, a storage array
+with >2 storage controllers would bring up a management endpoint on
+another storage controller in such an event. Cluster-style IP address load
+balancing is also a viable implementation in these scenarios.
+
+In the case of distributed storage controller architectures, the storage system
+provides redundant storage controller interfaces. E.g., Ceph's RADOS provides
+redundant paths to access an OSD for volume creation or access. In EMC's
+ScaleIO, there are redundant MetaData Managers for managing volume
+creation and access. In the case of the former, the access is via
+proprietary protocol, in the case of the latter, it is via HTTP-based
+REST API. Other storage implementations may also provide alternative
+methods, but any enterprise-class storage system will have built-in HA
+for management API access.
+
+Finally, note that single server-based storage solutions, such as LVM,
+do not have HA solutions for control paths. If the server is failed, the
+management of that server's storage is not available.
+
+5.3.2 OpenStack Controller Management
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+OpenStack cloud management is comprised of a number of different
+function-specific management modules such as Keystone for Identity and
+Access management (IAM), Nova for compute management, Cinder for block
+storage management, Swift for Object Storage delivery, Neutron for
+Network management, and Glance as an image repository. In smaller
+single-cloud environments, these management systems are managed in
+concert for High Availability; in larger multi-cloud environments, the
+Keystone IAM may logically stand alone in its own HA delivery across the
+multiple clouds, as might Swift as a common Object Store. Nova, Cinder,
+and Glance may have separate scopes of management, but they are more
+typically managed together as a logical cloud deployment.
+
+It is the OpenStack deployment mechanisms that are responsible for HA
+deployment of these HA management infrastructures. These tools, such as
+Fuel, RDO, and others, have matured to include highly available
+implementations for the database, the API, and each of the manager
+modules associated with the scope of cloud management domains.
+
+There are many interdependencies among these modules that impact Cinder high availability.
+For example:
+
+* Cinder is implemented as an Active/Standby failover implementation since it
+requires a single point of control at one time for the Cinder manager/driver implementation.
+The Cinder manager/driver is deployed on two of the three OpenStack controller nodes, and
+one is made active while the other is passive. This may be improved to active/active
+in a future release.
+
+* A highly available database implementation must be delivered
+using something like MySQL/Galera replication across the 3 OpenStack controller
+nodes. Cinder requires an HA database in order for it to be HA.
+
+* A redundant RabbitMQ messaging implementation across the same
+three OpenStack controller nodes. Likewise, Cinder requires an HA messaging system.
+
+* A redundant OpenStack API to ensure Cinder requests can be delivered.
+
+* An HA Cluster Manager, like PaceMaker for monitoring each of the
+deployed manager elements on the OpenStack controllers, with restart capability.
+Keepalived is an alternative implementation for monitoring processes and restarting on
+alternate OpenStack controller nodes. While statistics are lacking, it is generally
+believed that the PaceMaker implementation is more frequently implemented
+in HA environments.
+
+
+For more information on OpenStack and Cinder HA, see http://docs.openstack.org/ha-guide
+for current thinking.
+
+While the specific combinations of management functions in these
+redundant OpenStack controllers may vary with the specific small/large environment
+deployment requirements, the basic implementation of three OpenStack controller
+redundancy remains relatively common. In these implementations, the
+highly available OpenStack controller environment provides HA access to
+the highly available storage controllers via the highly available IP
+network.
+
+
+5.4 The Role of Storage in HA
+-----------------------------
+
+In the sections above, we describe data and control path requirements
+and example implementations for delivery of highly available storage
+infrastructure. In summary:
+
+* Most modern storage infrastructure implementations are inherently
+highly available. Exceptions certainly apply; e.g., simply using LVM for
+storage presentation at each server does not satisfy HA requirements.
+However, modern storage systems such as Ceph, ScaleIO, XIV, VNX, and
+many others with OpenStack integrations, certainly do have such HA
+capabilities.
+
+* This is predominantly through network-accessible shared storage
+systems in tightly coupled configurations such as clustered hosts, or in
+loosely coupled configurations such as with global object stores.
+
+
+Storage is an integral part of HA delivery today for applications,
+including VNFs. This is examined below in terms of using storage as a
+key part of HA delivery, the possible scope and limitations of that
+delivery, and example implementations for delivery of such service. We
+will examine this for both block and object storage infrastructures below.
+
+5.4.1 VNF, VNFC, and VM HA in a Block Storage HA Context
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Several scenarios were described in another section with regard to
+managing HA at the VNFC level, with variants of recovery based on either
+VIM- or VNFM-based reporting/detection/recovery mechanisms. In a block
+storage environment, these differentiations are abstract and
+meaningless, regardless of whether it is or is not intended to be HA.
+
+In a block storage context, HA is delivered via a logical block device
+(sometimes called a Logical Unit, or LUN), or in some cases, to a VM.
+VM and logical block devices are the units of currency.
+
+.. figure:: StorageImages/HostStorageCluster.png
+ :alt: Host Storage Cluster
+ :figclass: align-center
+
+ Figure 30: Typical HA Cluster With Shared Storage
+
+In Figure 30, several hosts all share access, via an IP network
+or via Fibre Channel, to a common set of logical storage devices. In an
+ESX cluster implementation, these hosts all access all devices with
+coordination provided with the SCSI Reservation mechanism. In the
+particular ESX case, the logical storage devices provided by the storage
+service actually aggregate volumes (VMDKs) utilized by VMs. As a result,
+multiple host access to the same storage service logical device is
+dynamic. The vSphere management layer provides for host cluster
+management.
+
+In other cases, such as for KVM, cluster management is not formally
+required, per se, because each logical block device presented by the
+storage service is uniquely allocated for one particular VM which can
+only execute on a single host at a time. In this case, any host that can
+access the same storage service is potentially a part of the "cluster".
+While *potential* access from another host to the same logical block
+device is necessary, the actual connectivity is restricted to one host
+at a time. This is more of a loosely coupled cluster implementation,
+rather than the tightly coupled cluster implementation of ESX.
+
+So, if a single VNF is implemented as a single VM, then HA is provided
+by allowing that VM to execute on a different host, with access to the
+same logical block device and persistent data for that VM, located on
+the storage service. This also applies to multiple VNFs implemented
+within a single VM, though it impacts all VNFs together.
+
+If a single VNF is implemented across multiple VMs as multiple VNFCs,
+then all of the VMs that comprise the VNF may need to be protected in a consistent
+fashion. The storage service is not aware of the
+distinction from the previous example. However, a higher level
+implementation, such as an HA Manager (perhaps implemented in a VNFM)
+may monitor and restart a collection of VMs on alternate hosts. In an ESX environment,
+VM restarts are most expeditiously handled by using vSphere-level HA
+mechanisms within an HA cluster for individual or collections of VMs.
+In KVM environments, a separate HA
+monitoring service, such as Pacemaker, can be used to monitor individual
+VMs, or entire multi-VM applications, and provide restart capabilities
+on separately configured hosts that also have access to the same logical
+storage devices.
+
+VM restart times, however, are measured in 10's of seconds. This may
+sometimes meet the SAL-3 recovery requirements for General Consumer,
+Public, and ISP Traffic, but will never meet the 5-6 seconds required
+for SAL-1 Network Operator Control and Emergency Services. For this,
+additional capabilities are necessary.
+
+In order to meet SAL-1 restart times, it is necessary to have: 1. A hot
+spare VM already up and running in an active/passive configuration 2.
+Little-to-no-state update requirements for the passive VM to takeover.
+
+Having a spare VM up and running is easy enough, but putting that VM in
+an appropriate state to take over execution is the difficult part. In shared storage
+implementations for Fault Tolerance, which can achieve SAL-1 requirements,
+the VMs share access to the same storage device, and another wrapper function
+is used to update internal memory state for every interaction to the active
+VM.
+
+This may be done in one of two ways, as illustrated in Figure 31. In the first way,
+the hypervisor sends all interface interactions to the passive as well
+as the active VM. The interaction is handled completely by
+hypervisor-to-hypervisor wrappers, as represented by the purple box encapsulating
+the VM in Figure 31, and is completely transparent to the VM.
+This is available with the vSphere Fault Tolerant option, but not with
+KVM at this time.
+
+.. figure:: StorageImages/FTCluster.png
+ :alt: FT host and storage cluster
+ :figclass: align-center
+
+ Figure 31: A Fault Tolerant Host/Storage Configuration
+
+In the second way, a VM-level wrapper is used to capture checkpoints of
+state from the active VM and transfers these to the passive VM, similarly represented
+as the purple box encapsulating the VM in Figure 3. There
+are various levels of application-specific integration required for this
+wrapper to capture and transfer checkpoints of state, depending on the
+level of state consistency required. OpenSAF is an example of an
+application wrapper that can be used for this purpose. Both techniques
+have significant network bandwidth requirements and may have certain
+limitations and requirements for implementation.
+
+In both cases, the active and passive VMs share the same storage infrastructure.
+Although the OpenSAF implementation may also utilize separate storage infrastructure
+as well (not shown in Figure 3).
+
+Looking forward to the long term, both of these may be made obsolete. As soon as 2016,
+PCIe fabrics will start to be available that enable shared NVMe-based
+storage systems. While these storage systems may be used with
+traditional protocols like SCSI, they will also be usable with true
+NVMe-oriented applications whose memory state are persisted, and can be
+shared, in an active/passive mode across hosts. The HA mechanisms here
+are yet to be defined, but will be far superior to either of the
+mechanisms described above. This is still a future.
+
+
+5.4.2 HA and Object stores in loosely coupled compute environments
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Whereas block storage services require tight coupling of hosts to
+storage services via SCSI protocols, the interaction of applications
+with HTTP-based object stores utilizes a very loosely coupled
+relationship. This means that VMs can come and go, or be organized as an
+N+1 redundant deployment of VMs for a given VNF. Each individual object
+transaction constitutes the duration of the coupling, whereas with
+SCSI-based logical block devices, the coupling is active for the
+duration of the VM's mounting of the device.
+
+However, the requirement for implementation here is that the state of a
+transaction being performed is made persistent to the object store by
+the VM, as the restartable checkpoint for high availability. Multiple
+VMs may access the object store somewhat simultaneously, and it is
+required that each object transaction is made idempotent by the
+application.
+
+HA restart of a transaction in this environment is dependent on failure
+detection and transaction timeout values for applications calling the
+VNFs. These may be rather high and even unachievable for the SAL
+requirements. For example, while the General Consumer, Public, and ISP
+Traffic recovery time for SAL-3 is 20-25 seconds, default browser
+timeouts are upwards of 120 seconds. Common default timeouts for
+applications using HTTP are typically around 10 seconds or higher
+(browsers are upward of 120 seconds), so this puts a requirement on the
+load balancers to manage and restart transactions in a timeframe that
+may be a challenge to meeting even SAL-3 requirements.
+
+Despite these issues of performance, the use of object storage for highly
+available solutions in native cloud applications is very powerful. Object
+storage services are generally globally distributed and replicated using
+eventual consistency techniques, though transaction-level consistency can
+also be achieved in some cases (at the cost of performance). (For an interesting
+discussion of this, lookup the CAP Theorem.)
+
+
+5.5 Summary
+-----------
+
+This section addressed several points:
+
+* Modern storage systems are inherently Highly Available based on modern and reasonable
+implementations and deployments.
+
+* Storage is typically a central component in offering highly available infrastructures,
+whether for block storage services for traditional applications, or through object
+storage services that may be shared globally with eventual consistency.
+
+* Cinder HA management capabilities are defined and available through the use of
+OpenStack deployment tools, making the entire storage control and data paths
+highly available.
+
diff --git a/Scenario_Seperate_Sections/Section_6_multi_site.rst b/Scenario_Seperate_Sections/Section_6_multi_site.rst
new file mode 100644
index 0000000..2e43471
--- /dev/null
+++ b/Scenario_Seperate_Sections/Section_6_multi_site.rst
@@ -0,0 +1,45 @@
+6, Multisite Scenario
+====================================================
+
+The Multisite scenario refers to the cases when VNFs are deployed on multiple VIMs.
+There could be three typical usecases for such scenario.
+
+One is in one DC, multiple openstack clouds are deployed. Taking into consideration that the
+number of compute nodes in one openstack cloud are quite limited (nearly 100) for
+both opensource and commercial product of openstack, multiple openstack clouds will
+have to be deployed in the DC to manage thousands of servers. In such a DC, it should
+be possible to deploy VNFs accross openstack clouds.
+
+
+Another typical usecase is Geographic Redundancy (GR). GR deployment is to deal with more
+catastrophic failures (flood, earthquake, propagating software fault, and etc.) of a single site.
+In the Geographic redundancy usecase, VNFs are deployed in two sites, which are
+geographically seperated and are deployed on NFVI managed by seperate VIM. When
+such a catastrophic failure happens, the VNFs at the failed site can failover to
+the redundant one so as to continue the service. Different VNFs may have specified
+requirement of such failover. Some VNFs may need stateful failover, while others
+may just need their VMs restarted on the redundant site in their initial state.
+The first would create the overhead of state replication. The latter may still
+have state replication through the storage. Accordingly for storage we don't want
+to loose any data, and for networking the NFs should be connected the same way as
+they were in the original site. We probably want also to have the same number of
+VMs on the redundant site coming up for the VNFs.
+
+
+The other usecase is the maintainance. When one site is planning for a maintaining,
+it should first replicate the service to another site before it stops them. Such
+replication should not disturb the service, nor should it cause any data loss. The
+service at the second site should be excuted, before the first site is stopped and
+began maintenance. In such case, the multisite schemes may be used.
+
+The multisite scenario is also captured by the Multisite project, in which specific
+requirements of openstack are also proposed for different usecases. However,
+the multisite project mainly focuses on the requirement of these multisite
+usecases on openstack. HA requirements are not necessarily the requirement
+for the approaches discussed in multisite. While the HA project tries to
+capture the HA requirements in these usecases. The following links are the scenarios
+and Usecases discussed in the Multisite project.
+https://gerrit.opnfv.org/gerrit/#/c/2123/
+https://gerrit.opnfv.org/gerrit/#/c/1438/.
+
+
diff --git a/Scenario_Seperate_Sections/StorageImages/FTCluster.png b/Scenario_Seperate_Sections/StorageImages/FTCluster.png
new file mode 100644
index 0000000..c42f4a0
--- /dev/null
+++ b/Scenario_Seperate_Sections/StorageImages/FTCluster.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/StorageImages/HostStorageCluster.png b/Scenario_Seperate_Sections/StorageImages/HostStorageCluster.png
new file mode 100644
index 0000000..baca591
--- /dev/null
+++ b/Scenario_Seperate_Sections/StorageImages/HostStorageCluster.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/StorageImages/RedundantStoragePaths.png b/Scenario_Seperate_Sections/StorageImages/RedundantStoragePaths.png
new file mode 100644
index 0000000..1850ca9
--- /dev/null
+++ b/Scenario_Seperate_Sections/StorageImages/RedundantStoragePaths.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/Slide10.png b/Scenario_Seperate_Sections/images/Slide10.png
new file mode 100644
index 0000000..b3545e8
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/Slide10.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/Slide11.png b/Scenario_Seperate_Sections/images/Slide11.png
new file mode 100644
index 0000000..3aa5f67
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/Slide11.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/Slide13.png b/Scenario_Seperate_Sections/images/Slide13.png
new file mode 100644
index 0000000..207c4a7
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/Slide13.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/Slide14.png b/Scenario_Seperate_Sections/images/Slide14.png
new file mode 100644
index 0000000..e6083c9
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/Slide14.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/Slide16.png b/Scenario_Seperate_Sections/images/Slide16.png
new file mode 100644
index 0000000..484ffa2
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/Slide16.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/Slide17.png b/Scenario_Seperate_Sections/images/Slide17.png
new file mode 100644
index 0000000..7240aaa
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/Slide17.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/Slide19.png b/Scenario_Seperate_Sections/images/Slide19.png
new file mode 100644
index 0000000..7e3c10b
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/Slide19.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/Slide20.png b/Scenario_Seperate_Sections/images/Slide20.png
new file mode 100644
index 0000000..2e9759b
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/Slide20.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/Slide4.png b/Scenario_Seperate_Sections/images/Slide4.png
new file mode 100644
index 0000000..a701f42
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/Slide4.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/Slide6.png b/Scenario_Seperate_Sections/images/Slide6.png
new file mode 100644
index 0000000..04a904f
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/Slide6.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/StatefullVNF-VMfailure.png b/Scenario_Seperate_Sections/images/StatefullVNF-VMfailure.png
new file mode 100644
index 0000000..2f62232
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/StatefullVNF-VMfailure.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/StatefullVNF-VMfailureNoRed.png b/Scenario_Seperate_Sections/images/StatefullVNF-VMfailureNoRed.png
new file mode 100644
index 0000000..6f3058d
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/StatefullVNF-VMfailureNoRed.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/StatefullVNF-VNFCfailure.png b/Scenario_Seperate_Sections/images/StatefullVNF-VNFCfailure.png
new file mode 100644
index 0000000..9021f2d
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/StatefullVNF-VNFCfailure.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/StatefullVNF-VNFCfailureNoRed.png b/Scenario_Seperate_Sections/images/StatefullVNF-VNFCfailureNoRed.png
new file mode 100644
index 0000000..4fd7e2e
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/StatefullVNF-VNFCfailureNoRed.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/StatelessVNF-VMfailure.png b/Scenario_Seperate_Sections/images/StatelessVNF-VMfailure.png
new file mode 100644
index 0000000..9b94183
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/StatelessVNF-VMfailure.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/StatelessVNF-VMfailureNoRed.png b/Scenario_Seperate_Sections/images/StatelessVNF-VMfailureNoRed.png
new file mode 100644
index 0000000..2a14b67
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/StatelessVNF-VMfailureNoRed.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/StatelessVNF-VNFCfailure.png b/Scenario_Seperate_Sections/images/StatelessVNF-VNFCfailure.png
new file mode 100644
index 0000000..f2dcc3b
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/StatelessVNF-VNFCfailure.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/StatelessVNF-VNFCfailureNoRed-Escalation.png b/Scenario_Seperate_Sections/images/StatelessVNF-VNFCfailureNoRed-Escalation.png
new file mode 100644
index 0000000..6719177
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/StatelessVNF-VNFCfailureNoRed-Escalation.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images/StatelessVNF-VNFCfailureNoRed.png b/Scenario_Seperate_Sections/images/StatelessVNF-VNFCfailureNoRed.png
new file mode 100644
index 0000000..a0970fc
--- /dev/null
+++ b/Scenario_Seperate_Sections/images/StatelessVNF-VNFCfailureNoRed.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images_network_nodes/DHCP_deployment.png b/Scenario_Seperate_Sections/images_network_nodes/DHCP_deployment.png
new file mode 100644
index 0000000..90bb740
--- /dev/null
+++ b/Scenario_Seperate_Sections/images_network_nodes/DHCP_deployment.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images_network_nodes/DPCH_failure.png b/Scenario_Seperate_Sections/images_network_nodes/DPCH_failure.png
new file mode 100644
index 0000000..07a51f8
--- /dev/null
+++ b/Scenario_Seperate_Sections/images_network_nodes/DPCH_failure.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images_network_nodes/L3_deployment.png b/Scenario_Seperate_Sections/images_network_nodes/L3_deployment.png
new file mode 100644
index 0000000..ff573b6
--- /dev/null
+++ b/Scenario_Seperate_Sections/images_network_nodes/L3_deployment.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images_network_nodes/L3_failure.png b/Scenario_Seperate_Sections/images_network_nodes/L3_failure.png
new file mode 100644
index 0000000..57485ad
--- /dev/null
+++ b/Scenario_Seperate_Sections/images_network_nodes/L3_failure.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images_network_nodes/L3_ha_principle.png b/Scenario_Seperate_Sections/images_network_nodes/L3_ha_principle.png
new file mode 100644
index 0000000..59a3161
--- /dev/null
+++ b/Scenario_Seperate_Sections/images_network_nodes/L3_ha_principle.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images_network_nodes/L3_ha_workflow.png b/Scenario_Seperate_Sections/images_network_nodes/L3_ha_workflow.png
new file mode 100644
index 0000000..d923f4f
--- /dev/null
+++ b/Scenario_Seperate_Sections/images_network_nodes/L3_ha_workflow.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images_network_nodes/LBaaS_deployment.png b/Scenario_Seperate_Sections/images_network_nodes/LBaaS_deployment.png
new file mode 100644
index 0000000..d4e5929
--- /dev/null
+++ b/Scenario_Seperate_Sections/images_network_nodes/LBaaS_deployment.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images_network_nodes/LBaaS_failure.png b/Scenario_Seperate_Sections/images_network_nodes/LBaaS_failure.png
new file mode 100644
index 0000000..5262fd0
--- /dev/null
+++ b/Scenario_Seperate_Sections/images_network_nodes/LBaaS_failure.png
Binary files differ
diff --git a/Scenario_Seperate_Sections/images_network_nodes/Network_nodes_deployment.png b/Scenario_Seperate_Sections/images_network_nodes/Network_nodes_deployment.png
new file mode 100644
index 0000000..bb0f3db
--- /dev/null
+++ b/Scenario_Seperate_Sections/images_network_nodes/Network_nodes_deployment.png
Binary files differ