docs: Add XCI overview document

Change-Id: I23405aec19a7e9bfce1c58d770b80b1c9c550d00
Signed-off-by: Fatih Degirmenci <fatih.degirmenci@ericsson.com>

6 files changed, 254 insertions, 25 deletions

diff --git a/docs/images/pipelines-parallel.png b/docs/images/pipelines-parallel.png
new file mode 100644
index 00000000..c46f00aa
--- /dev/null
+++ b/docs/images/pipelines-parallel.png
diff --git a/docs/images/pipelines-scenarios.png b/docs/images/pipelines-scenarios.png
new file mode 100644
index 00000000..a6b320c3
--- /dev/null
+++ b/docs/images/pipelines-scenarios.png
diff --git a/docs/images/pipelines-testprojects.png b/docs/images/pipelines-testprojects.png
new file mode 100644
index 00000000..6a37eb5d
--- /dev/null
+++ b/docs/images/pipelines-testprojects.png
diff --git a/docs/images/pipelines-upstream.png b/docs/images/pipelines-upstream.png
new file mode 100644
index 00000000..9750dc6d
--- /dev/null
+++ b/docs/images/pipelines-upstream.png
diff --git a/docs/images/source-based-dev.png b/docs/images/source-based-dev.png
new file mode 100644
index 00000000..1de2993a
--- /dev/null
+++ b/docs/images/source-based-dev.png
diff --git a/docs/xci-overview.rst b/docs/xci-overview.rst
index 7c677e3c..5a8aebb6 100644
--- a/docs/xci-overview.rst
+++ b/docs/xci-overview.rst
@@ -8,38 +8,267 @@
 Cross Community Continuous Integration
 ======================================
 
-
-This document will contain the overview, the pipelines, and stuff.
-
 Introduction
 ============
 
-OPNFV has an advanced Continuous Integration (CI) machinery that provides support
-to OPNFV community to develop, integrate, test and release the integrated
+OPNFV develops, operates, and maintains its Continuous Integration (CI) process
+used by OPNFV community to develop, integrate, test and release the integrated
 reference platform for NFV.
 
-During the past releases, OPNFV integrated, deployed and tested different
-flavors (scenarios) of the platform in an entirely automated fashion, resulting
-in feedback to OPNFV itself and the communities OPNFV works with. This enabled
-communities to implement new features directly in the upstream, identify bugs
-and issue fixes for them.
+During past releases, OPNFV released different flavors (scenarios) of the
+platform in an entirely automated fashion, resulting in feedback to OPNFV itself
+and other communities OPNFV works with. This enabled the community to implement
+new features, identify/fix bugs directly in corresponding
+upstream communities.
+
+The development and release model employed by OPNFV during the past releases
+used stable versions of upstream components. This helps developers and users
+who are focused on stability however requires a lot of time for development,
+integration, and testing, thus resulting in a slower pace of innovation.
+
+In order to increase the speed of development and evolution of the platform,
+OPNFV needs to provide means for its developers and users. One of the ways to
+achieve this is to ensure developers have access to the latest versions of the
+upstream components they are developing, integrating, and testing.
+
+Based on this need, OPNFV Infrastructure Working Group (Infra WG) started the
+Cross Community Continuous Integration (XCI) initiative, bringing in a new
+development and release model into OPNFV which is based on Continuous Delivery
+and DevOps practices and principles.
+
+Focus Areas
+===========
+
+Enabling Continuous Delivery
+----------------------------
+
+By definition, XCI focuses on master branches in order to
+
+* shorten the time it takes to introduce new features
+* make it easier to identify and fix bugs
+* ease the effort to develop, integrate, and test the reference
+  platform
+* establish additional feedback loops within OPNFV, towards the users
+  and between the communities OPNFV works with
+* increase the visibility regarding the state of things at all times
+
+XCI aims to enable this by applying basic CI/CD & DevOps principles and
+following best practices such as
+
+* fail fast, fix fast
+* always have working software
+* small and frequent commits
+* work against the trunk, shortening development time
+* fast and tailored feedback
+* everything is visible to everyone all the time
+* and others
+
+By doing this, the overall quality of the platform components provided by the
+upstream communities will increase greatly, making it easier for anyone to
+consume them when they need them with less trouble, helping to find and fix bugs
+much earlier than what it is today and develop new features.
+
+How good this can work depends on the nature of the changes. If the changes are
+atomic and more importantly complete, the value added by XCI will increase
+significantly.
+
+Putting Users and Developers First
+----------------------------------
+
+Apart from applying the principles and following the best practices, XCI puts
+the users and developers first by pushing for
+
+* less burden on developers as they do not need to be aware of all details that
+  are not of their interest/concern
+* reduced complexity
+* an easy way to try and develop things
+* speed, helping developers to bring their scenarios to OPNFV faster
+* a real scenario ownership
+
+The proof of the user and developer centric approach is that the first thing
+XCI made available is the sandbox for users and developers to try things out.
+
+Keeping Quality, Confidence and Predictability High
+---------------------------------------------------
+
+Another and perhaps the most crucial concern for XCI is to keep the quality high,
+increase the confidence, have predictability, and have the availability of the
+latest versions earlier. Some of the prerequisites to fulfill these goals are
+
+* Test early and often
+* Know the quality at all times
+* Make the platform available early so people have time to develop, integrate,
+  and test their work
+* Avoid big bang uplifts
+* Avoid surprises
+
+Source Based Deployments
+------------------------
+
+CI starts on developer workstation and this is the fastest feedback a developer
+can get. In order to ensure developers can apply this principle, they need the
+tools and ways to enable fast development and test cycles that are repeatable as
+many times as necessary without hassle.
+
+One way to achieve this is to bring developers closer to the source and remove
+anything between them. This means that what XCI brings is not only deploying from
+upstream master branches but doing that from source with no intermediaries in between.
+
+A simple scenario that demostrates the value of bringing capability of source based
+deployments to OPNFV can be seen on the diagram below.
+
+.. image:: images/source-based-dev.png
+   :height: 240px
+   :align: center
+
+As you can see on the diagram, XCI provides tools and ways for developers to
+
+* patch the source code on their laptop
+* get the patch deployed to the stack with a single command
+* run the tests
+* fix if something is broken
+* repeat the cycle until they are satisfied with it and have confidence in it
+* send the patch for review and CI
+
+This does not mean XCI will completely skip using artifacts. Artifact based
+deployments will be available in later CI loops such as daily/weekly, but the
+developer near loops will be run using source code.
+
+Multi-distro Support
+--------------------
+
+Giving choice and not imposing things on developers and users are two
+of the important aspects of XCI. This means that if they want to have all in one
+deployments, they should be able to do that by using
+:ref:`different flavors <sandbox-flavors>` provided by XCI.
+
+Multi-distro support falls into same category for XCI; giving choice and making
+sure people can pick and choose what Linux distribution they want to use.
+
+XCI currently supports Ubuntu 16.04, CentOS 7, and openSUSE Leap 42.3 and the
+choice is entirely left to the user.
+
+Feature parity between the OPNFV scenarios on different Linux distributions
+that are supported by XCI may vary and it is possible for OPNFV community
+to work on bringing them to the same level.
+
+XCI Pipelines
+=============
+
+Providing timely and tailored feedback is one of the most important things about
+CI. It is important to make the feedback easily accessible and consumable for the
+community so the issues can be analysed as quickly as possible and fixes can be
+issued appropriately.
+
+XCI focuses on feedback aspects of the CI and ensures that whatever feedback provided
+to community makes sense rather than just pointing to some logs. In order to
+achieve this, XCI enhances existing feedback loops and establishes new ones based on
+who needs the feedback. XCI does this by its pipelines as listed below.
+
+Pipelines for Upstream Projects
+-------------------------------
+
+OPNFV work upstream first which means that majority of the work is done in upstream
+projects. The upstream projects OPNFV works with have CI pipelines for the code
+contributed but the pipelines generally lack the testing that is important for
+the OPNFV community.
+
+XCI aims to provide patch level feedback and feedback based on the tip of the master
+branches for the upstream projects. This means that if an OPNFV developer contributes
+to an upstream project, it will be possible for developer to get additional feedback
+from OPNFV XCI in order to ensure the contribution works in OPNFV context
+as well. The level of testing will be adjusted based on the community needs and it
+is important not to duplicate the testing done by the upstream communities in their
+CI pipelines.
+
+Apart from providing feedback to the developers, these pipelines will be used for
+finding working versions of upstream components from their master branches to pin
+them for development purposes.
+
+.. image:: images/pipelines-upstream.png
+   :height: 240px
+   :align: center
+
+Pipelines for OPNFV Scenarios
+-----------------------------
+
+OPNFV CI has pipelines for scenarios constructed by the OPNFV projects. However
+the existing pipelines have number of areas that require improvements.
+
+The existing pipelines lack the granularity one might expect. This means that the
+changes to the scenarios are either not tested properly or tested together with
+unrelated scenarios, resulting in lack of testing or taking too much time to get
+feedback.
+
+Apart from the test coverage and the time it takes to test, scenarios are generally
+tested on a daily basis on baremetal even if it may not be necessary and there are
+no changes.
+
+XCI will change how the feedback is provided for the scenarios by pushing scenario
+ownership to corresponding projects and establishing loops for patchset verification,
+daily and weekly tests. This means that if a scenario changes in a project repo,
+verification for that scenario will directly be triggered and testing will be done using
+virtual deployments, providing feedback to the project.
+
+Daily and weekly loops will be run on baremetal if and only if the scenario is worth
+testing on baremetal. This will be achieved by applying promotion concepts; if a
+scenario passes virtual deployments, it will be tested by daily loops on baremetal
+and by weekly loops later on.
+
+.. image:: images/pipelines-scenarios.png
+   :height: 320px
+   :align: center
+
+Pipelines for OPNFV Test Projects
+---------------------------------
+
+OPNFV Test Projects generally lack the CI coverage they need. Most of the test projects
+only have unit tests, resulting in faults slipping into platform testing, making it
+harder for community to understand what really went wrong; for example, is there a
+problem with the scenario itself or is the problem with the test framework/cases?
+
+XCI aims to establish proper CI pipelines for the test projects by employing
+virtual deployments, so any change that is done to test frameworks/cases
+themselves will be tested against a real virtual deployment. The deployments
+will be brought up using a verified version of the relevant scenario and via snapshots
+so the patch verification will be relevant and quick. If the testing at this level fails,
+it is most probably due to the patch itself rather than the scenario, preventing
+faulty code from slipping into master branch of the test project. Further feedback loops,
+such as post-merge, can be established depending on the needs of the community.
+
+.. image:: images/pipelines-testprojects.png
+   :height: 320px
+   :align: center
+
+Pipelines for XCI Framework and Sandbox
+---------------------------------------
+
+XCI itself needs to be tested properly in order to ensure the changes to the framework
+or the sandbox do not break anything for the community.
+
+Putting All Together
+--------------------
 
+All the pipelines explained in earlier sections run in parallel and as independently
+from each other as possible, providing feedback to relevant communities and people
+so they can get the feedback that fit their purposes.
 
-The development and release model employed by OPNFV uses stable versions of
-upstream components. This helps developers and users who are after the stability
-however it slows down the speed of development, testing, resulting in slower pace
-in innovation.
+Output of these pipelines (verdicts, artifacts, and so on) are also used by each
+of the pipelines appropriately, ensuring that the pipeline uses well tested versions
+of artifacts they need.
 
-In order to provide means for developers to work with OpenStack master
-branch, cutting the time it takes to develop new features significantly and
-testing them on OPNFV Infrastructure
-    enable OPNFV developers to identify bugs earlier, issue fixes faster, and
-get feedback on a daily basis
-    establish mechanisms to run additional testing on OPNFV Infrastructure to
-provide feedback to OpenStack community
-    make the solutions we put in place available to other LF Networking Projects
-OPNFV works with closely
-    embrace the change and apply Continuous Delivery and DevOps principles and
-practices to OPNFV
+An example of this could be the pipelines for OPNFV test projects and the pipelines
+for the OPNFV scenarios. Pipelines for OPNFV projects need verified versions of the
+scenarios to gate changes coming to test project repositories so they can be tested
+in isolation. This means that whatever goes wrong during the gate is probably due to
+change itself and not because of the scenario since the scenario is tested long before
+and promoted to test pipeline to be used for gating.
 
+Similar thing is valid for the OPNFV scenarios as well; the pipelines
+need verified versions of test frameworks/cases so when scenario is put on a baremetal
+and tested, only thing that really changed and possibily of causing a failure
+is the scenario itself.
 
+.. image:: images/pipelines-parallel.png
+   :height: 480px
+   :align: center