Move design_docs into docs/design

Change HTML theme for sphinx
Fix makefile reference to old design_docs dir

Change-Id: I848115c5901db8dc33aa4157bdbc8af430695f31
Signed-off-by: blsaws <blsaws@hotmail.com>

11 files changed, 381 insertions, 116 deletions

diff --git a/docs/design/architecture.rst b/docs/design/architecture.rst
new file mode 100644
index 0000000..9949c1a
--- /dev/null
+++ b/docs/design/architecture.rst
@@ -0,0 +1,62 @@
+Architecture
+============
+
+Architectural Concept
+---------------------
+The following example diagram illustrates a "relationship diagram" type view of an NFVI platform, in which the roles of components focused on policy management, services, and infrastructure are shown. This view illustrates that a large-scale deployment of NFVI may leverage multiple components of the same "type" (e.g. SDN Controller), which fulfill specific purposes for which they are optimized. For example, a global SDN controller and cloud orchestrator can act as directed by a service orchestrator in the provisioning of VNFs per intent, while various components at a local and global level handle policy-related events directly and/or feed them back through a closed-loop policy design that responds as needed, directly or through the service orchestrator.
+ 
+.. image:: ./images/policy_architecture.png
+   :width: 700 px
+   :alt: policy_architecture.png
+   :align: center
+
+(source of the diagram above: https://git.opnfv.org/cgit/copper/plain/design_docs/images/policy_architecture.pptx)
+
+Architectural Aspects
+---------------------
+  * Policies are reflected in two high-level goals
+  
+    * Ensure resource requirements of VNFs and services are applied per VNF designer, service, and tenant intent
+    * Ensure that generic policies are not violated, e.g. *networks connected to VMs must either be public or owned by the VM owner*
+
+  * Policies are distributed through two main means
+  
+    * As part of VNF packages, customized if needed by Service Design tools, expressing intent of the VNF designer and service provider, and possibly customized or supplemented by service orchestrators per the intent of specific tenants
+    * As generic policies provisioned into VIMs (SDN controllers and cloud orchestrators), expressing intent of the service provider re what states/events need to be policy-governed independently of specific VNFs
+
+  * Policies are applied locally and in closed-loop systems per the capabilities of the local policy enforcer and the impact of the related state/event conditions
+
+    * VIMs should be able to execute most policies locally
+    * VIMs may need to pass policy-related state/events to a closed-loop system, where those events are relevant to other components in the architecture (e.g. service orchestrator), or some additional data/arbitration is needed to resolve the state/event condition
+
+  * Policies are localized as they are distributed/delegated
+
+    * High-level policies (e.g. expressing “intent”) can be translated into VNF package elements or generic policies, perhaps using distinct syntaxes
+    * Delegated policy syntaxes are likely VIM-specific, e.g. Datalog (Congress), YANG (ODL-based SDNC), or other schemas specific to other SDNCs (Contrail, ONOS)
+
+  * Closed-loop policy and VNF-lifecycle event handling are //somewhat// distinct
+ 
+    * Closed-loop policy is mostly about resolving conditions that can't be handled locally, but as above in some cases the conditions may be of relevance and either delivered directly or forwarded to service orchestrators
+    * VNF-lifecycle events that can't be handled by the VIM locally are delivered directly to the service orchestrator
+
+  * Some events/analytics need to be collected into a more "open-loop" system which can enable other actions, e.g.
+  
+    * audits and manual interventions
+    * machine-learning focused optimizations of policies (largely a future objective)
+    
+Issues to be investigated as part of establishing an overall cohesive/adaptive policy architecture:
+
+  * For the various components which may fulfill a specific purpose, what capabilities (e.g. APIs) do they have/need to
+  
+    * handle events locally
+    * enable closed-loop policy handling components to subscribe/optimize policy-related events that are of interest
+	
+  * For global controllers and cloud orchestrators
+  
+    * How do they support correlation of events impacting resources in different scopes (network and cloud)
+    * What event/response flows apply to various policy use cases
+	
+  * What specific policy use cases can/should fall into each overall class
+  
+    * locally handled by NFVI components
+    * handled by a closed-loop policy system, either VNF/service-specific or VNF-independent
diff --git a/docs/design/definitions.rst b/docs/design/definitions.rst
new file mode 100644
index 0000000..4423d45
--- /dev/null
+++ b/docs/design/definitions.rst
@@ -0,0 +1,38 @@
+Definitions
+===========
+.. list-table:: Definitions
+   :widths: 15 85
+   :header-rows: 1
+
+   * - Term
+     - Meaning
+
+   * - State
+     - Information that can be used to convey or imply the state of something, e.g. an application, resource, entity, etc. This can include data held inside OPNFV components, "events" that have occurred (e.g. "policy violation"), etc.
+
+   * - Event
+     - An item of significance to the policy engine, for which the engine has become aware through some method of discovery e.g. polling or notification.
+
+Abbreviations
+=============
+.. list-table:: Abbreviations
+   :widths: 15 85
+   :header-rows: 1
+
+   * - Term
+     - Meaning
+
+   * - CRUD
+     - Create, Read, Update, Delete (database operation types)
+
+   * - FCAPS
+     - Fault, Configuration, Accounting, Performance, Security
+
+   * - NF
+     - Network Function
+	 
+   * - SFC
+     - Service Function Chaining
+
+   * - VNF
+     - Virtual Network Function
diff --git a/docs/design/images/policy_architecture.png b/docs/design/images/policy_architecture.png
new file mode 100644
index 0000000..eb37e36
--- /dev/null
+++ b/docs/design/images/policy_architecture.png
diff --git a/docs/design/images/policy_architecture.pptx b/docs/design/images/policy_architecture.pptx
new file mode 100644
index 0000000..5739f0f
--- /dev/null
+++ b/docs/design/images/policy_architecture.pptx
diff --git a/docs/design/index.rst b/docs/design/index.rst
new file mode 100644
index 0000000..1f251a8
--- /dev/null
+++ b/docs/design/index.rst
@@ -0,0 +1,27 @@
+.. Copper documentation master file, created by
+   sphinx-quickstart on Tue Jun  9 19:12:31 2015.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+OPNFV Copper Project
+====================
+
+Contents:
+
+.. toctree::
+   :numbered: 
+   :maxdepth: 4
+   
+   introduction.rst
+   definitions.rst
+   usecases.rst
+   architecture.rst
+   requirements.rst
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+
diff --git a/docs/design/introduction.rst b/docs/design/introduction.rst
new file mode 100644
index 0000000..b9d9722
--- /dev/null
+++ b/docs/design/introduction.rst
@@ -0,0 +1,12 @@
+Introduction
+============
+
+..
+ This work is licensed under a Creative Commons Attribution 3.0 Unported License.
+
+ http://creativecommons.org/licenses/by/3.0/legalcode
+ 
+.. NOTE::
+   This is the working documentation for the Copper project.
+
+The `OPNFV Copper <https://wiki.opnfv.org/copper>`_ project aims to help ensure that virtualized infrastructure deployments comply with goals of the VNF designer/user, e.g. re affinity and partitioning (e.g. per regulation, control/user plane separation, cost…). This is a "requirements" project with initial goal to assess "off the shelf" basic OPNFV platform support for policy management, using existing open source projects such as OpenStack Congress and OpenDaylight Group-Based Policy (GBP). The project will assess what policy-related features are currently supported through research into the related projects in OpenStack and ODL, and testing of integrated vanilla distributions of those and other dependent open source projects in the OPNFV’s NFVI platform scope. 
\ No newline at end of file
diff --git a/docs/design/requirements.rst b/docs/design/requirements.rst
new file mode 100644
index 0000000..ee88b3c
--- /dev/null
+++ b/docs/design/requirements.rst
@@ -0,0 +1,72 @@
+Requirements
+============
+This section outlines general requirements for configuration policies, per the two main aspects in the Copper project scope:
+	* Ensuring resource requirements of VNFs and services are applied per VNF designer, service, and tenant intent
+	* Ensuring that generic policies are not violated, e.g. *networks connected to VMs must either be public or owned by the VM owner*
+	
+Resource Requirements
++++++++++++++++++++++
+Resource requirements describe the characteristics of virtual resources (compute, storage, network) that are needed for VNFs and services, and how those resources should be managed over the lifecycle of a VNF/service. Upstream projects  already include multiple ways in which resource requirements can be expressed and fulfilled, e.g.:
+	* OpenStack Nova
+		* the `image <http://docs.openstack.org/openstack-ops/content/user_facing_images.html>`_ feature, enabling "VM templates" to be defined for NFs, and referenced by name as a specific NF version to be used
+		* the `flavor <http://docs.openstack.org/openstack-ops/content/flavors.html>`_ feature, addressing basic compute and storage requirements, with extensibility for custom attributes
+	* OpenStack Heat
+		* the `Heat Orchestration Template <http://docs.openstack.org/developer/heat/template_guide/index.html>`_ feature, enabling a variety of VM aspects to be defined and managed by Heat throughout the VM lifecycle, notably
+			* alarm handling (requires `Ceilometer <https://wiki.openstack.org/wiki/Ceilometer>`_)
+			* attached volumes (requires `Cinder <https://wiki.openstack.org/wiki/Cinder>`_)
+			* domain name assignment (requires `Designate <https://wiki.openstack.org/wiki/Designate>`_)
+			* images (requires `Glance <https://wiki.openstack.org/wiki/Glance>`_)
+			* autoscaling
+			* software configuration associated with VM "lifecycle hooks (CREATE, UPDATE, SUSPEND, RESUME, DELETE"
+			* wait conditions and signaling for sequencing orchestration steps
+			* orchestration service user management (requires `Keystone <http://docs.openstack.org/developer/keystone/>`_)
+			* shared storage (requires `Manila <https://wiki.openstack.org/wiki/Manila>`_)
+			* load balancing (requires Neutron `LBaaS <http://docs.openstack.org/admin-guide-cloud/content/section_lbaas-overview.html>`_)
+			* firewalls (requires Neutron `FWaaS <http://docs.openstack.org/admin-guide-cloud/content/install_neutron-fwaas-agent.html>`_)
+			* various Neutron-based network and security configuration items
+			* Nova flavors
+			* Nova server attributes including access control
+			* Nova server group affinity and anti-affinity
+			* "Data-intensive application clustering" (requires `Sahara <https://wiki.openstack.org/wiki/Sahara>`_)
+			* DBaaS (requires `Trove <http://docs.openstack.org/developer/trove/>`_)
+			* "multi-tenant cloud messaging and notification service" (requires `Zaqar <http://docs.openstack.org/developer/zaqar/>`_)
+	* OpenStack `Group-Based Policy <https://wiki.openstack.org/wiki/GroupBasedPolicy>`_
+		* API-based grouping of endpoints with associated contractual expectations for data flow processing and service chaining
+	* OpenStack `Tacker <https://wiki.openstack.org/wiki/Tacker>`_
+		* "a fully functional ETSI MANO based general purpose NFV Orchestrator and VNF Manager for OpenStack"
+	* OpenDaylight `Group-Based Policy <https://wiki.opendaylight.org/view/Group_Based_Policy_(GBP)>`_
+		* model-based grouping of endpoints with associated contractual expectations for data flow processing
+	* OpenDaylight `Service Function Chaining (SFC) <https://wiki.opendaylight.org/view/Service_Function_Chaining:Main>`_
+		* model-based management of "service chains" and the infrastucture that enables them
+	* Additional projects that are commonly used for configuration management, implemented as client-server frameworks using model-based, declarative, or scripted configuration management data.
+		* `Puppet <https://puppetlabs.com/puppet/puppet-open-source>`_ 
+		* `Chef <https://www.chef.io/chef/>`_ 
+		* `Ansible <http://docs.ansible.com/ansible/index.html>`_
+		* `Salt <http://saltstack.com/community/>`_ 	
+	
+Generic Policy Requirements
++++++++++++++++++++++++++++	
+Generic policy requirements address conditions related to resource state and events which need to be monitored for, and optionally responded to or prevented. These conditions are typically expected to be VNF/service-independent, as VNF/service-dependent condition handling (e.g. scale in/out) are considered to be addressed by VNFM/NFVO/VIM functions as described under Resource Requirements or as FCAPS related functions. However the general capabilities below can be applied to VNF/service-specific policy handling as well, or in particular to invocation of VNF/service-specific management/orchestration actions. The high-level required capabilities include:
+	* Polled monitoring: Exposure of state via request-response APIs.
+	* Notifications: Exposure of state via pub-sub APIs.
+	* Realtime/near-realtime notifications: Notifications that occur in actual or near realtime.
+	* Delegated policy: CRUD operations on policies that are distributed to specific components for local handling, including one/more of monitoring, violation reporting, and enforcement.
+	* Violation reporting: Reporting of conditions that represent a policy violation.
+	* Reactive enforcement: Enforcement actions taken in response to policy violation events.
+	* Proactive enforcement: Enforcement actions taken in advance of policy violation events, e.g. blocking actions that could result in a policy violation.
+	* Compliance auditing: Periodic auditing of state against policies.
+ 
+ Upstream projects already include multiple ways in which configuration conditions can be monitored and responded to:
+	* OpenStack `Congress <https://wiki.openstack.org/wiki/Congress>`_ provides a table-based mechanism for state monitoring and proactive/reactive policy enforcement, including (as of the Kilo release) data obtained from internal databases of Nova, Neutron, Ceilometer, Cinder, Glance, Keystone, and Swift. The Congress design approach is also extensible to other VIMs (e.g. SDNCs) through development of data source drivers for the new monitored state information. See `Stackforge Congress Data Source Translators <https://github.com/stackforge/congress/tree/master/congress/datasources>`_, `congress.readthedocs.org <http://congress.readthedocs.org/en/latest/cloudservices.html#drivers>`_, and the `Congress specs <https://github.com/stackforge/congress-specs>`_ for more info.
+	* OpenStack `Ceilometer <https://wiki.openstack.org/wiki/Ceilometer>`_ provides means to trigger alarms upon a wide variety of conditions derived from its monitored OpenStack analytics.
+	* `Nagios <https://www.nagios.org/#/>`_ "offers complete monitoring and alerting for servers, switches, applications, and services".
+	
+Requirements Validation Approach
+++++++++++++++++++++++++++++++++
+The Copper project will assess the completeness of the upstream project solutions for requirements in scope though a process of:
+	* developing configuration policy use cases to focus solution assessment tests
+	* integrating the projects into the OPNFV platform for testing
+	* executing functional and performance tests for the solutions
+	* assessing overall requirements coverage and gaps in the most complete upstream solutions
+	
+Depending upon the priority of discovered gaps, new requirements will be submitted to upstream projects for the next available release cycle.
\ No newline at end of file
diff --git a/docs/design/usecases.rst b/docs/design/usecases.rst
new file mode 100644
index 0000000..046cfe2
--- /dev/null
+++ b/docs/design/usecases.rst
@@ -0,0 +1,169 @@
+Use Cases
+=========
+
+Resource Requirements
++++++++++++++++++++++
+
+Workload Placement
+------------------
+
+Affinity
+........
+
+Ensures that the VM instance is launched "with affinity to" specific resources, e.g. within a compute or storage cluster. This is analogous to the affinity rules in `VMWare vSphere DRS <https://pubs.vmware.com/vsphere-50/topic/com.vmware.vsphere.resmgmt.doc_50/GUID-FF28F29C-8B67-4EFF-A2EF-63B3537E6934.html>`_. Examples include: "Same Host Filter", i.e. place on the same compute node as a given set of instances, e.g. as defined in a scheduler hint list.
+
+As implemented by OpenStack Heat using server groups:
+
+*Note: untested example...*
+
+.. code::
+
+	resources: 
+	  servgrp1: 
+		type: OS::Nova::ServerGroup
+		properties: 
+		  policies: 
+		  - affinity
+		  serv1:
+			type: OS::Nova::Server
+			properties:
+				image: { get_param: image }
+				flavor: { get_param: flavor }
+				networks:
+					- network: {get_param: network}
+		  serv2:
+			type: OS::Nova::Server
+			properties:
+				image: { get_param: image }
+				flavor: { get_param: flavor }
+				networks:
+					- network: {get_param: network}
+
+Anti-Affinity
+.............
+
+Ensures that the VM instance is launched "with anti-affinity to" specific resources, e.g. outside a compute or storage cluster, or geographic location. This filter is analogous to the anti-affinity rules in vSphere DRS. Examples include: "Different Host Filter", i.e. ensures that the VM instance is launched on a different compute node from a given set of instances, as defined in a scheduler hint list.
+
+As implemented by OpenStack Heat using scheduler hints:
+
+*Note: untested example...*
+
+.. code::
+
+	heat template version: 2013-05-23
+	parameters:
+	  image:
+		type: string
+		default: TestVM
+	  flavor:
+		type: string
+		default: m1.micro
+	  network:
+		type: string
+		default: cirros_net2
+	resources:
+	  serv1:
+		type: OS::Nova::Server
+		properties:
+			image: { get_param: image }
+			flavor: { get_param: flavor }
+			networks:
+				- network: {get_param: network}
+			scheduler_hints: {different_host: {get_resource: serv2}}
+	  serv2:
+		type: OS::Nova::Server
+		properties:
+			image: { get_param: image }
+			flavor: { get_param: flavor }
+			networks:
+				- network: {get_param: network}
+			scheduler_hints: {different_host: {get_resource: serv1}}
+
+DMZ Deployment
+..............
+As a service provider, I need to ensure that applications which have not been designed for exposure in a DMZ zone, are not attached to DMZ networks.
+
+Configuration Auditing
+----------------------
+
+As a service provider or tenant, I need to periodically verify that resource configuration requirements have not been violated, as a backup means to proactive or reactive policy enforcement.
+
+Generic Policy Requirements
++++++++++++++++++++++++++++
+
+NFVI Self-Service Constraints
+-----------------------------
+
+As an NFVI provider, I need to ensure that my self-service tenants are not able to configure their VNFs in ways that would impact other tenants or the reliability, security, etc of the NFVI.
+
+Network Access Control
+......................
+
+Networks connected to VMs must be public, or owned by someone in the VM owner's group.
+
+This use case captures the intent of the following sub-use-cases:
+
+  * Link Mirroring: As a troubleshooter, I need to mirror traffic from physical or virtual network ports so that I can investigate trouble reports.
+  * Link Mirroring: As a NFVaaS tenant, I need to be able to mirror traffic on my virtual network ports so that I can investigate trouble reports.
+  * Unauthorized Link Mirroring Prevention: As a NFVaaS tenant, I need to be able to prevent other tenants from mirroring traffic on my virtual network ports so that I can protect the privacy of my service users.
+  * Link Mirroring Delegation: As a NFVaaS tenant, I need to be able to allow my NFVaaS SP customer support to mirror traffic on my virtual network ports so that they can assist in investigating trouble reports.
+
+As implemented through OpenStack Congress: 
+
+.. code:: 
+
+   error :- 
+   nova:vm(vm), 
+   neutron:network(network), 
+   nova:network(vm, network), 
+   neutron:private(network), 
+   nova:owner(vm, vm-own), 
+   neutron:owner(network, net-own), 
+   -same-group(vm-own, net-own)
+		 
+   same-group(user1, user2) :- 
+   ldap:group(user1, g), 
+   ldap:group(user2, g)
+   
+
+Storage Access Control
+......................
+
+Storage resources connected to VMs must be owned by someone in the VM owner's group.
+
+As implemented through OpenStack Congress: 
+
+.. code:: 
+
+  error :- 
+  nova:vm(vm), 
+  cinder:volumes(volume), 
+  nova:volume(vm, volume), 
+  nova:owner(vm, vm-own), 
+  neutron:owner(volume, vol-own), 
+  -same-group(vm-own, vol-own)
+ 
+  same-group(user1, user2) :- 
+  ldap:group(user1, g), 
+  ldap:group(user2, g)
+
+Resource Reclamation
+--------------------
+
+As a service provider or tenant, I need to be informed of VMs that are under-utilized so that I can reclaim the VI resources. (example from `RuleYourCloud blog <http://ruleyourcloud.com/2015/03/12/scaling-up-congress.html>`_) 
+
+As implemented through OpenStack Congress: 
+
+*Note: untested example...*
+
+.. code:: 
+
+  reclaim_server(vm) :-
+  ceilometer:stats("cpu_util",vm, avg_cpu),
+  lessthan(avg_cpu, 1)
+
+  error(user_id, email, vm_name) :-
+  reclaim_server(vm),
+  nova:servers(vm, vm_name, user_id),
+  keystone:users(user_id, email)
+
diff --git a/docs/etc/conf.py b/docs/etc/conf.py
index 0066035..cff2d96 100644
--- a/docs/etc/conf.py
+++ b/docs/etc/conf.py
@@ -17,6 +17,7 @@ source_suffix = '.rst'
 master_doc = 'index'
 pygments_style = 'sphinx'
 html_use_index = False
+html_theme = 'sphinx_rtd_theme'
 
 pdf_documents = [('index', u'OPNFV', u'OPNFV Project', u'OPNFV')]
 pdf_fit_mode = "shrink"
diff --git a/docs/how-to-use-docs/documentation-example.rst b/docs/how-to-use-docs/documentation-example.rst
deleted file mode 100644
index afcf758..0000000
--- a/docs/how-to-use-docs/documentation-example.rst
+++ /dev/null
@@ -1,86 +0,0 @@
-.. two dots create a comment. please leave this logo at the top of each of your rst files.
-.. image:: ../etc/opnfv-logo.png 
-  :height: 40
-  :width: 200
-  :alt: OPNFV
-  :align: left
-.. these two pipes are to seperate the logo from the first title
-|
-|
-How to create documentation for your OPNFV project
-==================================================
-
-this is the directory structure of the docs/ directory that can be found in the root of your project directory
-
-.. code-block:: bash
-
-    ./etc
-    ./etc/opnfv-logo.png
-    ./etc/conf.py
-    ./how-to-use-docs
-    ./how-to-use-docs/documentation-example.rst
-    ./how-to-use-docs/index.rst
-
-To create your own documentation, Create any number of directories (depending on your need) and place in each of them an index.rst.
-This index file must refence your other rst files.
-
-* Here is an example index.rst
-
-.. code-block:: bash
-
-  Example Documentation table of contents
-  =======================================
-
-  Contents:
-
-  .. toctree::
-     :numbered:
-     :maxdepth: 4
-
-     documentation-example.rst
-
-  Indices and tables
-  ==================
-
-  * :ref:`search`
-
-  Revision: _sha1_
-
-  Build date: |today|
-
-
-The Sphinx Build
-================
-
-When you push documentation changes to gerrit a jenkins job will create html documentation.
-
-* Verify Jobs
-For verify jobs a link to the documentation will show up as a comment in gerrit for you to see the result.
-
-* Merge jobs
-
-Once you are happy with the look of your documentation you can submit the patchset the merge job will 
-copy the output of each documentation directory to http://artifacts.opnfv.org/$project/docs/$name_of_your_folder/index.html
-
-Here are some quick examples of how to use rst markup
-
-This is a headline::
-
-  here is some code, note that it is indented
-
-links are easy to add: Here is a link to sphinx, the tool that we are using to generate documetation http://sphinx-doc.org/
-
-* Bulleted Items
-
-  **this will be bold**
-
-.. code-block:: bash
-
-  echo "Heres is a code block with bash syntax highlighting"
-
-
-Leave these at the bottom of each of your documents they are used internally
-
-Revision: _sha1_
-
-Build date: |today|
diff --git a/docs/how-to-use-docs/index.rst b/docs/how-to-use-docs/index.rst
deleted file mode 100644
index 36710b3..0000000
--- a/docs/how-to-use-docs/index.rst
+++ /dev/null
@@ -1,30 +0,0 @@
-.. OPNFV Release Engineering documentation, created by
-   sphinx-quickstart on Tue Jun  9 19:12:31 2015.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
-
-.. image:: ../etc/opnfv-logo.png
-  :height: 40
-  :width: 200
-  :alt: OPNFV
-  :align: left
-
-Example Documentation table of contents
-=======================================
-
-Contents:
-
-.. toctree::
-   :numbered:
-   :maxdepth: 4
-
-   documentation-example.rst
-
-Indices and tables
-==================
-
-* :ref:`search`
-
-Revision: _sha1_
-
-Build date: |today|