Add kingbird installation, configuration and user guide

Kingbird is a sub-project in multisite, and will be released
as part of OPNFV Colorado release, add Kingbird installation
, configuration and user guide to the Multisite repository, and
in later patch includes them into OPNFV documentation.

Change-Id: I3f8f3528c495f6f10bfa790763cc7c69017d7bdb
Signed-off-by: joehuang <joehuang@huawei.com>

10 files changed, 828 insertions, 21 deletions

diff --git a/docs/configurationguide/configuration.options.render.rst b/docs/configurationguide/configuration.options.render.rst
new file mode 100644
index 0000000..f1dc11a
--- /dev/null
+++ b/docs/configurationguide/configuration.options.render.rst
@@ -0,0 +1,27 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) Christopher Price (Ericsson AB)
+
+=====================
+Configuration Options
+=====================
+
+OPNFV provides a variety of virtual infrastructure deployments called scenarios designed to
+host Virtualised Network Functions(VNFs). Each scenario provide specific capabilities and/or
+components aimed to solve specific problems for the deployment of VNF's. A scenario may include
+components such as OpenStack, OpenDaylight, OVS, KVM etc. where each scenario will include
+different source components or configurations.
+
+OPNFV Scenarios
+===============
+
+Each OPNFV scenario provides unique features and capabilities, it is important to understand
+your target platform capabilities before installing and configuring your target scenario.
+This configuration guide outlines how to install and configure components in order to enable
+the features you require.
+
+.. include:: ../scenario/scenariomatrix.rst
+
+This document will describe how to install and configure your target OPNFV scenarios.
+Remember to check the associated validation procedures section following your installation for
+details of the use cases and tests that have been run.
diff --git a/docs/configguide/index.rst b/docs/configurationguide/index.rst
index 81c5d15..791d94d 100644
--- a/docs/configguide/index.rst
+++ b/docs/configurationguide/index.rst
@@ -9,4 +9,4 @@ Multisite Configuration Guide
    :numbered:
    :maxdepth: 2
 
-   multisite-configuration-guide.rst
+   multisite.kingbird.configuration.description.rst
diff --git a/docs/configguide/multisite-configuration-guide.rst b/docs/configurationguide/multisite-configuration-guide.rst
index 8e0c190..c005e8d 100644
--- a/docs/configguide/multisite-configuration-guide.rst
+++ b/docs/configurationguide/multisite-configuration-guide.rst
@@ -11,7 +11,7 @@ Multisite identity service management
 Goal
 ----
 
-a user should, using a single authentication point be able to manage virtual
+A user should, using a single authentication point be able to manage virtual
 resources spread over multiple OpenStack regions.
 
 Before you read
diff --git a/docs/configurationguide/multisite.kingbird.configuration.description.rst b/docs/configurationguide/multisite.kingbird.configuration.description.rst
new file mode 100644
index 0000000..d003019
--- /dev/null
+++ b/docs/configurationguide/multisite.kingbird.configuration.description.rst
@@ -0,0 +1,265 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV
+
+
+Configuration of Multisite.Kingbird
+===================================
+
+A brief introduction to configure Multisite Kingbird service. Only the
+configuration items for Kingbird will be described here. Logging,
+messaging, database, keystonemiddleware etc configuration which are
+generated from OpenStack OSLO libary, will not be described here, for
+these configuration items are common to Nova, Cinder, Neutron. So please
+refer to corresponding description from Nova or Cinder or Neutron.
+
+
+Configuration in [DEFAULT]
+--------------------------
+
+configuration items for kingbird-api
+""""""""""""""""""""""""""""""""""""
+
+bind_host
+*********
+- default value: *bind_host = 0.0.0.0*
+- description: The host IP to bind for kingbird-api service
+
+bind_port
+*********
+- default value: *bind_port = 8118*
+- description: The port to bind for kingbird-api service
+
+api_workers
+***********
+- default value: *api_workers = 2*
+- description: Number of kingbird-api workers
+
+configuration items for kingbird-engine
+"""""""""""""""""""""""""""""""""""""""
+
+host
+****
+- default value: *host = localhost*
+- description: The host name kingbird-engine service is running on
+
+workers
+*******
+- default value: *workers = 1*
+- description: Number of kingbird-engine workers
+
+report_interval
+***************
+- default value: *report_interval = 60*
+- description: Seconds between running periodic reporting tasks to
+  keep the engine alive in the DB. If the engine doesn't report its
+  aliveness to the DB more than two intervals, then the lock accquired
+  by the engine will be removed by other engines.
+
+common configuration items for kingbird-api and kingbird-engine
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+use_default_quota_class
+***********************
+- default value: *use_default_quota_class = true*
+- description: Enables or disables use of default quota class with default
+  quota, boolean value
+
+Configuration in [kingbird_global_limit]
+----------------------------------------
+
+For quota limit, a negative value means unlimited.
+
+configuration items for kingbird-api and kingbird-engine
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+quota_instances
+***************
+- default value: *quota_instances = 10*
+- description: Number of instances allowed per project, integer value.
+
+quota_cores
+***********
+- default value: *quota_cores = 20*
+- description: Number of instance cores allowed per project, integer value.
+
+quota_ram
+*********
+- default value: *quota_ram = 512*
+- description: Megabytes of instance RAM allowed per project, integer value.
+
+quota_metadata_items
+********************
+- default value: *quota_metadata_items = 128*
+- description: Number of metadata items allowed per instance, integer value.
+
+quota_key_pairs
+***************
+- default value: *quota_key_pairs = 10*
+- description: Number of key pairs per user, integer value.
+
+quota_fixed_ips
+***************
+- default value: *quota_fixed_ips = -1*
+- description: Number of fixed IPs allowed per project, this should be at
+  least the number of instances allowed, integer value.
+
+quota_security_groups
+*********************
+- default value: *quota_security_groups = 10*
+- description: Number of security groups per project, integer value.
+
+quota_floating_ips
+******************
+- default value: *quota_floating_ips = 10*
+- description: Number of floating IPs allowed per project, integer value.
+
+quota_network
+***************
+- default value: *quota_network = 10*
+- description: Number of networks allowed per project, integer value.
+
+quota_subnet
+***************
+- default value: *quota_subnet = 10*
+- description: Number of subnets allowed per project, integer value.
+
+quota_port
+***************
+- default value: *quota_port = 50*
+- description: Number of ports allowed per project, integer value.
+
+quota_security_group
+********************
+- default value: *quota_security_group = 10*
+- description: Number of security groups allowed per project, integer value.
+
+quota_security_group_rule
+*************************
+- default value: *quota_security_group_rule = 100*
+- description: Number of security group rules allowed per project, integer
+  value.
+
+quota_router
+************
+- default value: *quota_router = 10*
+- description: Number of routers allowed per project, integer value.
+
+quota_floatingip
+****************
+- default value: *quota_floatingip = 50*
+- description: Number of floating IPs allowed per project, integer value.
+
+quota_volumes
+***************
+- default value: *quota_volumes = 10*
+- description: Number of volumes allowed per project, integer value.
+
+quota_snapshots
+***************
+- default value: *quota_snapshots = 10*
+- description: Number of snapshots allowed per project, integer value.
+
+quota_gigabytes
+***************
+- default value: *quota_gigabytes = 1000*
+- description: Total amount of storage, in gigabytes, allowed for volumes
+  and snapshots per project, integer value.
+
+quota_backups
+*************
+- default value: *quota_backups = 10*
+- description: Number of volume backups allowed per project, integer value.
+
+quota_backup_gigabytes
+**********************
+- default value: *quota_backup_gigabytes = 1000*
+- description: Total amount of storage, in gigabytes, allowed for volume
+  backups per project, integer value.
+
+Configuration in [cache]
+----------------------------------------
+
+The [cache] section is used by kingbird engine to access the quota
+information for Nova, Cinder, Neutron in each region in order to reduce
+the KeyStone load while retrieving the endpoint information each time.
+
+configuration items for kingbird-engine
+"""""""""""""""""""""""""""""""""""""""
+
+auth_uri
+***************
+- default value:
+- description: Keystone authorization url, for example, http://127.0.0.1:5000/v3.
+
+admin_username
+**************
+- default value:
+- description: Username of admin account, for example, admin.
+
+admin_password
+**************
+- default value:
+- description: Password for admin account, for example, password.
+
+admin_tenant
+************
+- default value:
+- description: Tenant name of admin account, for example, admin.
+
+admin_user_domain_name
+**********************
+- default value: *admin_user_domain_name = Default*
+- description: User domain name of admin account.
+
+admin_project_domain_name
+*************************
+- default value: *admin_project_domain_name = Default*
+- description: Project domain name of admin account.
+
+Configuration in [scheduler]
+----------------------------------------
+
+The [scheduler] section is used by kingbird engine to periodically synchronize
+and rebalance the quota for each project.
+
+configuration items for kingbird-engine
+"""""""""""""""""""""""""""""""""""""""
+
+periodic_enable
+***************
+- default value: *periodic_enable = True*
+- description: Boolean value for enable/disable periodic tasks.
+
+periodic_interval
+*****************
+- default value: *periodic_interval = 900*
+- description: Periodic time interval for automatic quota sync job, unit is
+  seconds.
+
+Configuration in [batch]
+----------------------------------------
+
+The [batch] section is used by kingbird engine to periodicly synchronize
+and rebalance the quota for each project.
+
+batch_size
+***************
+- default value: *batch_size = 3*
+- description: Batch size number of projects will be synced at a time.
+
+Configuration in [locks]
+----------------------------------------
+
+The [locks] section is used by kingbird engine to periodically synchronize
+and rebalance the quota for each project.
+
+lock_retry_times
+****************
+- default value: *lock_retry_times = 3*
+- description: Number of times trying to grab a lock.
+
+lock_retry_interval
+*******************
+- default value: *lock_retry_interval =10*
+- description: Number of seconds between lock retries.
diff --git a/docs/installationprocedure/abstract.rst b/docs/installationprocedure/abstract.rst
new file mode 100644
index 0000000..550a809
--- /dev/null
+++ b/docs/installationprocedure/abstract.rst
@@ -0,0 +1,7 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) <optionally add copywriters name>
+
+This document will give the user instructions on how to deploy
+available scenarios verified for the Colorado release of OPNFV
+platform.
diff --git a/docs/installationprocedure/index.rst b/docs/installationprocedure/index.rst
new file mode 100644
index 0000000..bc51b95
--- /dev/null
+++ b/docs/installationprocedure/index.rst
@@ -0,0 +1,18 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) Sofia Wallin Ericsson AB
+
+**********************
+Installation procedure
+**********************
+Colorado 1.0
+------------
+
+.. toctree::
+   :numbered:
+   :maxdepth: 2
+
+   abstract.rst
+   multisite.kingbird.installation.instruction.rst
+   scenario.release.notes.rst
+
diff --git a/docs/installationprocedure/multisite.kingbird.installation.instruction.rst b/docs/installationprocedure/multisite.kingbird.installation.instruction.rst
new file mode 100644
index 0000000..8ef3f06
--- /dev/null
+++ b/docs/installationprocedure/multisite.kingbird.installation.instruction.rst
@@ -0,0 +1,299 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV
+
+===========================================
+Multisite.Kingbird installation instruction
+===========================================
+
+Preparing the installation
+--------------------------
+Kingbird is centralized synchronization service for multi-region OpenStack
+deployments. In OPNFV Colorado release, Kingbird provides centralized quota
+management feature. At least two OpenStack regions with shared KeyStone should
+be installed first.
+
+Kingbird includes kingbird-api and kingbird-engine, kingbird-api and
+kingbird-engine which talk to each other through message bus, and both
+services access the database. Kingbird-api receives the RESTful
+API request for quota management and forward the request to kingbird-engine
+to do quota synchronization etc task.
+
+Therefore install Kingbird on the controller nodes of one of the OpenStack
+region, these two services could be deployed in same node or different node.
+Both kingbird-api and kingbird-engine can run in multiple nodes with
+multi-workers mode. It's up to you how many nodes you want to deploy
+kingbird-api and kingbird-engine and they can work in same node or
+different nodes.
+
+HW requirements
+---------------
+No special hardware requirements
+
+Installation instruction
+------------------------
+
+In colorado release, Kingbird is recommended to be installed in a python
+virtual environment. So install and activate virtualenv first.
+
+.. code-block:: bash
+
+    sudo pip install virtualenv
+    virtualenv venv
+    source venv/bin/activate
+
+Get the latest code of Kingbird from git repository:
+
+.. code-block:: bash
+
+    git clone https://github.com/openstack/kingbird.git
+    cd kingbird/
+    pip install -e .
+
+
+or get the stable release from PyPI repository:
+
+.. code-block:: bash
+
+    pip install kingbird
+
+In case of the database package are not installed, you may need to install:
+
+.. code-block:: bash
+
+    pip install mysql
+    pip install pymysql
+
+In the Kingbird root folder, where you can find the source code of Kingbird,
+generate the configuration sample file for Kingbird:
+
+.. code-block:: bash
+
+    oslo-config-generator --config-file=./tools/config-generator.conf
+
+prepare the folder used for cache, log and configuration for Kingbird:
+
+.. code-block:: bash
+
+    sudo rm -rf /var/cache/kingbird
+    sudo mkdir -p /var/cache/kingbird
+    sudo chown `whoami` /var/cache/kingbird
+    sudo rm -rf /var/log/kingbird
+    sudo mkdir -p /var/log/kingbird
+    sudo chown `whoami` /var/log/kingbird
+    sudo rm -rf /etc/kingbird
+    sudo mkdir -p /etc/kingbird
+    sudo chown `whoami` /etc/kingbird
+
+Copy the sample configuration to the configuration folder /etc/kingbird:
+
+.. code-block:: bash
+
+    cp etc/kingbird/kingbird.conf.sample /etc/kingbird/kingbird.conf
+
+Before editing the configuration file, prepare the database info for Kingbird.
+
+.. code-block:: bash
+
+    mysql -uroot -e "CREATE DATABASE $kb_db CHARACTER SET utf8;"
+    mysql -uroot -e "GRANT ALL PRIVILEGES ON $kb_db.* TO '$kb_db_user'@'%' IDENTIFIED BY '$kb_db_pwd';"
+
+For example, the following command will create database "kingbird", and grant the
+privilege for the db user "kingbird" with password "password":
+
+.. code-block:: bash
+
+    mysql -uroot -e "CREATE DATABASE kingbird CHARACTER SET utf8;"
+    mysql -uroot -e "GRANT ALL PRIVILEGES ON kingbird.* TO 'kingbird'@'%' IDENTIFIED BY 'password';"
+
+Create the service user in OpenStack:
+
+.. code-block:: bash
+
+    source openrc admin admin
+    openstack user create --project=service --password=$kb_svc_pwd $kb_svc_user
+    openstack role add --user=$kb_svc_user --project=service admin
+
+For example, the following command will create service user "kingbird",
+and grant the user "kingbird" with password "password" the role of admin
+in service project:
+
+.. code-block:: bash
+
+    source openrc admin admin
+    openstack user create --project=service --password=password kingbird
+    openstack role add --user=kingbird --project=service admin
+
+
+
+Then edit the configuration file for Kingbird:
+
+.. code-block:: bash
+
+    vim /etc/kingbird/kingbird.conf
+
+By default, the bind_host of kingbird-api is local_host(127.0.0.1), and the
+port for the service is 8118, you can leave it as the default if no port
+conflict happened.
+
+To make the Kingbird work normally, you have to edit these configuration
+items. The [cache] section is used by kingbird engine to access the quota
+information of Nova, Cinder, Neutron in each region, replace the
+auth_uri to the keystone service in your environment,
+especially if the keystone service is not located in the same node, and
+also for the account to access the Nova, Cinder, Neutron in each region,
+in the following configuration, user "admin" with password "password" of
+the tenant "admin" is configured to access other Nova, Cinder, Neutron in
+each region:
+
+.. code-block:: bash
+
+    [cache]
+    auth_uri = http://127.0.0.1:5000/v3
+    admin_tenant = admin
+    admin_password = password
+    admin_username = admin
+
+Configure the database section with the service user "kingbird" and its
+password, to access database "kingbird". For detailed database section
+configuration, please refer to http://docs.openstack.org/developer/oslo.db/opts.html,
+and change the following configuration accordingly based on your
+environment.
+
+.. code-block:: bash
+
+    [database]
+    connection = mysql+pymysql://$kb_db_user:$kb_db_pwd@127.0.0.1/$kb_db?charset=utf8
+
+For example, if the database is "kingbird", and the db user "kingbird" with
+password "password", then the configuration is as following:
+
+.. code-block:: bash
+
+    [database]
+    connection = mysql+pymysql://kingbird:password@127.0.0.1/kingbird?charset=utf8
+
+The [keystone_authtoken] section is used by keystonemiddleware for token
+validation during the API request to the kingbird-api, please refer to
+http://docs.openstack.org/developer/keystonemiddleware/middlewarearchitecture.html
+on how to configure the keystone_authtoken section for the keystonemiddleware
+in detail, and change the following configuration accordingly based on your
+environment:
+
+*please specify the region_name where you want the token will be validated if the
+KeyStone is deployed in multiple regions*
+
+.. code-block:: bash
+
+    [keystone_authtoken]
+    signing_dir = /var/cache/kingbird
+    cafile = /opt/stack/data/ca-bundle.pem
+    auth_uri = http://127.0.0.1:5000/v3
+    project_domain_name = Default
+    project_name = service
+    user_domain_name = Default
+    password = $kb_svc_pwd
+    username = $kb_svc_user
+    auth_url = http://127.0.0.1:35357/v3
+    auth_type = password
+    region_name = RegionOne
+
+For example, if the service user is "kingbird, and the password for the user
+is "password", then the configuration will look like this:
+
+.. code-block:: bash
+
+    [keystone_authtoken]
+    signing_dir = /var/cache/kingbird
+    cafile = /opt/stack/data/ca-bundle.pem
+    auth_uri = http://127.0.0.1:5000/v3
+    project_domain_name = Default
+    project_name = service
+    user_domain_name = Default
+    password = password
+    username = kingbird
+    auth_url = http://127.0.0.1:35357/v3
+    auth_type = password
+    region_name = RegionOne
+
+
+And also configure the message bus connection, you can refer to the message
+bus configuration in Nova, Cinder, Neutron configuration file.
+
+.. code-block:: bash
+
+    [DEFAULT]
+    rpc_backend = rabbit
+    control_exchange = openstack
+    transport_url = None
+
+    [oslo_messaging_rabbit]
+    rabbit_host = 127.0.0.1
+    rabbit_port = 5671
+    rabbit_userid = guest
+    rabbit_password = guest
+    rabbit_virtual_host = /
+
+After these basic configuration items configured, now the database schema of
+"kingbird" should be created:
+
+.. code-block:: bash
+
+    python kingbird/cmd/manage.py --config-file=/etc/kingbird/kingbird.conf db_sync
+
+And create the service and endpoint for Kingbird, please change the endpoint url
+according to your cloud planning:
+
+.. code-block:: bash
+
+    openstack service create --name=kingbird synchronization
+    openstack endpoint create --region=RegionOne \
+    --publicurl=http://127.0.0.1:8118/v1.0 \
+    --adminurl=http://127.0.0.1:8118/v1.0 \
+    --internalurl=http://127.0.0.1:8118/v1.0 kingbird
+
+Now it's ready to run kingbird-api and kingbird-engine:
+
+.. code-block:: bash
+
+    nohup python kingbird/cmd/api.py --config-file=/etc/kingbird/kingbird.conf &
+    nohup python kingbird/cmd/engine.py --config-file=/etc/kingbird/kingbird.conf &
+
+Run the following command to check whether kingbird-api and kingbird-engine
+are running:
+
+.. code-block:: bash
+
+    ps aux|grep python
+
+
+Post-installation activities
+----------------------------
+
+Run the following commands to check whether kingbird-api is serving, please
+replace $token to the token you get from "openstack token issue":
+
+.. code-block:: bash
+
+    openstack token issue
+    curl  -H "Content-Type: application/json"  -H "X-Auth-Token: $token" \
+    http://127.0.0.1:8118/
+
+If the response looks like following: {"versions": [{"status": "CURRENT",
+"updated": "2016-03-07", "id": "v1.0", "links": [{"href":
+"http://127.0.0.1:8118/v1.0/", "rel": "self"}]}]},
+then that means the kingbird-api is working normally.
+
+Run the following commands to check whether kingbird-engine is serving, please
+replace $token to the token you get from "openstack token issue", and the
+$admin_project_id to the admin project id in your environment:
+
+.. code-block:: bash
+
+    curl  -H "Content-Type: application/json"  -H "X-Auth-Token: $token" \
+    -H  "X_ROLE: admin"  -X PUT \
+    http://127.0.0.1:8118/v1.0/$admin_project_id/os-quota-sets/$admin_project_id/sync
+
+If the response looks like following: "triggered quota sync for
+0320065092b14f388af54c5bd18ab5da", then that means the kingbird-engine
+is working normally.
diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst
index 3eeedcb..1429b33 100644
--- a/docs/userguide/index.rst
+++ b/docs/userguide/index.rst
@@ -10,3 +10,4 @@ Multisite Admin User Guide
    :maxdepth: 4
 
    multisite-admin-user-guide.rst
+   multisite.kingbird.user.guide.rst
diff --git a/docs/userguide/multisite-admin-user-guide.rst b/docs/userguide/multisite-admin-user-guide.rst
index a0e9b58..41f23c0 100644
--- a/docs/userguide/multisite-admin-user-guide.rst
+++ b/docs/userguide/multisite-admin-user-guide.rst
@@ -211,6 +211,7 @@ Cons:
   * Need to be aware of the chanllenge of key distribution and rotation
     for Fernet token.
 
+Note: PKI token will be deprecated soon, so Fernet token is encouraged.
 
 Multisite VNF Geo site disaster recovery
 ========================================
@@ -262,14 +263,10 @@ The disater recovery process will work like this:
   2) DR software boot VMs from bootable volumes from the remote Cinder in
      the backup site and attach the regarding data volumes.
 
-Note: It’s up to the DR policy and VNF character how to use the API. Some
-VNF may allow the standby of the VNF or member of the cluster to do
-quiece/unquiece to avoid interfering the service provided by the VNF. Some
-other VNF may afford short unavailable for DR purpose.
-
-This option provides application level consistency disaster recovery.
-This feature is WIP in OpenStack Mitaka release, and will be avaialle in next
-OPNFV release.
+Note: Quiesce/Unquiesce spec was approved in Mitaka, but code not get merged in
+time, https://blueprints.launchpad.net/nova/+spec/expose-quiesce-unquiesce-api
+The spec was rejected in Newton when it was reproposed:
+https://review.openstack.org/#/c/295595/. So this option will not work any more.
 
 Option2, Vitrual Machine Snapshot
 ---------------------------------
@@ -321,8 +318,7 @@ Cons:
   * "Standard" support in Openstack for Disaster Recovery currently fairly
     limited, though active work in this area.
 
-This feature is in discussion in OpenStack Mitaka release, and hopefully will
-be avaialle in next OPNFV release.
+Note: Volume replication v2.1 support project level replication.
 
 
 VNF high availability across VIM
@@ -373,21 +369,22 @@ plane. There are some interesting/hard requirements on the networking (L2/L3)
 between OpenStack instances, at lease the backup plane across different
 OpenStack instances:
 
-1) Overlay L2 networking or shared L2 provider networks as the backup plane
-   for heartbeat or state replication. Overlay L2 network is preferred, the
-   reason is:
+1) Overlay L2 networking is prefered as the backup plane for heartbeat or state
+   replication, the reason is:
 
    a) Support legacy compatibility: Some telecom app with built-in internal L2
    network, for easy to move these app to virtualized telecom application, it
    would be better to provide L2 network.
 
    b) Support IP overlapping: multiple telecom applications may have
-      overlapping IP address for cross OpenStack instance networking Therefore,
-      over L2 networking across Neutron feature is required in OpenStack.
+   overlapping IP address for cross OpenStack instance networking.
+   Therefore over L2 networking across Neutron feature is required
+   in OpenStack.
 
 2) L3 networking cross OpenStack instance for heartbeat or state replication.
-   For L3 networking, we can leverage the floating IP provided in current
-   Neutron, so no new feature requirement to OpenStack.
+   Can leverage FIP or vRouter inter-connected with overlay L2 network to
+   establish overlay L3 networking.
 
-Overlay L2 networking across OpenStack instances is in discussion with Neutron
-community.
+Note: L2 border gateway spec was merged in L2GW project:
+https://review.openstack.org/#/c/270786/. Code will be availabe in later
+release.
diff --git a/docs/userguide/multisite.kingbird.user.guide.rst b/docs/userguide/multisite.kingbird.user.guide.rst
new file mode 100644
index 0000000..6ae3881
--- /dev/null
+++ b/docs/userguide/multisite.kingbird.user.guide.rst
@@ -0,0 +1,193 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV
+
+=============================
+Multisite.Kingbird user guide
+=============================
+
+Quota management for OpenStack multi-region deployments
+-------------------------------------------------------
+Kingbird is centralized synchronization service for multi-region OpenStack
+deployments. In OPNFV Colorado release, Kingbird provides centralized quota
+management feature. Administrator can set quota per project based in Kingbird
+and sync the quota limit to multi-region OpenStack periodiclly or on-demand.
+The tenant can check the total quota limit and usage from Kingbird for all
+regions. Administrator can aslo manage the default quota by quota class
+setting.
+
+Following quota items are supported to be managed in Kingbird:
+
+- **instances**: Number of instances allowed per project.
+- **cores**: Number of instance cores allowed per project.
+- **ram**: Megabytes of instance RAM allowed per project.
+- **metadata_items**: Number of metadata items allowed per instance.
+- **key_pairs**: Number of key pairs per user.
+- **fixed_ips**: Number of fixed IPs allowed per project,
+  valid if Nova Network is used.
+- **security_groups**: Number of security groups per project,
+  valid if Nova Network is used.
+- **floating_ips**: Number of floating IPs allowed per project,
+  valid if Nova Network is used.
+- **network**: Number of networks allowed per project,
+  valid if Neutron is used.
+- **subnet**: Number of subnets allowed per project,
+  valid if Neutron is used.
+- **port**: Number of ports allowed per project,
+  valid if Neutron is used.
+- **security_group**: Number of security groups allowed per project,
+  valid if Neutron is used.
+- **security_group_rule**: Number of security group rules allowed per project,
+  valid if Neutron is used.
+- **router**: Number of routers allowed per project,
+  valid if Neutron is used.
+- **floatingip**: Number of floating IPs allowed per project,
+  valid if Neutron is used.
+- **volumes**: Number of volumes allowed per project.
+- **snapshots**: Number of snapshots allowed per project.
+- **gigabytes**: Total amount of storage, in gigabytes, allowed for volumes
+  and snapshots per project.
+- **backups**: Number of volume backups allowed per project.
+- **backup_gigabytes**: Total amount of storage, in gigabytes, allowed for volume
+  backups per project.
+
+Only restful APIs are provided for Kingbird in Colorado release, so curl or
+other http client can be used to call Kingbird API.
+
+Before use the following command, get token, project id, and kingbird service
+endpoint first. Use $kb_token to repesent the token, and $admin_tenant_id as
+administrator project_id, and $tenant_id as the target project_id for quota
+management and $kb_ip_addr for the kingbird service endpoint ip address.
+
+Note:
+To view all tenants (projects), run:
+
+.. code-block:: bash
+
+    openstack project list
+
+To get token, run:
+
+.. code-block:: bash
+
+    openstack token issue
+
+To get Kingbird service endpoint, run:
+
+.. code-block:: bash
+
+    openstack endpoint list
+
+Quota Management API
+--------------------
+
+1. Update global limit for a tenant
+
+   curl \
+   -H "Content-Type: application/json" \
+   -H "X-Auth-Token: $kb_token" \
+   -H  "ROLE: dmin" \
+   -X PUT \
+   -d '{"quota_set":{"cores": 10,"ram": 51200, "metadata_items": 100,"key_pairs": 100, "network":20,"security_group": 20,"security_group_rule": 20}}' \
+   http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-sets/$tenant_id
+
+2. Get global limit for a tenant
+
+   curl \
+   -H "Content-Type: application/json" \
+   -H "X-Auth-Token: $kb_token" \
+   -H  "X_ROLE: admin" \
+   http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-sets/$tenant_id
+
+3. A tenant can also get the global limit by himself
+
+   curl \
+   -H "Content-Type: application/json" \
+   -H "X-Auth-Token: $kb_token" \
+   http://$kb_ip_addr:8118/v1.0/$tenant_id/os-quota-sets/$tenant_id
+
+4. Get defaults limits
+
+   curl \
+   -H "Content-Type: application/json" \
+   -H "X-Auth-Token: $kb_token" \
+   -H  "X_ROLE: admin" \
+   http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-sets/defaults
+
+5. Get total usage for a tenant
+
+   curl \
+   -H "Content-Type: application/json" \
+   -H "X-Auth-Token: $kb_token" \
+   -H  "X_ROLE: admin" \
+   -X GET \
+   http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-sets/$tenant_id/detail
+
+6. A tenant can also get the total usage by himself
+
+   curl \
+   -H "Content-Type: application/json" \
+   -H "X-Auth-Token: $kb_token" \
+   -X GET \
+   http://$kb_ip_addr:8118/v1.0/$tenant_id/os-quota-sets/$tenant_id/detail
+
+7. On demand quota sync
+
+   curl \
+   -H "Content-Type: application/json" \
+   -H "X-Auth-Token: $kb_token" \
+   -H  "X_ROLE: admin" \
+   -X PUT \
+   http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-sets/$tenant_id/sync
+
+
+8. Delete specific global limit for a tenant
+
+   curl \
+   -H "Content-Type: application/json" \
+   -H "X-Auth-Token: $kb_token" \
+   -H  "X_ROLE: admin" \
+   -X DELETE \
+   -d '{"quota_set": [ "cores", "ram"]}' \
+   http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-sets/$tenant_id
+
+9. Delete all kingbird global limit for a tenant
+
+  curl \
+  -H "Content-Type: application/json" \
+  -H "X-Auth-Token: $kb_token" \
+  -H  "X_ROLE: admin" \
+  -X DELETE \
+  http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-sets/$tenant_id
+
+
+Quota Class API
+---------------
+
+1. Update default quota class
+
+   curl \
+   -H "Content-Type: application/json" \
+   -H "X-Auth-Token: $kb_token" \
+   -H  "ROLE: dmin" \
+   -X PUT \
+   -d '{"quota_class_set":{"cores": 100, "network":50,"security_group": 50,"security_group_rule": 50}}' \
+   http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-class-sets/default
+
+2. Get default quota class
+
+   curl \
+   -H "Content-Type: application/json" \
+   -H "X-Auth-Token: $kb_token" \
+   -H  "X_ROLE: admin" \
+   http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-class-sets/default
+
+3. Delete default quota class
+
+   curl \
+   -H "Content-Type: application/json" \
+   -H "X-Auth-Token: $kb_token" \
+   -H  "ROLE: dmin" \
+   -X DELETE \
+   http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-class-sets/default
+