summaryrefslogtreecommitdiffstats
path: root/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc
diff options
context:
space:
mode:
authorbobzhou <zhoubo8@lenovo.com>2018-12-17 17:41:39 +0800
committerbobzhou <zhoubo8@lenovo.com>2018-12-17 17:41:39 +0800
commit31560e9a26e6a7e2e65d18924c938ee7a7683fb3 (patch)
tree26ec12331a2ec612a0f51c7866fbe41262ca0fe1 /cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc
parentfc1b7908a77b5cce763f92b3d314057345f3ab7a (diff)
upload cyborg base mitaka version
Change-Id: Iff52024026c9eff899246be70cc89fe6e65befff Signed-off-by: bobzhou <zhoubo8@lenovo.com>
Diffstat (limited to 'cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc')
-rw-r--r--cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/contributing.rst144
-rw-r--r--cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/index.rst14
-rw-r--r--cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/pike/approved/cyborg-agent.rst164
-rw-r--r--cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/pike/approved/cyborg-api-proposal.rst410
-rw-r--r--cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/pike/approved/cyborg-conductor.rst142
-rw-r--r--cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/pike/approved/cyborg-driver-proposal.rst161
-rw-r--r--cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/template.rst392
7 files changed, 1427 insertions, 0 deletions
diff --git a/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/contributing.rst b/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/contributing.rst
new file mode 100644
index 0000000..9c67ca7
--- /dev/null
+++ b/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/contributing.rst
@@ -0,0 +1,144 @@
+============
+Contributing
+============
+
+Contributions are most welcome! You must first create a
+Launchpad account and `follow the instructions here <https://docs.openstack.org/infra/manual/developers.html#account-setup>`_
+to get started as a new OpenStack contributor.
+
+Once you've signed the contributor license agreement and read through
+the above documentation, add your public SSH key under the 'SSH Public Keys'
+section of review.openstack.org_.
+
+.. _review.openstack.org: https://review.openstack.org/#/settings/
+
+You can view your public key using:
+
+::
+
+ $ cat ~/.ssh/id_*.pub
+
+Set your username and email for review.openstack.org:
+
+::
+
+ $ git config --global user.email "example@example.com"
+ $ git config --global user.name "example"
+ $ git config --global --add gitreview.username "example"
+
+Next, Clone the github repository:
+
+::
+
+ $ git clone https://github.com/openstack/cyborg.git
+
+You need to have git-review in order to be able to submit patches using
+the gerrit code review system. You can install it using:
+
+::
+
+ $ sudo yum install git-review
+
+To set up your cloned repository to work with OpenStack Gerrit
+
+::
+
+ $ git review -s
+
+It's useful to create a branch to do your work, name it something
+related to the change you'd like to introduce.
+
+::
+
+ $ cd cyborg
+ $ git branch my_special_enhancement
+ $ git checkout !$
+
+Make your changes and then commit them using the instructions
+below.
+
+::
+
+ $ git add /path/to/files/changed
+ $ git commit
+
+Use a descriptive commit title followed by an empty space.
+You should type a small justification of what you are
+changing and why.
+
+Now you're ready to submit your changes for review:
+
+::
+
+ $ git review
+
+
+If you want to make another patchset from the same commit you can
+use the amend feature after further modification and saving.
+
+::
+
+ $ git add /path/to/files/changed
+ $ git commit --amend
+ $ git review
+
+If you want to submit a new patchset from a different location
+(perhaps on a different machine or computer for example) you can
+clone the Cyborg repo again (if it doesn't already exist) and then
+use git review against your unique Change-ID:
+
+::
+
+ $ git review -d Change-Id
+
+Change-Id is the change id number as seen in Gerrit and will be
+generated after your first successful submission.
+
+The above command downloads your patch onto a separate branch. You might
+need to rebase your local branch with remote master before running it to
+avoid merge conflicts when you resubmit the edited patch. To avoid this
+go back to a "safe" commit using:
+
+::
+
+ $ git reset --hard commit-number
+
+Then,
+
+::
+
+ $ git fetch origin
+
+::
+
+ $ git rebase origin/master
+
+Make the changes on the branch that was setup by using the git review -d
+(the name of the branch is along the lines of
+review/username/branch_name/patchsetnumber).
+
+Add the files to git and commit your changes using,
+
+::
+
+ $ git commit --amend
+
+You can edit your commit message as well in the prompt shown upon
+executing above command.
+
+Finally, push the patch for review using,
+
+::
+
+ $ git review
+
+Adding functionality
+--------------------
+
+If you are adding new functionality to Cyborg please add testing for that functionality
+and provide a detailed commit message outlining the goals of your commit and how you
+achived them.
+
+If the functionality you wish to add doesn't fix in an existing part of the Cyborg
+achitecture diagram drop by our team meetings to disscuss how it could be implemented
+
diff --git a/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/index.rst b/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/index.rst
new file mode 100644
index 0000000..f4ba1d5
--- /dev/null
+++ b/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/index.rst
@@ -0,0 +1,14 @@
+Cyborg Specs
+============
+
+Pike
+----
+
+This section has a list of specs for Pike release.
+
+
+.. toctree::
+ :maxdepth: 1
+ :glob:
+
+ specs/pike/approved/*
diff --git a/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/pike/approved/cyborg-agent.rst b/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/pike/approved/cyborg-agent.rst
new file mode 100644
index 0000000..f47ae11
--- /dev/null
+++ b/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/pike/approved/cyborg-agent.rst
@@ -0,0 +1,164 @@
+..
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
+ License.
+
+ http://creativecommons.org/licenses/by/3.0/legalcode
+
+==========================================
+ Cyborg Agent Proposal
+==========================================
+
+https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-agent
+
+This spec proposes the responsibilities and initial design of the
+Cyborg Agent.
+
+Problem description
+===================
+
+Cyborg requires an agent on the compute hosts to manage the several
+responsibilities, including locating accelerators, monitoring their
+status, and orchestrating driver operations.
+
+Use Cases
+---------
+
+Use of accelerators attached to virtual machine instances in OpenStack
+
+Proposed change
+===============
+
+Cyborg Agent resides on various compute hosts and monitors them for accelerators.
+On it's first run Cyborg Agent will run the detect accelerator functions of all
+it's installed drivers. The resulting list of accelerators available on the host
+will be reported to the conductor where it will be stored into the database and
+listed during API requests. By default accelerators will be inserted into the
+database in a inactive state. It will be up to the operators to manually set
+an accelerator to 'ready' at which point cyborg agent will be responsible for
+calling the drivers install function and ensuring that the accelerator is ready
+for use.
+
+In order to mirror the current Nova model of using the placement API each Agent
+will send updates on it's resources directly to the placement API endpoint as well
+as to the conductor for usage aggregation. This should keep placement API up to date
+on accelerators and their usage.
+
+Alternatives
+------------
+
+There are lots of alternate ways to lay out the communication between the Agent
+and the API endpoint or the driver. Almost all of them involving exactly where we
+draw the line between the driver, Conductor , and Agent. I've written my proposal
+with the goal of having the Agent act mostly as a monitoring tool, reporting to
+the cloud operator or other Cyborg components to take action. A more active role
+for Cyborg Agent is possible but either requires significant synchronization with
+the Conductor or potentially steps on the toes of operators.
+
+Data model impact
+-----------------
+
+Cyborg Agent will create new entries in the database for accelerators it detects
+it will also update those entries with the current status of the accelerator
+at a high level. More temporary data like the current usage of a given accelerator
+will be broadcast via a message passing system and won't be stored.
+
+Cyborg Agent will retain a local cache of this data with the goal of not losing accelerator
+state on system interruption or loss of connection.
+
+
+REST API impact
+---------------
+
+TODO once we firm up who's responsible for what.
+
+Security impact
+---------------
+
+Monitoring capability might be useful to an attacker, but without root
+this is a fairly minor concern.
+
+Notifications impact
+--------------------
+
+Notifying users that their accelerators are ready?
+
+Other end user impact
+---------------------
+
+Interaction details around adding/removing/setting up accelerators
+details TBD.
+
+Performance Impact
+------------------
+
+Agent heartbeat for updated accelerator performance stats might make
+scaling to many accelerator hosts a challenge for the Cyborg endpoint
+and database. Perhaps we should consider doing an active 'load census'
+before scheduling instances? But that just moves the problem from constant
+load to issues with a bootstorm.
+
+
+Other deployer impact
+---------------------
+
+By not placing the drivers with the Agent we keep the deployment footprint
+pretty small. We do add development complexity and security concerns sending
+them over the wire though.
+
+Developer impact
+----------------
+
+TBD
+
+Implementation
+==============
+
+Assignee(s)
+-----------
+
+Primary assignee:
+ <jkilpatr>
+
+Other contributors:
+ <launchpad-id or None>
+
+Work Items
+----------
+
+* Agent implementation
+
+Dependencies
+============
+
+* Cyborg Driver Spec
+* Cyborg API Spec
+* Cyborg Conductor Spec
+
+Testing
+=======
+
+CI infrastructure with a set of accelerators, drivers, and hardware will be
+required for testing the Agent installation and operation regularly.
+
+Documentation Impact
+====================
+
+Little to none. Perhaps on an on compute config file that may need to be
+documented. But I think it's best to avoid local configuration where possible.
+
+References
+==========
+
+Other Cyborg Specs
+
+History
+=======
+
+
+.. list-table:: Revisions
+ :header-rows: 1
+
+ * - Release
+ - Description
+ * - Pike
+ - Introduced
diff --git a/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/pike/approved/cyborg-api-proposal.rst b/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/pike/approved/cyborg-api-proposal.rst
new file mode 100644
index 0000000..42ddad7
--- /dev/null
+++ b/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/pike/approved/cyborg-api-proposal.rst
@@ -0,0 +1,410 @@
+..
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
+ License.
+
+ http://creativecommons.org/licenses/by/3.0/legalcode
+
+===================
+Cyborg API proposal
+===================
+
+https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-api
+
+This spec proposes to provide the initial API design for Cyborg.
+
+Problem description
+===================
+
+Cyborg as a common management framework for dedicated devices (hardware/
+software accelerators, high-speed storage, etc) needs RESTful API to expose
+the basic functionalities.
+
+Use Cases
+---------
+
+* As a user I want to be able to spawn VM with dedicated hardware, so
+that I can utilize provided hardware.
+* As a compute service I need to know how requested resource should be
+attached to the VM.
+* As a scheduler service I'd like to know on which resource provider
+requested resource can be found.
+
+Proposed change
+===============
+
+In general we want to develop the APIs that support basic life cycle management
+for Cyborg.
+
+Life Cycle Management Phases
+----------------------------
+
+For cyborg, LCM phases include typical create, retrieve, update, delete operations.
+One thing should be noted that deprovisioning mainly refers to detach(delete) operation
+which deactivate an acceleration capability but preserve the resource itself
+for future usage. For Cyborg, from functional point of view, the LCM includes provision,
+attach,update,list, and detach. There is no notion of deprovisioning for Cyborg API
+in a sense that we decomission or disconnect an entire accelerator device from
+the bus.
+
+Difference between Provision and Attach/Detach
+----------------------------------------------
+
+Noted that while the APIs support provisioning via CRUD operations, attach/detach
+are considered different:
+
+* Provision operations (create) will involve api->
+conductor->agent->driver workflow, where as attach/detach (update/delete) could be taken
+care of at the driver layer without the involvement of the pre-mentioned workflow. This
+is similar to the difference between create a volume and attach/detach a volume in Cinder.
+
+* The attach/detach in Cyborg API will mainly involved in DB status modification.
+
+Difference between Attach/Detach To VM and Host
+-----------------------------------------------
+
+Moreover there are also differences when we attach an accelerator to a VM or
+a host, similar to Cinder.
+
+* When the attachment happens to a VM, we are expecting that Nova could call
+the virt driver to perform the action for the instance. In this case Nova
+needs to support the acc-attach and acc-detach action.
+
+* When the attachment happens to a host, we are expecting that Cyborg could
+take care of the action itself via Cyborg driver. Althrough currently there
+is the generic driver to accomplish the job, we should consider a os-brick
+like standalone lib for accelerator attach/detach operations.
+
+Alternatives
+------------
+
+* For attaching an accelerator to a VM, we could let Cyborg perform the action
+itself, however it runs into the risk of tight-coupling with Nova of which Cyborg
+needs to get instance related information.
+* For attaching an accelerator to a host, we could consider to use Ironic drivers
+however it might not bode well with the standalone accelerator rack scenarios where
+accelerators are not attached to server at all.
+
+Data model impact
+-----------------
+
+A new table in the API database will be created::
+
+ CREATE TABLE accelerators (
+ accelerator_id INT NOT NULL,
+ device_type STRING NOT NULL,
+ acc_type STRING NOT NULL,
+ acc_capability STRING NOT NULL,
+ vendor_id STRING,
+ product_id STRING,
+ remotable INT,
+ );
+
+Note that there is an ongoing discussion on nested resource
+provider new data structures that will impact Cyborg DB imp-
+lementation. For code implementation it should be aligned
+with resource provider db requirement as much as possible.
+
+
+REST API impact
+---------------
+
+The API changes add resource endpoints to:
+
+* `GET` a list of all the accelerators
+* `GET` a single accelerator for a given id
+* `POST` create a new accelerator resource
+* `PUT` an update to an existing accelerator spec
+* `PUT` attach an accelerator to a VM or a host
+* `DELETE` detach an existing accelerator for a given id
+
+The following new REST API call will be created:
+
+'GET /accelerators'
+*************************
+
+Return a list of accelerators managed by Cyborg
+
+Example message body of the response to the GET operation::
+
+ 200 OK
+ Content-Type: application/json
+
+ {
+ "accelerator":[
+ {
+ "uuid":"8e45a2ea-5364-4b0d-a252-bf8becaa606e",
+ "acc_specs":
+ {
+ "remote":0,
+ "num":1,
+ "device_type":"CRYPTO"
+ "acc_capability":
+ {
+ "num":2
+ "ipsec":
+ {
+ "aes":
+ {
+ "3des":50,
+ "num":1,
+ }
+ }
+ }
+ }
+ },
+ {
+ "uuid":"eaaf1c04-ced2-40e4-89a2-87edded06d64",
+ "acc_specs":
+ {
+ "remote":0,
+ "num":1,
+ "device_type":"CRYPTO"
+ "acc_capability":
+ {
+ "num":2
+ "ipsec":
+ {
+ "aes":
+ {
+ "3des":40,
+ "num":1,
+ }
+ }
+ }
+ }
+ }
+ ]
+ }
+
+'GET /accelerators/{uuid}'
+*************************
+
+Retrieve a certain accelerator info indetified by '{uuid}'
+
+Example GET Request::
+
+ GET /accelerators/8e45a2ea-5364-4b0d-a252-bf8becaa606e
+
+ 200 OK
+ Content-Type: application/json
+
+ {
+ "uuid":"8e45a2ea-5364-4b0d-a252-bf8becaa606e",
+ "acc_specs":{
+ "remote":0,
+ "num":1,
+ "device_type":"CRYPTO"
+ "acc_capability":{
+ "num":2
+ "ipsec":{
+ "aes":{
+ "3des":50,
+ "num":1,
+ }
+ }
+ }
+ }
+ }
+
+If the accelerator does not exist a `404 Not Found` must be
+returned.
+
+'POST /accelerators/{uuid}'
+*******************
+
+Create a new accelerator
+
+Example POST Request::
+
+ Content-type: application/json
+
+ {
+ "name": "IPSec Card",
+ "uuid": "8e45a2ea-5364-4b0d-a252-bf8becaa606e"
+ }
+
+The body of the request must match the following JSONSchema document::
+
+ {
+ "type": "object",
+ "properties": {
+ "name": {
+ "type": "string"
+ },
+ "uuid": {
+ "type": "string",
+ "format": "uuid"
+ }
+ },
+ "required": [
+ "name"
+ ]
+ "additionalProperties": False
+ }
+
+The response body is empty. The headers include a location header
+pointing to the created accelerator resource::
+
+ 201 Created
+ Location: /accelerators/8e45a2ea-5364-4b0d-a252-bf8becaa606e
+
+A `409 Conflict` response code will be returned if another accelerator
+exists with the provided name.
+
+'PUT /accelerators/{uuid}/{acc_spec}'
+*************************
+
+Update the spec for the accelerator identified by `{uuid}`.
+
+Example::
+
+ PUT /accelerator/8e45a2ea-5364-4b0d-a252-bf8becaa606e
+
+ Content-type: application/json
+
+ {
+ "acc_specs":{
+ "remote":0,
+ "num":1,
+ "device_type":"CRYPTO"
+ "acc_capability":{
+ "num":2
+ "ipsec":{
+ "aes":{
+ "3des":50,
+ "num":1,
+ }
+ }
+ }
+ }
+ }
+
+The returned HTTP response code will be one of the following:
+
+* `200 OK` if the spec is successfully updated
+* `404 Not Found` if the accelerator identified by `{uuid}` was
+ not found
+* `400 Bad Request` for bad or invalid syntax
+* `409 Conflict` if another process updated the same spec.
+
+
+'PUT /accelerators/{uuid}'
+*************************
+
+Attach the accelerator identified by `{uuid}`.
+
+Example::
+
+ PUT /accelerator/8e45a2ea-5364-4b0d-a252-bf8becaa606e
+
+ Content-type: application/json
+
+ {
+ "name": "IPSec Card",
+ "uuid": "8e45a2ea-5364-4b0d-a252-bf8becaa606e"
+ }
+
+The returned HTTP response code will be one of the following:
+
+* `200 OK` if the accelerator is successfully attached
+* `404 Not Found` if the accelerator identified by `{uuid}` was
+ not found
+* `400 Bad Request` for bad or invalid syntax
+* `409 Conflict` if another process attach the same accelerator.
+
+
+'DELETE /accelerator/{uuid}'
+****************************
+
+Detach the accelerator identified by `{uuid}`.
+
+The body of the request and the response is empty.
+
+The returned HTTP response code will be one of the following:
+
+* `204 No Content` if the request was successful and the accelerator was detached.
+* `404 Not Found` if the accelerator identified by `{uuid}` was
+ not found.
+* `409 Conflict` if there exist allocations records for any of the
+ accelerator resource that would be detached as a result of detaching the accelerator.
+
+
+Security impact
+---------------
+
+None
+
+Notifications impact
+--------------------
+
+None
+
+Other end user impact
+---------------------
+
+None
+
+Performance Impact
+------------------
+
+None
+
+Other deployer impact
+---------------------
+
+None
+
+Developer impact
+----------------
+
+Developers can use this REST API after it has been implemented.
+
+Implementation
+==============
+
+Assignee(s)
+-----------
+
+Primary assignee:
+ zhipengh <huangzhipeng@huawei.com>
+
+Work Items
+----------
+
+* Implement the APIs specified in this spec
+* Proposal to Nova about the new accelerator
+attach/detach api
+* Implement the DB specified in this spec
+
+
+Dependencies
+============
+
+None.
+
+Testing
+=======
+
+* Unit tests will be added to Cyborg API.
+
+Documentation Impact
+====================
+
+None
+
+References
+==========
+
+None
+
+History
+=======
+
+
+.. list-table:: Revisions
+ :header-rows: 1
+
+ * - Release
+ - Description
+ * - Pike
+ - Introduced
diff --git a/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/pike/approved/cyborg-conductor.rst b/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/pike/approved/cyborg-conductor.rst
new file mode 100644
index 0000000..a1e8ffc
--- /dev/null
+++ b/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/pike/approved/cyborg-conductor.rst
@@ -0,0 +1,142 @@
+..
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
+ License.
+
+ http://creativecommons.org/licenses/by/3.0/legalcode
+
+==========================================
+ Cyborg Conductor Proposal
+==========================================
+
+https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-agent
+
+This spec proposes the responsibilities and initial design of the
+Cyborg Conductor.
+
+Problem description
+===================
+
+Cyborg requires a conductor on the controller hosts to manage the cyborg
+system state and coalesce database operations.
+
+Use Cases
+---------
+
+Use of accelerators attached to virtual machine instances in OpenStack
+
+Proposed change
+===============
+
+Cyborg Conductor will reside on the control node and will be
+responsible for stateful actions taken by Cyborg. Acting as both a cache to
+the database and as a method of combining reads and writes to the database.
+All other Cyborg components will go through the conductor for database operations.
+
+Alternatives
+------------
+
+Having each Cyborg Agent instance hit the database on it's own is a possible
+alternative, and it may even be feasible if the accelerator load monitoring rate is
+very low and the vast majority of operations are reads. But since we intend to store
+metadata about accelerator usage updated regularly this model probably will not scale
+well.
+
+Data model impact
+-----------------
+
+Using the conductor 'properly' will result in little or no per instance state and stateful
+operations moving through the conductor with the exception of some local caching where it
+can be garunteed to work well.
+
+REST API impact
+---------------
+
+N/A
+
+Security impact
+---------------
+
+Negligible
+
+Notifications impact
+--------------------
+
+N/A
+
+Other end user impact
+---------------------
+
+Faster Cybrog operation and less database load.
+
+Performance Impact
+------------------
+
+Generally positive so long as we don't overload the messaging bus trying
+to pass things to the Conductor to write out.
+
+Other deployer impact
+---------------------
+
+Conductor must be installed and configured on the controllers.
+
+
+Developer impact
+----------------
+
+None for API users, internally heavy use of message passing will
+be required if we want to keep all system state in the controllers.
+
+
+Implementation
+==============
+
+Assignee(s)
+-----------
+
+Primary assignee:
+ jkilpatr
+
+Other contributors:
+ None
+
+Work Items
+----------
+
+* Implementation
+* Integration with API and Agent
+
+Dependencies
+============
+
+* Cyborg API spec
+* Cyborg Agent spec
+
+Testing
+=======
+
+This component should be possible to fully test using unit tests and functional
+CI using the dummy driver.
+
+Documentation Impact
+====================
+
+Some configuration values tuning save out rate and other parameters on the controller
+will need to be documented for end users
+
+References
+==========
+
+Cyborg API Spec
+Cyborg Agent Spec
+
+History
+=======
+
+
+.. list-table:: Revisions
+ :header-rows: 1
+
+ * - Release
+ - Description
+ * - Pike
+ - Introduced
diff --git a/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/pike/approved/cyborg-driver-proposal.rst b/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/pike/approved/cyborg-driver-proposal.rst
new file mode 100644
index 0000000..4fabf56
--- /dev/null
+++ b/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/pike/approved/cyborg-driver-proposal.rst
@@ -0,0 +1,161 @@
+..
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
+ License.
+
+ http://creativecommons.org/licenses/by/3.0/legalcode
+
+==============================
+Cyborg Generic Driver Proposal
+==============================
+
+https://blueprints.launchpad.net/openstack-cyborg/+spec/generic-driver-cyborg
+
+This spec proposes to provide the initial design for Cyborg's generic driver.
+
+Problem description
+===================
+
+This blueprint proposes to add a generic driver for openstack-cyborg.
+The goal is to provide users & operators with a reliable generic
+implementation that is hardware agnostic and provides basic
+accelerator functionality.
+
+Use Cases
+---------
+
+* As an admin user and a non-admin user with elevated privileges, I should be
+ able to identify and discover attached accelerator backends.
+* As an admin user and a non-admin user with elevated privileges, I should be
+ able to view services on each attached backend after the agent has
+ discovered services on each backend.
+* As an admin user and a non-admin user, I should be able to list and update
+ attached accelerators by driver by querying nova with the Cyborg-API.
+* As an admin user and a non-admin user with elevated privileges, I should be
+ able to install accelerator generic driver.
+* As an admin user and a non-admin user with elevated privileges, I should be
+ able to uninstall accelerator generic driver.
+* As an admin user and a non-admin user with elevated privileges, I should be
+ able to issue attach command to the instance via the driver which gets
+ routed to Nova via the Cyborg API.
+* As an admin user and a non-admin user with elevated privileges, I should be
+ able to issue detach command to the instance via the driver which gets
+ routed to Nova via the Cyborg API.
+
+Proposed change
+===============
+
+* Cyborg needs a reference implementation that can be used as a model for
+ future driver implementations and that will be referred to as the generic
+ driver implementation
+* Develop the generic driver implementation that supports CRUD operations for
+ accelerators for single backend and multi backend setup scenarios.
+
+
+Alternatives
+------------
+
+None
+
+Data model impact
+-----------------
+
+* The generic driver will update the central database when any CRUD or
+ attach/detach operations take place
+
+REST API impact
+---------------
+
+This blueprint proposes to add the following APIs:
+*cyborg install-driver <driver_id>
+*cyborg uninstall-driver <driver_id>
+*cyborg attach-instance <instance_id>
+*cyborg detach-instance <instance_id>
+*cyborg service-list
+*cyborg driver-list
+*cyborg update-driver <driver_id>
+*cyborg discover-services
+
+Security impact
+---------------
+
+None
+
+Notifications impact
+--------------------
+
+None
+
+Other end user impact
+---------------------
+
+None
+
+Performance Impact
+------------------
+
+None
+
+Other deployer impact
+---------------------
+
+None
+
+Developer impact
+----------------
+
+Developers will have access to a reference generic implementation which
+can be used to build vendor-specific drivers.
+
+Implementation
+==============
+
+Assignee(s)
+-----------
+
+Primary assignee:
+ Rushil Chugh <rushil.chugh@gmail.com>
+
+Work Items
+----------
+
+This change would entail the following:
+* Add a feature to identify and discover attached accelerator backends.
+* Add a feature to list services running on the backend
+* Add a feature to attach accelerators to the generic backend.
+* Add a feature to detach accelerators from the generic backend.
+* Add a feature to list accelerators attached to the generic backend.
+* Add a feature to modify accelerators attached to the generic backend.
+* Defining a reference implementation detailing the flow of requests between
+ the cyborg-api, cyborg-conductor and nova-compute services.
+
+Dependencies
+============
+
+Dependent on Cyborg API and Agent implementations.
+
+Testing
+=======
+
+* Unit tests will be added test Cyborg generic driver.
+
+Documentation Impact
+====================
+
+None
+
+References
+==========
+
+None
+
+History
+=======
+
+
+.. list-table:: Revisions
+ :header-rows: 1
+
+ * - Release
+ - Description
+ * - Pike
+ - Introduced
diff --git a/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/template.rst b/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/template.rst
new file mode 100644
index 0000000..6a1fe37
--- /dev/null
+++ b/cyborg_enhancement/mitaka_version/cyborg/doc/source/devdoc/specs/template.rst
@@ -0,0 +1,392 @@
+..
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
+ License.
+
+ http://creativecommons.org/licenses/by/3.0/legalcode
+
+==========================================
+Example Spec - The title of your blueprint
+==========================================
+
+Include the URL of your launchpad blueprint:
+
+https://blueprints.launchpad.net/openstack-cyborg/+spec/example
+
+Introduction paragraph -- why are we doing anything? A single paragraph of
+prose that operators can understand. The title and this first paragraph
+should be used as the subject line and body of the commit message
+respectively.
+
+Some notes about the cyborg-spec and blueprint process:
+
+* Not all blueprints need a spec. For more information see
+ http://docs.openstack.org/developer/cyborg/blueprints.html#specs
+
+* The aim of this document is first to define the problem we need to solve,
+ and second agree the overall approach to solve that problem.
+
+* This is not intended to be extensive documentation for a new feature.
+ For example, there is no need to specify the exact configuration changes,
+ nor the exact details of any DB model changes. But you should still define
+ that such changes are required, and be clear on how that will affect
+ upgrades.
+
+* You should aim to get your spec approved before writing your code.
+ While you are free to write prototypes and code before getting your spec
+ approved, its possible that the outcome of the spec review process leads
+ you towards a fundamentally different solution than you first envisaged.
+
+* But, API changes are held to a much higher level of scrutiny.
+ As soon as an API change merges, we must assume it could be in production
+ somewhere, and as such, we then need to support that API change forever.
+ To avoid getting that wrong, we do want lots of details about API changes
+ upfront.
+
+Some notes about using this template:
+
+* Your spec should be in ReSTructured text, like this template.
+
+* Please wrap text at 79 columns.
+
+* The filename in the git repository should match the launchpad URL, for
+ example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing
+ should be named awesome-thing.rst
+
+* Please do not delete any of the sections in this template. If you have
+ nothing to say for a whole section, just write: None
+
+* For help with syntax, see http://sphinx-doc.org/rest.html
+
+* To test out your formatting, build the docs using tox and see the generated
+ HTML file in doc/build/html/specs/<path_of_your_file>
+
+* If you would like to provide a diagram with your spec, ascii diagrams are
+ required. http://asciiflow.com/ is a very nice tool to assist with making
+ ascii diagrams. The reason for this is that the tool used to review specs is
+ based purely on plain text. Plain text will allow review to proceed without
+ having to look at additional files which can not be viewed in gerrit. It
+ will also allow inline feedback on the diagram itself.
+
+* If your specification proposes any changes to the Cyborg REST API such
+ as changing parameters which can be returned or accepted, or even
+ the semantics of what happens when a client calls into the API, then
+ you should add the APIImpact flag to the commit message. Specifications with
+ the APIImpact flag can be found with the following query:
+
+ https://review.openstack.org/#/q/status:open+project:openstack/cyborg+message:apiimpact,n,z
+
+
+Problem description
+===================
+
+A detailed description of the problem. What problem is this blueprint
+addressing?
+
+Use Cases
+---------
+
+What use cases does this address? What impact on actors does this change have?
+Ensure you are clear about the actors in each use case: Developer, End User,
+Deployer etc.
+
+Proposed change
+===============
+
+Here is where you cover the change you propose to make in detail. How do you
+propose to solve this problem?
+
+If this is one part of a larger effort make it clear where this piece ends. In
+other words, what's the scope of this effort?
+
+At this point, if you would like to just get feedback on if the problem and
+proposed change fit in Cyborg, you can stop here and post this for review to get
+preliminary feedback. If so please say:
+Posting to get preliminary feedback on the scope of this spec.
+
+Alternatives
+------------
+
+What other ways could we do this thing? Why aren't we using those? This doesn't
+have to be a full literature review, but it should demonstrate that thought has
+been put into why the proposed solution is an appropriate one.
+
+Data model impact
+-----------------
+
+Changes which require modifications to the data model often have a wider impact
+on the system. The community often has strong opinions on how the data model
+should be evolved, from both a functional and performance perspective. It is
+therefore important to capture and gain agreement as early as possible on any
+proposed changes to the data model.
+
+Questions which need to be addressed by this section include:
+
+* What new data objects and/or database schema changes is this going to
+ require?
+
+* What database migrations will accompany this change.
+
+* How will the initial set of new data objects be generated, for example if you
+ need to take into account existing instances, or modify other existing data
+ describe how that will work.
+
+REST API impact
+---------------
+
+Each API method which is either added or changed should have the following
+
+* Specification for the method
+
+ * A description of what the method does suitable for use in
+ user documentation
+
+ * Method type (POST/PUT/GET/DELETE)
+
+ * Normal http response code(s)
+
+ * Expected error http response code(s)
+
+ * A description for each possible error code should be included
+ describing semantic errors which can cause it such as
+ inconsistent parameters supplied to the method, or when an
+ instance is not in an appropriate state for the request to
+ succeed. Errors caused by syntactic problems covered by the JSON
+ schema definition do not need to be included.
+
+ * URL for the resource
+
+ * URL should not include underscores, and use hyphens instead.
+
+ * Parameters which can be passed via the url
+
+ * JSON schema definition for the request body data if allowed
+
+ * Field names should use snake_case style, not CamelCase or MixedCase
+ style.
+
+ * JSON schema definition for the response body data if any
+
+ * Field names should use snake_case style, not CamelCase or MixedCase
+ style.
+
+* Example use case including typical API samples for both data supplied
+ by the caller and the response
+
+* Discuss any policy changes, and discuss what things a deployer needs to
+ think about when defining their policy.
+
+Note that the schema should be defined as restrictively as
+possible. Parameters which are required should be marked as such and
+only under exceptional circumstances should additional parameters
+which are not defined in the schema be permitted (eg
+additionaProperties should be False).
+
+Reuse of existing predefined parameter types such as regexps for
+passwords and user defined names is highly encouraged.
+
+Security impact
+---------------
+
+Describe any potential security impact on the system. Some of the items to
+consider include:
+
+* Does this change touch sensitive data such as tokens, keys, or user data?
+
+* Does this change alter the API in a way that may impact security, such as
+ a new way to access sensitive information or a new way to login?
+
+* Does this change involve cryptography or hashing?
+
+* Does this change require the use of sudo or any elevated privileges?
+
+* Does this change involve using or parsing user-provided data? This could
+ be directly at the API level or indirectly such as changes to a cache layer.
+
+* Can this change enable a resource exhaustion attack, such as allowing a
+ single API interaction to consume significant server resources? Some examples
+ of this include launching subprocesses for each connection, or entity
+ expansion attacks in XML.
+
+For more detailed guidance, please see the OpenStack Security Guidelines as
+a reference (https://wiki.openstack.org/wiki/Security/Guidelines). These
+guidelines are a work in progress and are designed to help you identify
+security best practices. For further information, feel free to reach out
+to the OpenStack Security Group at openstack-security@lists.openstack.org.
+
+Notifications impact
+--------------------
+
+Please specify any changes to notifications. Be that an extra notification,
+changes to an existing notification, or removing a notification.
+
+Other end user impact
+---------------------
+
+Aside from the API, are there other ways a user will interact with this
+feature?
+
+* Does this change have an impact on python-cyborgclient? What does the user
+ interface there look like?
+
+Performance Impact
+------------------
+
+Describe any potential performance impact on the system, for example
+how often will new code be called, and is there a major change to the calling
+pattern of existing code.
+
+Examples of things to consider here include:
+
+* A periodic task might look like a small addition but if it calls conductor or
+ another service the load is multiplied by the number of nodes in the system.
+
+* Scheduler filters get called once per host for every instance being created,
+ so any latency they introduce is linear with the size of the system.
+
+* A small change in a utility function or a commonly used decorator can have a
+ large impacts on performance.
+
+* Calls which result in a database queries (whether direct or via conductor)
+ can have a profound impact on performance when called in critical sections of
+ the code.
+
+* Will the change include any locking, and if so what considerations are there
+ on holding the lock?
+
+Other deployer impact
+---------------------
+
+Discuss things that will affect how you deploy and configure OpenStack
+that have not already been mentioned, such as:
+
+* What config options are being added? Should they be more generic than
+ proposed (for example a flag that other hypervisor drivers might want to
+ implement as well)? Are the default values ones which will work well in
+ real deployments?
+
+* Is this a change that takes immediate effect after its merged, or is it
+ something that has to be explicitly enabled?
+
+* If this change is a new binary, how would it be deployed?
+
+* Please state anything that those doing continuous deployment, or those
+ upgrading from the previous release, need to be aware of. Also describe
+ any plans to deprecate configuration values or features. For example, if we
+ change the directory name that instances are stored in, how do we handle
+ instance directories created before the change landed? Do we move them? Do
+ we have a special case in the code? Do we assume that the operator will
+ recreate all the instances in their cloud?
+
+Developer impact
+----------------
+
+Discuss things that will affect other developers working on OpenStack,
+such as:
+
+* If the blueprint proposes a change to the driver API, discussion of how
+ other hypervisors would implement the feature is required.
+
+
+Implementation
+==============
+
+Assignee(s)
+-----------
+
+Who is leading the writing of the code? Or is this a blueprint where you're
+throwing it out there to see who picks it up?
+
+If more than one person is working on the implementation, please designate the
+primary author and contact.
+
+Primary assignee:
+ <launchpad-id or None>
+
+Other contributors:
+ <launchpad-id or None>
+
+Work Items
+----------
+
+Work items or tasks -- break the feature up into the things that need to be
+done to implement it. Those parts might end up being done by different people,
+but we're mostly trying to understand the timeline for implementation.
+
+
+Dependencies
+============
+
+* Include specific references to specs and/or blueprints in cyborg, or in other
+ projects, that this one either depends on or is related to.
+
+* If this requires functionality of another project that is not currently used
+ by Cyborg, document that fact.
+
+* Does this feature require any new library dependencies or code otherwise not
+ included in OpenStack? Or does it depend on a specific version of library?
+
+
+Testing
+=======
+
+Please discuss the important scenarios needed to test here, as well as
+specific edge cases we should be ensuring work correctly. For each
+scenario please specify if this requires specialized hardware, a full
+OpenStack environment, or can be simulated inside the Cyborg tree.
+
+Please discuss how the change will be tested. We especially want to know what
+tempest tests will be added. It is assumed that unit test coverage will be
+added so that doesn't need to be mentioned explicitly, but discussion of why
+you think unit tests are sufficient and we don't need to add more tempest
+tests would need to be included.
+
+Is this untestable in gate given current limitations (specific hardware /
+software configurations available)? If so, are there mitigation plans (3rd
+party testing, gate enhancements, etc).
+
+
+Documentation Impact
+====================
+
+Which audiences are affected most by this change, and which documentation
+titles on docs.openstack.org should be updated because of this change? Don't
+repeat details discussed above, but reference them here in the context of
+documentation for multiple audiences. For example, the Operations Guide targets
+cloud operators, and the End User Guide would need to be updated if the change
+offers a new feature available through the CLI or dashboard. If a config option
+changes or is deprecated, note here that the documentation needs to be updated
+to reflect this specification's change.
+
+References
+==========
+
+Please add any useful references here. You are not required to have any
+reference. Moreover, this specification should still make sense when your
+references are unavailable. Examples of what you could include are:
+
+* Links to mailing list or IRC discussions
+
+* Links to notes from a summit session
+
+* Links to relevant research, if appropriate
+
+* Related specifications as appropriate (e.g. if it's an EC2 thing, link the
+ EC2 docs)
+
+* Anything else you feel it is worthwhile to refer to
+
+
+History
+=======
+
+Optional section intended to be used each time the spec is updated to describe
+new design, API or any database schema updated. Useful to let reader understand
+what's happened along the time.
+
+.. list-table:: Revisions
+ :header-rows: 1
+
+ * - Release Name
+ - Description
+ * - Pike
+ - Introduced