Merge "Add documentation of testapi internship"

2 files changed, 203 insertions, 16 deletions

diff --git a/docs/devguide/index.rst b/docs/devguide/index.rst
index 42ad04451..eee013678 100644
--- a/docs/devguide/index.rst
+++ b/docs/devguide/index.rst
@@ -335,10 +335,6 @@ The API can described as follows. For detailed information, please go to
 
  Authentication: opnfv/api@opnfv
 
-Please notes that POST/DELETE/PUT operations for test or study purpose via
-swagger website is not allowed, because it will change the real data in
-the database.
-
 Version:
 
  +--------+--------------------------+-----------------------------------------+
@@ -503,17 +499,39 @@ Scenarios:
 
 
 The code of the API is hosted in the releng repository `[6]`_.
+The static documentation of the API can be found at `[17]`_.
 The test API has been dockerized and may be installed locally in your
 lab. See `[15]`_ for details.
 
 The deployment of the test API has been automated.
 A jenkins job manages:
   * the unit tests of the test api
-  * the cration of a new docker file
+  * the creation of a new docker file
   * the deployment of the new test api
   * the archive of the old test api
   * the backup of the Mongo DB
 
+Test API Authorization
+~~~~~~~~~~~~~~~~~~~~~~
+
+PUT/DELETE/POST operations of the testapi now require token based authorization. The token needs
+to be added in the request using a header 'X-Auth-Token' for access to the database.
+
+e.g::
+    headers['X-Auth-Token']
+
+The value of the header i.e the token can be accessed in the jenkins environment variable
+*TestApiToken*. The token value is added as a masked password.
+
+.. code-block:: python
+
+    headers['X-Auth-Token'] = os.environ.get('TestApiToken')
+
+The above example is in Python. Token based authentication has been added so that only ci pods
+jenkins job can have access to the database.
+
+Please note that currently token authorization is implemented but is not yet enabled.
+
   Automatic reporting
   ===================
 
@@ -960,6 +978,8 @@ _`[15]`: https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api
 
 _`[16]`: https://git.opnfv.org/cgit/releng/tree/utils/test/scripts/mongo_to_elasticsearch.py
 
+_`[17]`: http://artifacts.opnfv.org/releng/docs/testapi.html
+
 OPNFV main site: http://www.opnfv.org
 
 OPNFV functional test page: https://wiki.opnfv.org/opnfv_functional_testing
diff --git a/docs/internship/testapi_evolution/index.rst b/docs/internship/testapi_evolution/index.rst
index f2583e2f0..9cca9ebce 100644
--- a/docs/internship/testapi_evolution/index.rst
+++ b/docs/internship/testapi_evolution/index.rst
@@ -11,12 +11,17 @@ If not, see <http://creativecommons.org/licenses/by/4.0/>.
 Test API evolution
 ==================
 
-Author: Rohit Sakala
+Author: Sakala Venkata Krishna Rohit
 Mentors: S. Feng, J.Lausuch, M.Richomme
 
 Abstract
 ========
 
+The testapi is used by all the test opnfv projects to report results.
+It is also used to declare projects, test cases and labs. A major refactoring
+has been done in Colorado with the introduction of swagger. The testapi is defined in Functest
+developer guide. The purpose of this project is to add more features to the testapi that automate
+the tasks that are done manually now, though there are tasks other than automation.
 
 Version history
 ===============
@@ -25,46 +30,208 @@ Version history
 | **Date**   | **Ver.** | **Author**       | **Comment**            |
 |            |          |                  |                        |
 +------------+----------+------------------+------------------------+
-| 2016-??-?? | 0.0.1    | Morgan Richomme  | Beginning of the       |
+| 2016-11-14 | 0.0.1    | Morgan Richomme  | Beginning of the       |
 |            |          | (Orange)         | Internship             |
 +------------+----------+------------------+------------------------+
-
+| 2017-02-17 | 0.0.2    | S.V.K Rohit      | End of the Internship  |
+|            |          | (IIIT Hyderabad) |                        |
++------------+----------+------------------+------------------------+
 
 Overview:
 =========
 
-
-
+The internhip time period was from Nov 14th to Feb 17th. The project prosposal page is here `[1]`_.
+The intern project was assigned to Svk Rohit and was mentored by S. Feng, J.Lausuch, M.Richomme.
+The link to the patches submitted is `[2]`_. The internship was successfully completed and the
+documentation is as follows.
 
 Problem Statement:
 ------------------
 
+The problem statement could be divided into pending features that needed to be added into testapi
+repo. The following were to be accomplished within the internship time frame.
+
+* **Add verification jenkins job for the testapi code**
+    The purpose of this job is to verify whehter the unit tests are successful or not with the
+    inclusion of the patchset submitted.
+
+* **Automatic update of opnfv/testapi docker image**
+    The docker image of testapi is hosted in the opnfv docker hub. To ensure that the testapi image
+    is always updated with the repository, automatic updation of the image is necessary and a job
+    is triggered whenever a new patch gets merged.
+
+* **Automation deployment of testresults.opnfv.org/test/ website**
+    In the same manner as the docker image of testapi is updated, the testapi website needs to be
+    in sync with the repository code. So, a job has been added to the opnfv jenkins ci for the
+    updation of the testresults website.
 
+* **Generate static documentation of testapi calls**
+    The purpose of this is to give an static/offline view of testapi. If someone wants to have a
+    look at the Restful apis of testapi, he/she does't need to go to the website, he can download
+    a html page and view it anytime.
 
-Curation Phase
---------------
+* **Backup MongoDB of testapi**
+    The mongoDB needs to be backed up every week. Till now it was done manually, but due to this
+    internship, it is now automated using a jenkins job.
 
+* **Add token based authorization to the testapi calls**
+    The token based authorization was implemented to ensure that only ci_pods could access the
+    database. Authentication has been added to only delete/put/post requests.
 
+Curation Phase:
+---------------
 
+The curation phase was the first 3 to 4 weeks of the internship. This phase was to get familiar
+with the testapi code and functionality and propose the solutions/tools for the tasks mentioned
+above. Swagger codegen was choosen out of the four tools proposed `[3]`_ for generating static
+documentaion.
 
+Also, specific amount of time was spent on the script flow of the jenkins jobs. The automatic
+deployment task involves accessing a remote server from inside the jenkins build. The deployment
+had to be done only after the docker image update is done. For these constraints to satisfy, a
+multijob jenkins job was choosen instead of a freestyle job.
+
+Important Links:
+----------------
+
+* MongoDB Backup Link                 - `[4]`_
+* Static Documentation                - `[5]`_
+* TestAPI Token addition to ci_pods   - `[6]`_
 
 Schedule:
 =========
 
-
+The progress and completion of the tasks is described in the below table.
 
 +--------------------------+------------------------------------------+
 | **Date**                 | **Comment**                              |
 |                          |                                          |
 +--------------------------+------------------------------------------+
-| December  - January      | ........                                 |
+| Nov 14th - Dec 31st      | Understand Testapi code and the          |
+|                          | requirements.                            |
++--------------------------+------------------------------------------+
+| Jan 1st  - Jan 7th       | Add jenkins job to create static         |
+|                          | documentation and write build scripts.   |
++--------------------------+------------------------------------------+
+| Jan 8th  - Jan 21st      | Add verification jenkins job for unit    |
+|                          | tests.                                   |
++--------------------------+------------------------------------------+
+| Jan 22nd - Jan 28th      | Add jenkins job for mongodb backup       |
+|                          |                                          |
++--------------------------+------------------------------------------+
+| Jan 29th - Feb 11th      | Enable automatic deployment of           |
+|                          | testresults.opnfv.org/test/              |
 +--------------------------+------------------------------------------+
-| January  - february      | ........                                 |
+| Feb 12th - Feb 17th      | Add token based authentication           |
+|                          |                                          |
 +--------------------------+------------------------------------------+
 
+FAQ's
+=====
+
+This section lists the problems that I have faced and the understanding that I have acquired during
+the internship. This section may help other developers in solving any errors casused because of the
+code written as a part of this internship.
+
+
+Test Api
+--------
+
+What is the difference between defining data_file as "/etc/.." and "etc/.." in setup.cfg ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If in the setup.cfg, it is defined as
+
+[files]
+data_files =
+etc/a.conf = etc/a.conf.sample
+
+then it ends up installed in the /usr/etc/. With this configuration, it would be installed
+correctly within a venv. but when it is defined as
+
+[files]
+data_files =
+/etc/a.conf = etc/a.conf.sample
+
+then it ends up installed on the root of the filesystem instead of properly be installed within the
+venv.
+
+Which attribute does swagger-codegen uses as the title in the generation of document generation ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It uses the nickname of the api call in swagger as the title in the generation of the document
+generation.
+
+Does swagger-codegen take more than one yaml file as input ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+No, swagger-codegen only takes one yaml file as input to its jar file. If there more than one yaml
+file, one needs to merge them and give it as an input keeping mind the swagger specs.
+
+
+Jenkins & JJB
+-------------
+
+Which scm macro is used for verification jenkins jobs ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are two macros for scm one is git-scm and other git-scm-gerrit. git-scm-gerrit is used for
+verification jenkins job.
+
+Does the virtualenv created in one build script exists in other build scripts too ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+No, the virtualenv created in one build script only exists in that build script/shell.
+
+What parameters are needed for the scm macros ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Project and Branch are the two parameters needed for scm macros.
+
+What is the directory inside the jenkins build ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The directory of the jenkins build is the directory of the repo. `ls $WORKSPACE` command will give
+you all the contents of the directory.
+
+How to include a bash script in jenkins job yaml file ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An example might be apt here as an answer.
+
+builders:
+    - shell:
+        !include-raw: include-raw001-hello-world.sh
+
+
+How do you make a build server run on a specific machine ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It can be done by defining a label parameter 'SLAVE_LABEL' or in OPNFV , there are macros for each
+server, one can use those parameter macros.
+Ex: opnfv-build-defaults. Note, if we use macro, then no need to define GIT_BASE, but if one uses
+SLAVE_LABEL, one needs to define a parameter GIT_BASE. This is because macro already has GIT_BASE
+defined.
+
+What job style should be used when there is a situation like one build should trigger other builds
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+or when different build scripts need to be run on different machines ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+MultiJob style should be used as it has phases where each phase can be taken as a build scipt and
+can have its own parameters by which one can define the SLAVE_LABEL parameter.
 
 References:
 ===========
 
-.. _`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+testapi+evolution
+_`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+testapi+evolution
+
+_`[2]` : https://gerrit.opnfv.org/gerrit/#/q/status:merged+owner:%22Rohit+Sakala+%253Crohitsakala%2540gmail.com%253E%22
+
+_`[3]` : https://docs.google.com/document/d/1jWwVZ1ZpKgKcOS_zSz2KzX1nwg4BXxzBxcwkesl7krw/edit?usp=sharing
+
+_`[4]` : http://artifacts.opnfv.org/testapibackup.html
+
+_`[5]` : http://artifacts.opnfv.org/releng/docs/testapi.html
 
+_`[6]` : http://artifacts.opnfv.org/functest/review/26047/devguide/index.html#test-api-authorization