From 9b2ba68b75ecd50600e6f375aeb4ea410cec0128 Mon Sep 17 00:00:00 2001 From: Shrenik Date: Wed, 13 Sep 2017 00:49:05 +0530 Subject: Adds StorPerf developer guide The wiki for Development Environment has been converted in .rst format. JIRA: STORPERF-201 Change-Id: I4e2bc792a8704e4f933f0ea61533c1bf5955c64d Signed-off-by: Shrenik --- docs/dev/gerrit.rst | 203 ++++++++++++++++++++++++++++ docs/dev/ide.rst | 110 +++++++++++++++ docs/dev/index.rst | 18 +++ docs/dev/initial.rst | 42 ++++++ docs/dev/unit_tests.rst | 88 ++++++++++++ docs/images/Access_denied.png | Bin 0 -> 38465 bytes docs/images/Add_git.png | Bin 0 -> 10665 bytes docs/images/Browse.png | Bin 0 -> 8526 bytes docs/images/CLA_agreement.png | Bin 0 -> 96078 bytes docs/images/Code_analysis.jpeg | Bin 0 -> 64860 bytes docs/images/Code_formatter.jpeg | Bin 0 -> 105583 bytes docs/images/Forced_Builtins.jpeg | Bin 0 -> 49854 bytes docs/images/Git_Selection.png | Bin 0 -> 12061 bytes docs/images/Import_Project.png | Bin 0 -> 18579 bytes docs/images/Local_Repo.png | Bin 0 -> 13494 bytes docs/images/PyDev_Interpreter.jpeg | Bin 0 -> 67063 bytes docs/images/PyDev_Interpreter_Folders.jpeg | Bin 0 -> 96283 bytes docs/images/PyDev_Interpreters_List.jpeg | Bin 0 -> 90878 bytes docs/images/PyDev_New_Interpreter.jpeg | Bin 0 -> 29977 bytes docs/images/PyDev_StorPerf_Interpreter.jpeg | Bin 0 -> 65829 bytes docs/images/Python_Interpreters.jpeg | Bin 0 -> 157750 bytes docs/images/Save_Actions.jpeg | Bin 0 -> 127033 bytes docs/images/StorPerf_Tests-Arguments.jpeg | Bin 0 -> 79609 bytes docs/images/StorPerf_Tests-Console.jpeg | Bin 0 -> 37284 bytes docs/images/StorPerf_Tests-Interpreter.jpeg | Bin 0 -> 30423 bytes docs/images/StorPerf_Tests-Main.jpeg | Bin 0 -> 124481 bytes docs/images/StorPerf_Tests-PyUnit.jpeg | Bin 0 -> 114504 bytes docs/images/Unused_imports.jpeg | Bin 0 -> 116906 bytes docs/images/add_reviewers.png | Bin 0 -> 332659 bytes docs/images/amend_commit.png | Bin 0 -> 47179 bytes docs/images/gerrit_bot.png | Bin 0 -> 12256 bytes docs/images/git_review.png | Bin 0 -> 27771 bytes docs/images/git_review_2.png | Bin 0 -> 269817 bytes docs/images/git_status.png | Bin 0 -> 39525 bytes docs/images/git_status_2.png | Bin 0 -> 24436 bytes docs/index.rst | 11 ++ 36 files changed, 472 insertions(+) create mode 100644 docs/dev/gerrit.rst create mode 100644 docs/dev/ide.rst create mode 100644 docs/dev/index.rst create mode 100644 docs/dev/initial.rst create mode 100644 docs/dev/unit_tests.rst create mode 100644 docs/images/Access_denied.png create mode 100644 docs/images/Add_git.png create mode 100644 docs/images/Browse.png create mode 100644 docs/images/CLA_agreement.png create mode 100644 docs/images/Code_analysis.jpeg create mode 100644 docs/images/Code_formatter.jpeg create mode 100644 docs/images/Forced_Builtins.jpeg create mode 100644 docs/images/Git_Selection.png create mode 100644 docs/images/Import_Project.png create mode 100644 docs/images/Local_Repo.png create mode 100644 docs/images/PyDev_Interpreter.jpeg create mode 100644 docs/images/PyDev_Interpreter_Folders.jpeg create mode 100644 docs/images/PyDev_Interpreters_List.jpeg create mode 100644 docs/images/PyDev_New_Interpreter.jpeg create mode 100644 docs/images/PyDev_StorPerf_Interpreter.jpeg create mode 100644 docs/images/Python_Interpreters.jpeg create mode 100644 docs/images/Save_Actions.jpeg create mode 100644 docs/images/StorPerf_Tests-Arguments.jpeg create mode 100644 docs/images/StorPerf_Tests-Console.jpeg create mode 100644 docs/images/StorPerf_Tests-Interpreter.jpeg create mode 100644 docs/images/StorPerf_Tests-Main.jpeg create mode 100644 docs/images/StorPerf_Tests-PyUnit.jpeg create mode 100644 docs/images/Unused_imports.jpeg create mode 100644 docs/images/add_reviewers.png create mode 100644 docs/images/amend_commit.png create mode 100644 docs/images/gerrit_bot.png create mode 100644 docs/images/git_review.png create mode 100644 docs/images/git_review_2.png create mode 100644 docs/images/git_status.png create mode 100644 docs/images/git_status_2.png diff --git a/docs/dev/gerrit.rst b/docs/dev/gerrit.rst new file mode 100644 index 0000000..b227d21 --- /dev/null +++ b/docs/dev/gerrit.rst @@ -0,0 +1,203 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Dell EMC and others. + +====== +Gerrit +====== + +Installing and configuring Git and Git-Review is necessary in order to follow +this guide. The `Getting Started `_ page will provide you with some help for that. + +Committing the code with Gerrit +=============================== + +* Open a terminal window and set the project's directory to the working + directory using the cd command. In this case "/home/tim/OPNFV/storperf" is + the path to the StorPerf project folder on my computer. Replace this with the + path of your own project. + +.. code-block:: bash + + cd /home/tim/OPNFV/storperf + +* Start a new topic for your change. + +.. code-block:: bash + + git checkout -b TOPIC-BRANCH + +* Tell Git which files you would like to take into account for the next commit. + This is called 'staging' the files, by placing them into the staging area, + using the 'git add' command (or the synonym 'git stage' command). + +.. code-block:: bash + + git add storperf/utilities/math.py + git add storperf/tests/utilities/math.py + ... + +* Alternatively, you can choose to stage all files that have been modified + (that is the files you have worked on) since the last time you generated a + commit, by using the -a argument. + +.. code-block:: bash + + git add -a + +* Git won't let you push (upload) any code to Gerrit if you haven't pulled the + latest changes first. So the next step is to pull (download) the latest + changes made to the project by other collaborators using the 'pull' command. + +.. code-block:: bash + + git pull + +* Now that you have the latest version of the project and you have staged the + files you wish to push, it is time to actually commit your work to your local + Git repository. + +.. code-block:: bash + + git commit --signoff -m "Title of change + + Test of change that describes in high level what + was done. There is a lot of documentation in code + so you do not need to repeat it here. + + JIRA: STORPERF-54" + +The message that is required for the commit should follow a specific set of +rules. This practice allows to standardize the description messages attached to +the commits, and eventually navigate among the latter more easily. This +`document `_ happened to be very clear +and useful to get started with that. + + +Pushing the code to Git for review +================================== + +* Now that the code has been comitted into your local Git repository the + following step is to push it online to Gerrit for it to be reviewed. The + command we will use is 'git review'. + +.. code-block:: bash + + git review + +* This will automatically push your local commit into Gerrit, and the command + should get back to you with a Gerrit URL that looks like this : + +.. image:: ../images/git_review.png + +* The OPNFV-Gerrit-Bot in #opnfv-storperf IRC channel will send a message with + the URL as well. + +.. image:: ../images/gerrit_bot.png + +* Copy/Paste the URL into a web browser to get to the Gerrit code review you + have just generated, and click the 'add' button to add reviewers to review + your changes : + +.. image:: ../images/add_reviewers.png + +.. note:: + + Check out this `section `_ if the git + review command returns to you with an "access denied" error. + + +Fetching a Git review +===================== + +If you want to collaborate with another developer, you can fetch their review +by the Gerrit change id (which is part of the URL, and listed in the top left +as Change NNNNN). + +.. code-block:: bash + + git review -d 16213 + +would download the patchset for change 16213. If there were a topic branch +associated with it, it would switch you to that branch, allowing you to look at +different patch sets locally at the same time without conflicts. + + +Modifying the code under review in Gerrit +========================================= + +At the same time the code is being reviewed in Gerrit, you may need to edit it +to make some changes and then send it back for review. The following steps go +through the procedure. + +* Once you have modified/edited your code files under your IDE, you will have + to stage them. The 'status' command is very helpful at this point as it + provides an overview of Git's current state. + +.. code-block:: bash + + git status + +.. image:: ../images/git_status.png + +* The output of the command provides us with the files that have been modified + after the latest commit (in this case I modified storperf/tests/utilities/ + math.py and storperf/utilities/math.py). + +* We can now stage the files that have been modified as part of the Gerrit code + review edition/modification/improvement : + +.. code-block:: bash + + git add storperf/tests/utilities/math.py + git add storperf/utilities/math.py + +* The 'git status' command should take this into consideration : + +.. image:: ../images/git_status_2.png + +* It is now time to commit the newly modified files, but the objective here is + not to create a new commit, we simply want to inject the new changes into the + previous commit. We can achieve that with the '--amend' option on the + 'commit' command : + +.. code-block:: bash + + git commit --amend + +.. image:: ../images/amend_commit.png + +* If the commit was successful, the 'status' command should not return the + updated files as about to be committed. + +* The final step consists in pushing the newly modified commit to Gerrit. + +.. code-block:: bash + + git review + +.. image:: ../images/git_review_2.png + +The Gerrit code review should be updated, which results in a 'patch set 2' +notification appearing in the history log. 'patch set 1' being the original +code review proposition. + + +If Gerrit upload is denied +========================== + +The 'git review' command might return to you with an "access denied" error that +looks like this : + +.. image:: ../images/Access_denied.png + +In this case, you need to make sure your Gerrit account has been added as a +member of the StorPerf contributors group : ldap/opnfv-gerrit-storperf- +contributors. You also want to check that have signed the CLA (Contributor +License Agreement), if not you can sign it in the "Agreements" section of your +Gerrit account : + +.. image:: ../images/CLA_agreement.png + diff --git a/docs/dev/ide.rst b/docs/dev/ide.rst new file mode 100644 index 0000000..3af4b6c --- /dev/null +++ b/docs/dev/ide.rst @@ -0,0 +1,110 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Dell EMC and others. + +=== +IDE +=== + +While PyCharm as an excellent IDE, some aspects of it require licensing, and +the PyDev Plugin for Eclipse (packaged as LiClipse) is fully open source +(although donations are welcome). Therefore this section focuses on using +LiClipse for StorPerf development. + + +Download +============ + +.. code-block:: bash + + http://www.liclipse.com/download.html + + +Storperf virtualenv Interpretor +================================= + +Setting up interpreter under PyDev (LiClipse): + +* Go to Project -> Properties, PyDev Interpreter: + +.. image:: ../images/PyDev_Interpreter.jpeg + +* Click to configure an interpreter not listed. + +.. image:: ../images/PyDev_Interpreters_List.jpeg + +* Click New, and create a new interpreter called StorPerf that points to your + Virtual Env. + +.. image:: ../images/PyDev_New_Interpreter.jpeg + +* You should get a pop up similar to: + +.. image:: ../images/PyDev_Interpreter_Folders.jpeg + +* And then you can change the Interpreter to StorPerf. + +.. image:: ../images/PyDev_StorPerf_Interpreter.jpeg + + +Code Formatting +=============== + +Pep8 and Flake8 rule. These are part of the Gerrit checks and I'm going to +start enforcing style guidelines soon. + +* Go to Window -> Preferences, under PyDev, Editor, Code Style, Code Formatter + and select autopep8.py for code formatting. + +.. image:: ../images/Code_formatter.jpeg + +* Next, under Save Actions, enable "Auto-format editor contents before saving", + and "Sort imports on save". + +.. image:: ../images/Save_Actions.jpeg + +* And under Imports, select Delete unused imports. + +.. image:: ../images/Unused_imports.jpeg + +* Go to PyDev -> Editor -> Code Analysis and under the pycodestye.py (pep8), + select Pep8 as Error. This flag highlight badly formatted lines as errors. + These must be fixed before Jenkins will +1 any review. + +.. image:: ../images/Code_analysis.jpeg + + +Import Storperf as Git Project +============================== + +I prefer to do the git clone from the command line, and then import that as a +local project in LiClipse. + +* From the menu: File -> Import Project + +.. image:: ../images/Import_Project.png + +| + +.. image:: ../images/Local_Repo.png + +| + +.. image:: ../images/Add_git.png + +| + +* Browse to the directory where you cloned StorPerf + +.. image:: ../images/Browse.png + +| + +* You should now have storperf as a valid local git repo: + +.. image:: ../images/Git_Selection.png + +| + +* Choose Import as general project + diff --git a/docs/dev/index.rst b/docs/dev/index.rst new file mode 100644 index 0000000..48000cf --- /dev/null +++ b/docs/dev/index.rst @@ -0,0 +1,18 @@ +.. _storperf-devguide: + +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Dell EMC and others. + +====================== +StorPerf Dev Guide +====================== + +.. toctree:: + :maxdepth: 2 + + initial.rst + ide.rst + unit_tests.rst + gerrit.rst diff --git a/docs/dev/initial.rst b/docs/dev/initial.rst new file mode 100644 index 0000000..04b1c45 --- /dev/null +++ b/docs/dev/initial.rst @@ -0,0 +1,42 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Dell EMC and others. + +================ +Initial Set up +================ + +Getting the Code +================ + +Replace your LFID with your actual Linux Foundation ID. + +.. code-block:: bash + + git clone ssh://YourLFID@gerrit.opnfv.org:29418/storperf + + +Virtual Environment +======================= +It is preferred to use virtualenv for Python dependencies. This way it is known +exactly what libraries are needed, and can restart from a clean state at any +time to ensure any library is not missing. Simply running the script: + +.. code-block:: bash + + ci/verify.sh + +from inside the storperf directory will automatically create a virtualenv in +the home directory called 'storperf_venv'. This will be used as the Python +interpreter for the IDE. + + +Docker Version +======================= +In order to run the full set of StorPerf services, docker and docker-compose +are required to be installed. This requires docker 17.05 at a minimum. + +.. code-block:: bash + + https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/ + diff --git a/docs/dev/unit_tests.rst b/docs/dev/unit_tests.rst new file mode 100644 index 0000000..98ed3ce --- /dev/null +++ b/docs/dev/unit_tests.rst @@ -0,0 +1,88 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Dell EMC and others. + +========== +Unit Tests +========== + +Running from CLI +================ + +You technically already did when you ran: + +.. code-block:: bash + + ci/verify.sh + +The shortcut to running the unit tests again from the command line is: + +.. code-block:: bash + + source ~/storperf_venv/bin/activate + nosetests --with-xunit \ + --with-coverage \ + --cover-package=storperf\ + --cover-xml \ + storperf + +.. note:: + + You must be in the top level storperf directory in order to run the tests. + + +Set up under LiClipse +===================== + +Running the tests: + +Right click on the tests folder and select Run as Python Unit Test. Chances +are, you'll get: + +.. code-block:: bash + + Traceback (most recent call last): + File "/home/mark/Documents/EMC/git/opnfv/storperf/storperf/tests/storperf_master_test.py", line 24, in setUp + self.storperf = StorPerfMaster() + File "/home/mark/Documents/EMC/git/opnfv/storperf/storperf/storperf_master.py", line 38, in __init__ + template_file = open("storperf/resources/hot/agent-group.yaml") + IOError: [Errno 2] No such file or directory: 'storperf/resources/hot/agent-group.yaml' + +This means we need to set the working directory of the run configuration. + +* Under the menu: Run -> Run Configurations: + +.. image:: ../images/StorPerf_Tests-Main.jpeg + +* Go to the Arguments tab and change the radio button for Working Directory to + "Default" + +.. image:: ../images/StorPerf_Tests-Arguments.jpeg + +* And on interpreter tab, change the interpreter to StorPerf: + +.. image:: ../images/StorPerf_Tests-Interpreter.jpeg + +* Click Apply. From now on, the run should be clean: + +.. image:: ../images/StorPerf_Tests-Console.jpeg + +| + +.. image:: ../images/StorPerf_Tests-PyUnit.jpeg + + +Adding builtins +=============== + +For some reason, sqlite needs to be added as a builtin. + +* Go to Window -> Preferences, PyDev > Interpreters > Python Interpreter and + select the StorPerf interpreter: + +.. image:: ../images/Python_Interpreters.jpeg + +* Go to the Forced Builtins tab, click New and add sqlite3. + +.. image:: ../images/Forced_Builtins.jpeg + diff --git a/docs/images/Access_denied.png b/docs/images/Access_denied.png new file mode 100644 index 0000000..6a54c0b Binary files /dev/null and b/docs/images/Access_denied.png differ diff --git a/docs/images/Add_git.png b/docs/images/Add_git.png new file mode 100644 index 0000000..d7459fc Binary files /dev/null and b/docs/images/Add_git.png differ diff --git a/docs/images/Browse.png b/docs/images/Browse.png new file mode 100644 index 0000000..81c4fcd Binary files /dev/null and b/docs/images/Browse.png differ diff --git a/docs/images/CLA_agreement.png b/docs/images/CLA_agreement.png new file mode 100644 index 0000000..bf9574d Binary files /dev/null and b/docs/images/CLA_agreement.png differ diff --git a/docs/images/Code_analysis.jpeg b/docs/images/Code_analysis.jpeg new file mode 100644 index 0000000..1299636 Binary files /dev/null and b/docs/images/Code_analysis.jpeg differ diff --git a/docs/images/Code_formatter.jpeg b/docs/images/Code_formatter.jpeg new file mode 100644 index 0000000..84d9a49 Binary files /dev/null and b/docs/images/Code_formatter.jpeg differ diff --git a/docs/images/Forced_Builtins.jpeg b/docs/images/Forced_Builtins.jpeg new file mode 100644 index 0000000..95870dd Binary files /dev/null and b/docs/images/Forced_Builtins.jpeg differ diff --git a/docs/images/Git_Selection.png b/docs/images/Git_Selection.png new file mode 100644 index 0000000..4ca0e42 Binary files /dev/null and b/docs/images/Git_Selection.png differ diff --git a/docs/images/Import_Project.png b/docs/images/Import_Project.png new file mode 100644 index 0000000..9d7d46a Binary files /dev/null and b/docs/images/Import_Project.png differ diff --git a/docs/images/Local_Repo.png b/docs/images/Local_Repo.png new file mode 100644 index 0000000..3c917a3 Binary files /dev/null and b/docs/images/Local_Repo.png differ diff --git a/docs/images/PyDev_Interpreter.jpeg b/docs/images/PyDev_Interpreter.jpeg new file mode 100644 index 0000000..17a9e8b Binary files /dev/null and b/docs/images/PyDev_Interpreter.jpeg differ diff --git a/docs/images/PyDev_Interpreter_Folders.jpeg b/docs/images/PyDev_Interpreter_Folders.jpeg new file mode 100644 index 0000000..bd13e5f Binary files /dev/null and b/docs/images/PyDev_Interpreter_Folders.jpeg differ diff --git a/docs/images/PyDev_Interpreters_List.jpeg b/docs/images/PyDev_Interpreters_List.jpeg new file mode 100644 index 0000000..31a2f90 Binary files /dev/null and b/docs/images/PyDev_Interpreters_List.jpeg differ diff --git a/docs/images/PyDev_New_Interpreter.jpeg b/docs/images/PyDev_New_Interpreter.jpeg new file mode 100644 index 0000000..ad479b9 Binary files /dev/null and b/docs/images/PyDev_New_Interpreter.jpeg differ diff --git a/docs/images/PyDev_StorPerf_Interpreter.jpeg b/docs/images/PyDev_StorPerf_Interpreter.jpeg new file mode 100644 index 0000000..f2c6cc4 Binary files /dev/null and b/docs/images/PyDev_StorPerf_Interpreter.jpeg differ diff --git a/docs/images/Python_Interpreters.jpeg b/docs/images/Python_Interpreters.jpeg new file mode 100644 index 0000000..37d4159 Binary files /dev/null and b/docs/images/Python_Interpreters.jpeg differ diff --git a/docs/images/Save_Actions.jpeg b/docs/images/Save_Actions.jpeg new file mode 100644 index 0000000..cb261e6 Binary files /dev/null and b/docs/images/Save_Actions.jpeg differ diff --git a/docs/images/StorPerf_Tests-Arguments.jpeg b/docs/images/StorPerf_Tests-Arguments.jpeg new file mode 100644 index 0000000..be2d3fb Binary files /dev/null and b/docs/images/StorPerf_Tests-Arguments.jpeg differ diff --git a/docs/images/StorPerf_Tests-Console.jpeg b/docs/images/StorPerf_Tests-Console.jpeg new file mode 100644 index 0000000..85610a5 Binary files /dev/null and b/docs/images/StorPerf_Tests-Console.jpeg differ diff --git a/docs/images/StorPerf_Tests-Interpreter.jpeg b/docs/images/StorPerf_Tests-Interpreter.jpeg new file mode 100644 index 0000000..7882e75 Binary files /dev/null and b/docs/images/StorPerf_Tests-Interpreter.jpeg differ diff --git a/docs/images/StorPerf_Tests-Main.jpeg b/docs/images/StorPerf_Tests-Main.jpeg new file mode 100644 index 0000000..228ebad Binary files /dev/null and b/docs/images/StorPerf_Tests-Main.jpeg differ diff --git a/docs/images/StorPerf_Tests-PyUnit.jpeg b/docs/images/StorPerf_Tests-PyUnit.jpeg new file mode 100644 index 0000000..acfcca8 Binary files /dev/null and b/docs/images/StorPerf_Tests-PyUnit.jpeg differ diff --git a/docs/images/Unused_imports.jpeg b/docs/images/Unused_imports.jpeg new file mode 100644 index 0000000..d4619cd Binary files /dev/null and b/docs/images/Unused_imports.jpeg differ diff --git a/docs/images/add_reviewers.png b/docs/images/add_reviewers.png new file mode 100644 index 0000000..fdf6016 Binary files /dev/null and b/docs/images/add_reviewers.png differ diff --git a/docs/images/amend_commit.png b/docs/images/amend_commit.png new file mode 100644 index 0000000..3630ed7 Binary files /dev/null and b/docs/images/amend_commit.png differ diff --git a/docs/images/gerrit_bot.png b/docs/images/gerrit_bot.png new file mode 100644 index 0000000..bcc5186 Binary files /dev/null and b/docs/images/gerrit_bot.png differ diff --git a/docs/images/git_review.png b/docs/images/git_review.png new file mode 100644 index 0000000..4efd3c7 Binary files /dev/null and b/docs/images/git_review.png differ diff --git a/docs/images/git_review_2.png b/docs/images/git_review_2.png new file mode 100644 index 0000000..a4838f0 Binary files /dev/null and b/docs/images/git_review_2.png differ diff --git a/docs/images/git_status.png b/docs/images/git_status.png new file mode 100644 index 0000000..8fd4898 Binary files /dev/null and b/docs/images/git_status.png differ diff --git a/docs/images/git_status_2.png b/docs/images/git_status_2.png new file mode 100644 index 0000000..1331c77 Binary files /dev/null and b/docs/images/git_status_2.png differ diff --git a/docs/index.rst b/docs/index.rst index 9729a44..ae9802d 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -37,6 +37,17 @@ StorPerf User Guide ./testing/user/introduction.rst ./testing/user/test-usage.rst +****************************** +StorPerf Dev Guide +****************************** + +.. toctree:: + :caption: StorPerf Dev Guide + :maxdepth: 5 + :numbered: 5 + + ./dev/index.rst + Indices ======= -- cgit 1.2.3-korg