Merge "update builder script and how-to-use-docs"

8 files changed, 281 insertions, 132 deletions

diff --git a/.gitignore b/.gitignore
index 1377554eb..c23a0d69f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1 +1,4 @@
 *.swp
+/build/
+/output/
+/releng/
diff --git a/docs/etc/conf.py b/docs/etc/conf.py
index f42b11055..147c50b7d 100644
--- a/docs/etc/conf.py
+++ b/docs/etc/conf.py
@@ -1,15 +1,13 @@
 '''
-Base configuration file for sphinx-build.
+Base configuration file for building OPNFV docs
 
 You can override this configuration by putting 'conf.py' in the document
-directory (e.g. how-to-use-docs/conf.py). If there is no 'conf.py' in the
-document directory, this file will be copied to that directory before the
-document builder jobs in 'opnfv-docs-verify' and 'opnfv-docs-merge'.
-The logo image (opnfv-logo.png) will be also copied from
-docs/etc/opnfv-logo.png during the build jobs.
-'''
+directory (e.g. docs/how-to-use-docs/conf.py). If there is no 'conf.py'
+in the document directory, this file will be copied to that directory
+before the document builder jobs ('opnfv-docs-verify' and 'opnfv-docs-merge').
 
-import datetime
+See https://wiki.opnfv.org/documentation/tools .
+'''
 
 needs_sphinx = '1.3'
 master_doc = 'index'
@@ -19,9 +17,5 @@ html_use_index = False
 numfig = True
 html_logo = 'opnfv-logo.png'
 
-latex_elements = {'printindex': ''}
+latex_domain_indices = False
 latex_logo = 'opnfv-logo.png'
-
-copyright = u'%s, OPNFV' % datetime.date.today().year
-version = u'1.0.0'
-release = u'1.0.0'
diff --git a/docs/how-to-use-docs/documentation-example.rst b/docs/how-to-use-docs/documentation-example.rst
index e18ce598b..5fc2b1420 100644
--- a/docs/how-to-use-docs/documentation-example.rst
+++ b/docs/how-to-use-docs/documentation-example.rst
@@ -1,70 +1,172 @@
+==================================================
 How to create documentation for your OPNFV project
 ==================================================
 
-this is the directory structure of the docs/ directory that can be found in the root of your project directory
+Directory Structure
+===================
+
+This is the directory structure of the docs/ directory which have to be placed
+in the root of your project directory.
 
 .. code-block:: bash
 
-    ./etc
-    ./etc/opnfv-logo.png
-    ./etc/conf.py
-    ./how-to-use-docs
     ./how-to-use-docs/documentation-example.rst
     ./how-to-use-docs/index.rst
 
-To create your own documentation, Create any number of directories (depending on your need) and place in each of them an index.rst.
-This index file must refence your other rst files.
+To create your own document, create any number of directories (depending
+on your need, e.g. manual) under the docs/ and place an index.rst in each
+directories.
+The depth of all directory should be one, so that you can make sure that
+all directory names are unique. If you want to have set of all documents in
+your repo, create new ``docs/all/index.rst`` and list document links in OPNFV
+artifact server (artifact.opnfv.org) instead of including all other rst files
+or having ``docs/index.rst``, in order to avoid having duplicated contents in
+your documents.
+
+Note:
+You may have "docs/how-to-use-docs/" in you project repo. You can delete it,
+since it is sample and master version is stored in releng repo.
+
+Index File
+==========
+
+This index file must refence your other rst files in that directory.
+
+Here is an example index.rst :
+
+.. code-block:: bash
+
+    *******************
+    Documentation Title
+    *******************
+
+    .. toctree::
+       :numbered:
+       :maxdepth: 2
+
+       documentation-example.rst
 
-* Here is an example index.rst
+Source Files
+============
+
+Document source files have to be written in reStructuredText format (rst).
+Each file would be build as an html page and a chapter in PDF.
+
+Here is an example source rst file :
 
 .. code-block:: bash
 
-  =====================
-  Example Documentation
-  =====================
+    =============
+    Chapter Title
+    =============
 
-  Contents:
+    Section Title
+    =============
 
-  .. toctree::
-     :numbered:
-     :maxdepth: 4
+    Hello!
 
-     documentation-example.rst
+Writing RST Markdown
+====================
 
-  ..
-      Leave these at the bottom of 'index.rst' file
+See http://sphinx-doc.org/rest.html .
 
-  Revision: _sha1_
+You can add dedicated contents by using 'only' directive with build type
+('html' and 'pdf') for OPNFV document
 
-  Build date: |today|
+Example :
 
+.. code-block:: bash
 
-The Sphinx Build
-================
+    .. only:: html
+        This line will be shown only in html version.
 
-When you push documentation changes to gerrit a jenkins job will create html documentation.
+Configuration
+=============
 
-* Verify Jobs
+If you need to change the default configuration for document build, create
+new conf.py in the document directory (e.g. 'docs/how-to-use-docs/conf.py')
+that will be used in build process instead of default for OPNFV document
+build. The OPNFV default configuration can be found in releng repo
+(see `conf.py`_).
 
-For verify jobs a link to the documentation will show up as a comment in gerrit for you to see the result.
+.. _conf.py:
+    https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=blob;f=docs/etc/conf.py;
 
-* Merge jobs
+In the build process, the following parameters are automatically added if they
+are not set in the conf.py .
 
-Once you are happy with the look of your documentation you can submit the patchset the merge job will
-copy the output of each documentation directory to http://artifacts.opnfv.org/$project/docs/$name_of_your_folder/index.html
+* **release**, **version** : ``git last tag name`` (``git last commit hash``)
+* **project** : ``git repo name``
+* **copyright** : ``year``, OPNFV
+* **latex_documents** (set of pdf configuration) :
+  [('index', '``document directory name``.tex',
+  '``document title in index.rst``', 'OPNFV', 'manual'),]
 
-Here are some quick examples of how to use rst markup
+See http://sphinx-doc.org/config.html to learn sphinx configuration.
 
-This is a headline::
+Note: you can leave the file path for OPNFV logo image which will be prepared
+before each document build.
 
-  here is some code, note that it is indented
+Versioning
+==========
 
-links are easy to add: Here is a link to sphinx, the tool that we are using to generate documetation http://sphinx-doc.org/
+The relevant release and version information will be added to your documents
+by using tags from your project's git repository.
+The tags will be applied by Releng Project.
 
-* Bulleted Items
+Testing
+=======
 
-  **this will be bold**
+You can test document build in your laptop by using build script which is
+used in document build jobs:
 
 .. code-block:: bash
 
-  echo "Heres is a code block with bash syntax highlighting"
+    $ cd /loacal/repo/path/to/project
+    $ git clone ssh://gerrit.opnfv.org:29418/releng
+    $ ./releng/utils/docs-build.sh
+
+Then, you can see docs in output directory if build succeeded.
+
+This script will generate files in 'build' and 'output'. You should consider
+to add the following entries in '.gitignore' file, so that git can ignore
+built files.
+
+.. code-block:: bash
+
+    /build/
+    /output/
+    /releng/
+
+Verify Jobs
+===========
+
+The verify job name is **opnfv-docs-verify**.
+
+When you send document changes to gerrit, jenkins will create your documents
+in HTML and PDF formats to verify that new document can be built successfully.
+Please check the jenkins log and artifact carefully.
+You can improve your document even though if the build job succeeded.
+
+Documents will be uploaded to
+``http://artifacts.opnfv.org/review/<Change Number>/`` for review.
+Those documents will be replaced if you update the change by sending new
+patch set to gerrit, and deleted after the change is merged.
+Document link(s) can be found in your change page on gerrit as a review
+comment.
+
+Note:
+Currently, the job reports 'SUCCESS' as result of document build even if the
+PDF creation failed. This is a provisional workaround, since many projects are
+not ready for PDF creation yet.
+
+Merge Jobs
+==========
+
+The merge job name is **opnfv-docs-merge**.
+
+Once you are happy with the look of your documentation, you can submit the
+change. Then, the merge job will upload latest build documents to
+``http://artifacts.opnfv.org/<Project Name>/docs/`` .
+You can put links in your project wiki page, so that everyone can see the
+latest document always.
diff --git a/docs/how-to-use-docs/index.rst b/docs/how-to-use-docs/index.rst
index 0965eb3cd..8e4dbd266 100644
--- a/docs/how-to-use-docs/index.rst
+++ b/docs/how-to-use-docs/index.rst
@@ -1,19 +1,12 @@
 ..
-    This is new template created on Thu Nov 26 16:18:09 JST 2015
+    This is new template created on Nov 27, 2015.
 
-=====================
+*********************
 Example Documentation
-=====================
+*********************
 
 .. toctree::
    :numbered:
-   :maxdepth: 4
+   :maxdepth: 2
 
    documentation-example.rst
-
-..
-    Leave these at the bottom of 'index.rst' file
-
-Revision: _sha1_
-
-Build date: |today|
diff --git a/docs/jenkins-job-builder/index.rst b/docs/jenkins-job-builder/index.rst
index f8f568339..b85b1320f 100644
--- a/docs/jenkins-job-builder/index.rst
+++ b/docs/jenkins-job-builder/index.rst
@@ -1,13 +1,9 @@
-===========================
+***************************
 Release Engineering Project
-===========================
+***************************
 
 .. toctree::
    :numbered:
-   :maxdepth: 8
+   :maxdepth: 2
 
    opnfv-jjb-usage.rst
-
-Revision: _sha1_
-
-Build date: |today|
diff --git a/docs/jenkins-job-builder/opnfv-jjb-usage.rst b/docs/jenkins-job-builder/opnfv-jjb-usage.rst
index 4aecc6de6..7daacaffb 100644
--- a/docs/jenkins-job-builder/opnfv-jjb-usage.rst
+++ b/docs/jenkins-job-builder/opnfv-jjb-usage.rst
@@ -1,5 +1,6 @@
+===========================================
 Creating/Configuring/Verifying Jenkins Jobs
-============================================
+===========================================
 
 Clone the repo::
 
diff --git a/jjb/releng-macros.yaml b/jjb/releng-macros.yaml
index 3afe8482a..c009abd97 100644
--- a/jjb/releng-macros.yaml
+++ b/jjb/releng-macros.yaml
@@ -152,73 +152,9 @@
     builders:
         - shell: |
             #!/bin/bash -e
-            set -o pipefail
             export PATH=$PATH:/usr/local/bin/
-
-            [[ $GERRIT_CHANGE_NUMBER =~ .+ ]]
-
-            _get_title_script="
-            import os
-            from docutils import core, nodes
-            with open('index.rst', 'r') as file:
-                data = file.read()
-            doctree = core.publish_doctree(data,
-                settings_overrides={'report_level': 5,
-                                    'halt_level': 5})
-            if isinstance(doctree[0], nodes.title):
-                title = doctree[0]
-            else:
-                for c in doctree.children:
-                    if isinstance(c, nodes.section):
-                        title = c[0]
-                        break
-            print title.astext()"
-            _git_sha1="$(git rev-parse HEAD)"
-
             git clone ssh://gerrit.opnfv.org:29418/releng
-            [[ -d releng ]]
-
-            find docs/ -name 'index.rst' -printf '%h\n' | while read dir
-            do
-                _name="${dir##*/}"
-                _build="$dir/build"
-                _output="docs/output/$_name"
-
-                echo
-                echo "#################${dir//?/#}"
-                echo "Building DOCS in $dir"
-                echo "#################${dir//?/#}"
-                echo
-
-                sed -i "s/_sha1_/$_git_sha1/g" "$dir/index.rst"
-
-                if [[ ! -f "$dir/conf.py" ]] ; then
-                    cp releng/docs/etc/conf.py "$dir/conf.py"
-                    _title=$(cd $dir; python -c "$_get_title_script")
-                    echo "latex_documents = [('index', '$_name.tex', \"$_title\", 'OPNFV', 'manual'),]" >> "$dir/conf.py"
-                fi
-                cp -f releng/docs/etc/opnfv-logo.png "$dir/opnfv-logo.png"
-
-                mkdir -p "$_output"
-
-                sphinx-build -b html -E "$dir" "$_output"
-
-                # Note: PDF creation may fail in project doc builds.
-                #       We allow this test to be marked as succeeded with
-                #       failure in PDF creation, but leave message to fix it.
-                #       Any failure has to be fixed before B release.
-                {
-                    sphinx-build -b latex -E "$dir" "$_build"
-                    make -C "$_build" LATEXOPTS='--interaction=nonstopmode' all-pdf
-                    mv "$_build/$_name.pdf" "$_output"
-                } || {
-                    _msg="Error: PDF creation for $dir has failed, please fix source rst file(s)."
-                    echo
-                    echo "$_msg"
-                    echo
-                    echo "$_msg" >> gerrit_comment.txt
-                }
-            done
+            GERRIT_COMMENT=gerrit_comment.txt ./releng/utils/docs-build.sh
 
 - builder:
     name: upload-under-review-docs-to-opnfv-artifacts
diff --git a/utils/docs-build.sh b/utils/docs-build.sh
new file mode 100755
index 000000000..bf9ab5b1e
--- /dev/null
+++ b/utils/docs-build.sh
@@ -0,0 +1,124 @@
+#!/bin/bash -e
+
+export PATH=$PATH:/usr/local/bin/
+
+
+SRC_DIR=${SRC_DIR:-docs}
+INDEX_RST=${INDEX_RST:-index.rst}
+BUILD_DIR=${BUILD_DIR:-build}
+OUTPUT_DIR=${OUTPUT_DIR:-output}
+RELENG_DIR=${RELENG_DIR:-releng}
+GERRIT_COMMENT=${GERRIT_COMMENT:-}
+
+get_title_script="
+import os
+from docutils import core, nodes
+
+try:
+    with open('index.rst', 'r') as file:
+        data = file.read()
+    doctree = core.publish_doctree(data,
+        settings_overrides={'report_level': 5,
+                            'halt_level': 5})
+    if isinstance(doctree[0], nodes.title):
+        title = doctree[0]
+    else:
+        for c in doctree.children:
+            if isinstance(c, nodes.section):
+                title = c[0]
+                break
+    print title.astext()
+except:
+    print 'None'"
+revision="$(git rev-parse --short HEAD)"
+rev_full="$(git rev-parse HEAD)"
+version="$(git describe --abbrev=0 2> /dev/null || echo draft) ($revision)"
+project="$(basename $(git rev-parse --show-toplevel))"
+html_notes="\n    Revision: $rev_full\n\n    Build date: |today|"
+default_conf='releng/docs/etc/conf.py'
+opnfv_logo='releng/docs/etc/opnfv-logo.png'
+
+function add_html_notes() {
+    _src="$1"
+    _dir="$2"
+
+    if grep -q -e ' _sha1_' "$_src"/*.rst ; then
+        # TODO: remove this, once old templates were removed from all repos.
+        echo
+        echo "Warn: '_sha1_' was found in $_dir , use the latest document template."
+        echo "      See https://wiki.opnfv.org/documentation/tools ."
+        echo
+        sed -i "s/ _sha1_/ $git_sha1/g" "$_src"/*.rst
+    fi
+    sed -i -e "\$a\\\n.. only:: html\n$html_notes" "$_src"/*.rst
+}
+
+function add_config() {
+    _conf="$1"
+    _param="$2"
+    _val="$3"
+
+    if ! grep -q -e "^$_param = " "$_conf" ; then
+        echo "Adding '$_param' into $_conf ..."
+        echo "$_param = $_val" >> "$_conf"
+    fi
+}
+
+
+if [[ ! -d "$RELENG_DIR" ]] ; then
+    echo "Error: $RELENG_DIR dir not found. See https://wiki.opnfv.org/documentation/tools ."
+    exit 1
+fi
+
+find $SRC_DIR -name $INDEX_RST -printf '%h\n' | while read dir
+do
+    name="${dir##*/}"
+    src="$BUILD_DIR/src/$name"
+    build="$BUILD_DIR/$name"
+    output="$OUTPUT_DIR/$name"
+    conf="$src/conf.py"
+
+    echo
+    echo "#################${dir//?/#}"
+    echo "Building DOCS in $dir"
+    echo "#################${dir//?/#}"
+    echo
+
+    mkdir -p "$BUILD_DIR/src"
+    [[ -e "$src" ]] && rm -rf "$src"
+    cp -r "$dir" "$src"
+
+    add_html_notes "$src" "$dir"
+
+    [[ ! -f "$conf" ]] && cp "$default_conf" "$conf"
+    title=$(cd $src; python -c "$get_title_script")
+    latex_conf="[('index', '$name.tex', \"$title\", 'OPNFV', 'manual'),]"
+    add_config "$conf" 'latex_documents' "$latex_conf"
+    add_config "$conf" 'release' "u'$version'"
+    add_config "$conf" 'version' "u'$version'"
+    add_config "$conf" 'project' "u'$project'"
+    add_config "$conf" 'copyright' "u'$(date +%Y), OPNFV'"
+    cp -f $opnfv_logo "$src/opnfv-logo.png"
+
+    mkdir -p "$output"
+
+    sphinx-build -b html -t html -E "$src" "$output"
+
+    # Note: PDF creation may fail in project doc builds.
+    #       We allow this build job to be marked as succeeded with
+    #       failure in PDF creation, but leave message to fix it.
+    #       Any failure has to be fixed before OPNFV B release.
+    {
+        sphinx-build -b latex -t pdf -E "$src" "$build" && \
+            make -C "$build" LATEXOPTS='--interaction=nonstopmode' all-pdf
+    } && {
+        mv "$build/$name.pdf" "$output"
+    } || {
+        msg="Error: PDF creation for $dir has failed, please fix source rst file(s)."
+        echo
+        echo "$msg"
+        echo
+        [[ -n "$GERRIT_COMMENT" ]] && echo "$msg" >> "$GERRIT_COMMENT"
+    }
+
+done