Support for building Fuel behind a http proxy

The build system is now able to work also behind a traditional web
proxy setup if the http_proxy, https_proxy and (if needed) no_proxy
environment variables has been set prior to invoking make.

This is a joint work by Gillian Dunne <gillian.dunne@intel.com> and
Stefan Berg <stefan.k.berg@ericsson.com>.

Verification so far has been with a mock setup, placing the build
machine behind a Squid proxy and blocking outgoing traffic not going
through the proxy by firewall rules. The following environment
variables was set in the host for these tests:

RSYNC_PROXY=10.0.0.1:8888
http_proxy=http://10.0.0.1:8888
https_proxy=http://10.0.0.1:8888
no_proxy=localhost,127.0.0.1,.consultron.com,.sock

*** IMPORTANT NOTE ABOUT THE HOST PROXY SETTINGS ***

The build system will make use the following proxy environment
variables:

http_proxy:
https_proxy
no_proxy
RSYNC_PROXY
RSYNC_CONNECT_PROG

During the build phase, a local Ubuntu package repository is fetched
from upstream in order to be added to the OPNFV Fuel ISO and for parts
of this process rsync is used.

This will require that either RSYNC_PROXY is set according to the
format "<proxy host>:<proxy port>" and that the proxy indicated indeed
allows rsync traffic *or* that RSYNC_CONNECT_PROG is set to use an
alternative transport. For a detailed explanation of these settings,
see the rsync manual page.

*** IMPORTANT NOTE ABOUT THE HOST DOCKER DAEMON SETTINGS ***

The Docker daemon on the host must be configured to use the http proxy
for it to be able to pull the base Ubuntu 14.04 image from the Docker
registry before invoking make! In Ubuntu this is done by adding a line
like:

export http_proxy="http://10.0.0.1:8888/"

to /etc/default/docker and restarting the Docker daemon.

Change-Id: Ieed2269af295d90a4b33d834f723889bdf9c7dc6
Signed-off-by: Stefan K. Berg <stefan.k.berg@ericsson.com>

1 files changed, 103 insertions, 16 deletions

diff --git a/fuel/docs/src/build-instructions.rst b/fuel/docs/src/build-instructions.rst
index 790bcf513..08b560375 100644
--- a/fuel/docs/src/build-instructions.rst
+++ b/fuel/docs/src/build-instructions.rst
@@ -1,5 +1,5 @@
 ============================================================================
-OPNFV Build instructions for - Fuel deployment tool - OPNFV Arno SR1 release
+OPNFV Build instructions for the Fuel deployment tool
 ============================================================================
 
 .. contents:: Table of Contents
@@ -8,13 +8,13 @@ OPNFV Build instructions for - Fuel deployment tool - OPNFV Arno SR1 release
 Abstract
 ========
 
-This document describes how to build the Fuel deployment tool for the Arno SR1 release of OPNFV, the build system, dependencies and required system resources.
+This document describes how to build the Fuel deployment tool for the B release of OPNFV - the build system, dependencies and required system resources.
 
 License
 =======
-Arno SR1 release of OPNFV when using Fuel as a deployment tool DOCs (c) by Jonas Bjurel (Ericsson AB)
+Brahmaputra release of OPNFV when using Fuel as a deployment tool DOCs (c) by Jonas Bjurel (Ericsson AB) and others.
 
-Arno SR1 release of OPNFV when using Fuel as a deployment tool DOCs (c) are licensed under a Creative Commons Attribution 4.0 International License. You should have received a copy of the license along with this. If not, see <http://creativecommons.org/licenses/by/4.0/>.
+Brahmaputra release of OPNFV when using Fuel as a deployment tool DOCs (c) are licensed under a Creative Commons Attribution 4.0 International License. You should have received a copy of the license along with this. If not, see <http://creativecommons.org/licenses/by/4.0/>.
 
 
 
@@ -31,11 +31,14 @@ Version history
 | 2015-09-24         | 1.1.0              | Jonas Bjurel       | Instructions for     |
 |                    |                    | (Ericsson AB)      | the Arno SR1 release |
 +--------------------+--------------------+--------------------+----------------------+
+| 2015-10-23         | 1.1.1              | Stefan Berg        | Added instructions   |
+|                    |                    | (Ericsson AB)      | for proxy builds     |
++--------------------+--------------------+--------------------+----------------------+
 
 Introduction
 ============
 
-This document describes the build system used to build the Fuel deployment tool for the Arno SR1 release of OPNFV, required dependencies and minimum requirements on the host to be used for the buildsystem.
+This document describes the build system used to build the Fuel deployment tool for the Brahmaputra release of OPNFV, required dependencies and minimum requirements on the host to be used for the buildsystem.
 
 The Fuel build system is desigened around Docker containers such that dependencies outside of the build system can be kept to a minimum. It also shields the host from any potential dangerous operations performed by the build system.
 
@@ -78,7 +81,7 @@ After having installed Docker, add yourself to the docker group:
 <usermod -a -G docker [userid]>
 
 Also make sure to define relevant DNS servers part of the global dns chain in
-in your </etc/default/docker> configuration file, eg.
+in your </etc/default/docker> configuration file, for example:
 
 <DOCKER_OPTS=" --dns=8.8.8.8 --dns=8.8.8.4">
 
@@ -95,17 +98,26 @@ Setting up OPNFV Gerrit in order to being able to clone the code
 
 - Select "SSH Public Keys" to the left and then "Add Key" and paste your public key in.
 
-Clone the OPNFV code git repository
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Clone the OPNFV code Git repository with your SSH key
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Now it is time to clone the code repository:
 
-<git clone ssh://'Linux foundation user'@gerrit.opnfv.org:29418/genesis>
+<git clone ssh://'Linux foundation user'@gerrit.opnfv.org:29418/fuel>
+
+Now you should have the OPNFV fuel repository with the Fuel directories stored locally on your build host.
+
+Check out the Brahmaputra release:
+<cd fuel>
+<git checkout insert-b-release-tag-here0>
+
+Clone the OPNFV code Git repository without a SSH key
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+You can also opt to clone the code repository without a SSH key:
 
-Now you should have the OPNFV genesis repository with the Fuel directories stored locally on your build host.
+<git clone https://gerrit.opnfv.org:29418/gerrit/fuel>
+
+Make sure to checkout the release tag as described above.
 
-Check out the Arno SR1 release:
-<cd genesis>
-<git checkout arno.2015.2.0>
 
 Building
 ========
@@ -116,10 +128,85 @@ There are two methods available for building Fuel:
 
 - An abstracted method using build.sh
 
+
+Support for building behind a http/https/rsync proxy
+----------------------------------------------------
+
+The build system is able to make use of a web proxy setup if the
+http_proxy, https_proxy, no_proxy (if needed) and RSYNC_PROXY or
+RSYNC_CONNECT_PROG environment variables have been set before invoking make.
+
+The proxy setup must permit port 80 (http), 443 (https) and 873
+(rsync).
+
+Important note about the host Docker daemon settings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The Docker daemon on the host must be configured to use the http proxy
+for it to be able to pull the base Ubuntu 14.04 image from the Docker
+registry before invoking make! In Ubuntu this is done by adding a line
+like:
+
+export http_proxy="http://10.0.0.1:8888/"
+
+to /etc/default/docker and restarting the Docker daemon.
+
+Setting proxy environment variables prior to build
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The build system will make use the following environment variables
+that needs to be exported to subshells by using export (bash) or
+setenv (csh/tcsh).
+
+| http_proxy (or HTTP_PROXY)
+| https_proxy (or HTTP_PROXY)
+| no_proxy (or NO_PROXY)
+| RSYNC_PROXY
+| RSYNC_CONNECT_PROG
+
+As an example, these are the settings that were put in the user's
+.bashrc when verifying the proxy build functionality:
+
+| export RSYNC_PROXY=10.0.0.1:8888
+| export http_proxy=http://10.0.0.1:8888
+| export https_proxy=http://10.0.0.1:8888
+| export no_proxy=localhost,127.0.0.1,.consultron.com,.sock
+
+Using a ssh proxy for the rsync connection
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If the proxy setup is not allowing the rsync protocol, an alternative
+solution is to use a SSH tunnel to a machine capable of accessing the
+outbound port 873. Set the RSYNC_CONNECT_PROG according to the rsync
+manual page (for example to "ssh <username>@<hostname> nc %H 873") to enable
+this. Also note that netcat needs to be installed on the remote
+system!
+
+Make sure that the ssh command also refers to the user on the remote
+system, as the command itself will be run from the Docker build container
+as the root user (but with the invoking user's SSH keys).
+
+Disabling the Ubuntu repo cache if rsync is not allowed
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+During the build phase, a local Ubuntu package repository is fetched
+from upstream in order to be added to the OPNFV Fuel ISO and for parts
+of this process rsync is used.
+
+If neither of the two available methods for proxying rsync are
+available, the last resort is to turn off the caching of the Ubuntu
+packages in the build system. This is done by removing the
+"f_repobuild" from SUBDIRS in the beginning of
+the fuel/build/f_isoroot/Makefile.
+
+Note! Doing this will require the Fuel master node to have Internet
+access when installing the ISO artifact built as no Ubuntu package
+cache will be on the ISO!
+
 Configure your build environment
 -------------------------------------
 
-** Configuring the build environment should not be performed if building standard Arno release **
+** Configuring the build environment should not be performed if building standard Brahmaputra release **
 
 Select the versions of the components you want to build by editing the fuel/build/config.mk file.
 
@@ -172,8 +259,8 @@ References
 ==========
 -
 
-:Authors: Jonas Bjurel (Ericsson)
-:Version: 1.1.0
+:Authors: Jonas Bjurel (Ericsson), Stefan Berg (Ericsson)
+:Version: x.x.x
 
 **Documentation tracking**