Merge "[docs] Refresh for Gambia release" into stable/gambia

1 files changed, 885 insertions, 279 deletions

diff --git a/docs/release/userguide/userguide.rst b/docs/release/userguide/userguide.rst
index 76639abcf..c6602f3cb 100644
--- a/docs/release/userguide/userguide.rst
+++ b/docs/release/userguide/userguide.rst
@@ -2,410 +2,1016 @@
 .. http://creativecommons.org/licenses/by/4.0
 .. (c) Open Platform for NFV Project, Inc. and its contributors
 
-========
+*********************
+OPNFV Fuel User Guide
+*********************
+
 Abstract
 ========
 
-This document contains details about how to use OPNFV Fuel - Fraser
-release - after it was deployed. For details on how to deploy check the
-installation instructions in the :ref:`fuel_userguide_references` section.
+This document contains details about using OPNFV Fuel ``Gambia`` release after
+it was deployed. For details on how to deploy OpenStack, check
+the installation instructions in the :ref:`fuel_userguide_references` section.
 
-This is an unified documentation for both x86_64 and aarch64
+This is an unified documentation for both ``x86_64`` and ``aarch64``
 architectures. All information is common for both architectures
 except when explicitly stated.
 
-
-
-================
 Network Overview
 ================
 
 Fuel uses several networks to deploy and administer the cloud:
 
-+------------------+---------------------------------------------------------+
-| Network name     | Description                                             |
-|                  |                                                         |
-+==================+=========================================================+
-| **PXE/ADMIN**    | Used for booting the nodes via PXE and/or Salt          |
-|                  | control network                                         |
-+------------------+---------------------------------------------------------+
-| **MCPCONTROL**   | Used to provision the infrastructure VMs (Salt & MaaS)  |
-+------------------+---------------------------------------------------------+
-| **Mgmt**         | Used for internal communication between                 |
-|                  | OpenStack components                                    |
-+------------------+---------------------------------------------------------+
-| **Internal**     | Used for VM data communication within the               |
-|                  | cloud deployment                                        |
-+------------------+---------------------------------------------------------+
-| **Public**       | Used to provide Virtual IPs for public endpoints        |
-|                  | that are used to connect to OpenStack services APIs.    |
-|                  | Used by Virtual machines to access the Internet         |
-+------------------+---------------------------------------------------------+
++------------------+----------------------------------------------------------+
+| Network name     | Description                                              |
+|                  |                                                          |
++==================+==========================================================+
+| **PXE/admin**    | Used for booting the nodes via PXE and/or Salt           |
+|                  | control network                                          |
++------------------+----------------------------------------------------------+
+| **mcpcontrol**   | Used to provision the infrastructure hosts (Salt & MaaS) |
++------------------+----------------------------------------------------------+
+| **management**   | Used for internal communication between                  |
+|                  | OpenStack components                                     |
++------------------+----------------------------------------------------------+
+| **internal**     | Used for VM data communication within the                |
+|                  | cloud deployment                                         |
++------------------+----------------------------------------------------------+
+| **public**       | Used to provide Virtual IPs for public endpoints         |
+|                  | that are used to connect to OpenStack services APIs.     |
+|                  | Used by Virtual machines to access the Internet          |
++------------------+----------------------------------------------------------+
+
+These networks - except ``mcpcontrol`` - can be Linux bridges configured
+before the deploy on the Jumpserver.
+If they don't exists at deploy time, they will be created by the scripts as
+``libvirt`` managed networks.
+
+Network ``mcpcontrol``
+~~~~~~~~~~~~~~~~~~~~~~
+
+``mcpcontrol`` is a virtual network, managed by libvirt. Its only purpose is to
+provide a simple method of assigning an arbitrary ``INSTALLER_IP`` to the Salt
+master node (``cfg01``), to maintain backwards compatibility with old OPNFV
+Fuel behavior. Normally, end-users only need to change the ``INSTALLER_IP`` if
+the default CIDR (``10.20.0.0/24``) overlaps with existing lab networks.
+
+``mcpcontrol`` has both NAT and DHCP enabled, so the Salt master (``cfg01``)
+and the MaaS VM (``mas01``, when present) get assigned predefined IPs (``.2``,
+``.3``, while the jumpserver bridge port gets ``.1``).
+
++------------------+---------------------------+-----------------------------+
+| Host             | Offset in IP range        | Default address             |
++==================+===========================+=============================+
+| ``jumpserver``   | 1st                       | ``10.20.0.1``               |
++------------------+---------------------------+-----------------------------+
+| ``cfg01``        | 2nd                       | ``10.20.0.2``               |
++------------------+---------------------------+-----------------------------+
+| ``mas01``        | 3rd                       | ``10.20.0.3``               |
++------------------+---------------------------+-----------------------------+
+
+This network is limited to the ``jumpserver`` host and does not require any
+manual setup.
+
+Network ``PXE/admin``
+~~~~~~~~~~~~~~~~~~~~~
+
+.. TIP::
+
+    ``PXE/admin`` does not usually use an IP range offset in ``IDF``.
+
+.. NOTE::
+
+    During ``MaaS`` commissioning phase, IP addresses are handed out by
+    ``MaaS``'s DHCP.
 
+.. NOTE::
 
-These networks - except mcpcontrol - can be linux bridges configured before the deploy on the
-Jumpserver. If they don't exists at deploy time, they will be created by the scripts as virsh
-networks.
+    Default addresses in below table correspond to a ``PXE/admin`` CIDR of
+    ``192.168.11.0/24`` (the usual value used in OPNFV labs).
+
+    This is defined in ``IDF`` and can easily be changed to something else.
+
+.. TODO: detail MaaS DHCP range start/end
+
++------------------+-----------------------+---------------------------------+
+| Host             | Offset in IP range    | Default address                 |
++==================+=======================+=================================+
+| ``jumpserver``   | 1st                   | ``192.168.11.1``                |
+|                  |                       | (manual assignment)             |
++------------------+-----------------------+---------------------------------+
+| ``cfg01``        | 2nd                   | ``192.168.11.2``                |
++------------------+-----------------------+---------------------------------+
+| ``mas01``        | 3rd                   | ``192.168.11.3``                |
++------------------+-----------------------+---------------------------------+
+| ``prx01``,       | 4th,                  | ``192.168.11.4``,               |
+| ``prx02``        | 5th                   | ``192.168.11.5``                |
++------------------+-----------------------+---------------------------------+
+| ``gtw01``,       | ...                   | ``...``                         |
+| ``gtw02``,       |                       |                                 |
+| ``gtw03``        |                       |                                 |
++------------------+-----------------------+---------------------------------+
+| ``kvm01``,       |                       |                                 |
+| ``kvm02``,       |                       |                                 |
+| ``kvm03``        |                       |                                 |
++------------------+-----------------------+---------------------------------+
+| ``dbs01``,       |                       |                                 |
+| ``dbs02``,       |                       |                                 |
+| ``dbs03``        |                       |                                 |
++------------------+-----------------------+---------------------------------+
+| ``msg01``,       |                       |                                 |
+| ``msg02``,       |                       |                                 |
+| ``msg03``        |                       |                                 |
++------------------+-----------------------+---------------------------------+
+| ``mdb01``,       |                       |                                 |
+| ``mdb02``,       |                       |                                 |
+| ``mdb03``        |                       |                                 |
++------------------+-----------------------+---------------------------------+
+| ``ctl01``,       |                       |                                 |
+| ``ctl02``,       |                       |                                 |
+| ``ctl03``        |                       |                                 |
++------------------+-----------------------+---------------------------------+
+| ``odl01``,       |                       |                                 |
+| ``odl02``,       |                       |                                 |
+| ``odl03``        |                       |                                 |
++------------------+-----------------------+---------------------------------+
+| ``mon01``,       |                       |                                 |
+| ``mon02``,       |                       |                                 |
+| ``mon03``,       |                       |                                 |
+| ``log01``,       |                       |                                 |
+| ``log02``,       |                       |                                 |
+| ``log03``,       |                       |                                 |
+| ``mtr01``,       |                       |                                 |
+| ``mtr02``,       |                       |                                 |
+| ``mtr03``        |                       |                                 |
++------------------+-----------------------+---------------------------------+
+| ``cmp001``,      |                       |                                 |
+| ``cmp002``,      |                       |                                 |
+| ``...``          |                       |                                 |
++------------------+-----------------------+---------------------------------+
+
+Network ``management``
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. TIP::
+
+    ``management`` often has an IP range offset defined in ``IDF``.
 
-Mcpcontrol exists only on the Jumpserver and needs to be virtual because a DHCP server runs
-on this network and associates static host entry IPs for Salt and Maas VMs.
+.. NOTE::
 
+    Default addresses in below table correspond to a ``management`` CIDR of
+    ``172.16.10.0/24`` (one of the commonly used values in OPNFV labs).
+    This is defined in ``IDF`` and can easily be changed to something else.
+
+.. WARNING::
+
+    Default addresses in below table correspond to a ``management`` IP range of
+    ``172.16.10.10-172.16.10.254`` (one of the commonly used values in OPNFV
+    labs). This is defined in ``IDF`` and can easily be changed to something
+    else. Since the ``jumpserver`` address is manually assigned, this is
+    usually not subject to the IP range restriction in ``IDF``.
+
++------------------+-----------------------+---------------------------------+
+| Host             | Offset in IP range    | Default address                 |
++==================+=======================+=================================+
+| ``jumpserver``   | N/A                   | ``172.16.10.1``                 |
+|                  |                       | (manual assignment)             |
++------------------+-----------------------+---------------------------------+
+| ``cfg01``        | 1st                   | ``172.16.10.2``                 |
+|                  |                       | (IP range ignored for now)      |
++------------------+-----------------------+---------------------------------+
+| ``mas01``        | 2nd                   | ``172.16.10.12``                |
++------------------+-----------------------+---------------------------------+
+| ``prx``          | 3rd,                  | ``172.16.10.13``,               |
+|                  |                       |                                 |
+| ``prx01``,       | 4th,                  | ``172.16.10.14``,               |
+| ``prx02``        | 5th                   | ``172.16.10.15``                |
++------------------+-----------------------+---------------------------------+
+| ``gtw01``,       | ...                   | ``...``                         |
+| ``gtw02``,       |                       |                                 |
+| ``gtw03``        |                       |                                 |
++------------------+-----------------------+---------------------------------+
+| ``kvm``,         |                       |                                 |
+|                  |                       |                                 |
+| ``kvm01``,       |                       |                                 |
+| ``kvm02``,       |                       |                                 |
+| ``kvm03``        |                       |                                 |
++------------------+-----------------------+---------------------------------+
+| ``dbs``,         |                       |                                 |
+|                  |                       |                                 |
+| ``dbs01``,       |                       |                                 |
+| ``dbs02``,       |                       |                                 |
+| ``dbs03``        |                       |                                 |
++------------------+-----------------------+---------------------------------+
+| ``msg``,         |                       |                                 |
+|                  |                       |                                 |
+| ``msg01``,       |                       |                                 |
+| ``msg02``,       |                       |                                 |
+| ``msg03``        |                       |                                 |
++------------------+-----------------------+---------------------------------+
+| ``mdb``,         |                       |                                 |
+|                  |                       |                                 |
+| ``mdb01``,       |                       |                                 |
+| ``mdb02``,       |                       |                                 |
+| ``mdb03``        |                       |                                 |
++------------------+-----------------------+---------------------------------+
+| ``ctl``,         |                       |                                 |
+|                  |                       |                                 |
+| ``ctl01``,       |                       |                                 |
+| ``ctl02``,       |                       |                                 |
+| ``ctl03``        |                       |                                 |
++------------------+-----------------------+---------------------------------+
+| ``odl``,         |                       |                                 |
+|                  |                       |                                 |
+| ``odl01``,       |                       |                                 |
+| ``odl02``,       |                       |                                 |
+| ``odl03``        |                       |                                 |
++------------------+-----------------------+---------------------------------+
+| ``mon``,         |                       |                                 |
+|                  |                       |                                 |
+| ``mon01``,       |                       |                                 |
+| ``mon02``,       |                       |                                 |
+| ``mon03``,       |                       |                                 |
+|                  |                       |                                 |
+| ``log``,         |                       |                                 |
+|                  |                       |                                 |
+| ``log01``,       |                       |                                 |
+| ``log02``,       |                       |                                 |
+| ``log03``,       |                       |                                 |
+|                  |                       |                                 |
+| ``mtr``,         |                       |                                 |
+|                  |                       |                                 |
+| ``mtr01``,       |                       |                                 |
+| ``mtr02``,       |                       |                                 |
+| ``mtr03``        |                       |                                 |
++------------------+-----------------------+---------------------------------+
+| ``cmp001``,      |                       |                                 |
+| ``cmp002``,      |                       |                                 |
+| ``...``          |                       |                                 |
++------------------+-----------------------+---------------------------------+
+
+Network ``internal``
+~~~~~~~~~~~~~~~~~~~~
+
+.. TIP::
+
+    ``internal`` does not usually use an IP range offset in ``IDF``.
 
+.. NOTE::
 
-===================
-Accessing the Cloud
-===================
+    Default addresses in below table correspond to an ``internal`` CIDR of
+    ``10.1.0.0/24`` (the usual value used in OPNFV labs).
+    This is defined in ``IDF`` and can easily be changed to something else.
+
++------------------+------------------------+--------------------------------+
+| Host             | Offset in IP range     | Default address                |
++==================+========================+================================+
+| ``jumpserver``   | N/A                    | ``10.1.0.1``                   |
+|                  |                        | (manual assignment, optional)  |
++------------------+------------------------+--------------------------------+
+| ``gtw01``,       | 1st,                   | ``10.1.0.2``,                  |
+| ``gtw02``,       | 2nd,                   | ``10.1.0.3``,                  |
+| ``gtw03``        | 3rd                    | ``10.1.0.4``                   |
++------------------+------------------------+--------------------------------+
+| ``cmp001``,      | 4th,                   | ``10.1.0.5``,                  |
+| ``cmp002``,      | 5th,                   | ``10.1.0.6``,                  |
+| ``...``          | ...                    | ``...``                        |
++------------------+------------------------+--------------------------------+
 
-Access to any component of the deployed cloud is done from Jumpserver to user *ubuntu* with
-ssh key ``/var/lib/opnfv/mcp.rsa``. The example below is a connection to Salt master.
+Network ``public``
+~~~~~~~~~~~~~~~~~~
 
-    .. code-block:: bash
+.. TIP::
 
-        $ ssh -o StrictHostKeyChecking=no -i /var/lib/opnfv/mcp.rsa -l ubuntu 10.20.0.2
+    ``public`` often has an IP range offset defined in ``IDF``.
 
 .. NOTE::
 
-    The Salt master IP is not hard set, it is configurable via ``INSTALLER_IP`` during deployment
+    Default addresses in below table correspond to a ``public`` CIDR of
+    ``172.30.10.0/24`` (one of the used values in OPNFV labs).
+    This is defined in ``IDF`` and can easily be changed to something else.
+
+.. WARNING::
+
+    Default addresses in below table correspond to a ``public`` IP range of
+    ``172.30.10.100-172.30.10.254`` (one of the used values in OPNFV
+    labs). This is defined in ``IDF`` and can easily be changed to something
+    else. Since the ``jumpserver`` address is manually assigned, this is
+    usually not subject to the IP range restriction in ``IDF``.
+
++------------------+------------------------+--------------------------------+
+| Host             | Offset in IP range     | Default address                |
++==================+========================+================================+
+| ``jumpserver``   | N/A                    | ``172.30.10.72``               |
+|                  |                        | (manual assignment, optional)  |
++------------------+------------------------+--------------------------------+
+| ``prx``,         | 1st,                   | ``172.30.10.101``,             |
+|                  |                        |                                |
+| ``prx01``,       | 2nd,                   | ``172.30.10.102``,             |
+| ``prx02``        | 3rd                    | ``172.30.10.103``              |
++------------------+------------------------+--------------------------------+
+| ``gtw01``,       | 4th,                   | ``172.30.10.104``,             |
+| ``gtw02``,       | 5th,                   | ``172.30.10.105``,             |
+| ``gtw03``        | 6th                    | ``172.30.10.106``              |
++------------------+------------------------+--------------------------------+
+| ``ctl01``,       | ...                    | ``...``                        |
+| ``ctl02``,       |                        |                                |
+| ``ctl03``        |                        |                                |
++------------------+------------------------+--------------------------------+
+| ``odl``,         |                        |                                |
++------------------+------------------------+--------------------------------+
+| ``cmp001``,      |                        |                                |
+| ``cmp002``,      |                        |                                |
+| ``...``          |                        |                                |
++------------------+------------------------+--------------------------------+
+
+Accessing the Salt Master Node (``cfg01``)
+==========================================
+
+The Salt Master node (``cfg01``) runs a ``sshd`` server listening on
+``0.0.0.0:22``.
+
+To login as ``ubuntu`` user, use the RSA private key ``/var/lib/opnfv/mcp.rsa``:
+
+.. code-block:: console
+
+    jenkins@jumpserver:~$ ssh -o StrictHostKeyChecking=no \
+                              -i /var/lib/opnfv/mcp.rsa \
+                              -l ubuntu 10.20.0.2
+    ubuntu@cfg01:~$
 
-Logging in to cluster nodes is possible from the Jumpserver and from Salt master. On the Salt master
-cluster hostnames can be used instead of IP addresses:
+.. NOTE::
 
-    .. code-block:: bash
+    User ``ubuntu`` has sudo rights.
 
-        $ sudo -i
-        $ ssh -i mcp.rsa ubuntu@ctl01
+.. TIP::
 
-User *ubuntu* has sudo rights.
+    The Salt master IP (``10.20.0.2``) is not hard set, it is configurable via
+    ``INSTALLER_IP`` during deployment.
 
+.. TIP::
 
-=============================
-Exploring the Cloud with Salt
-=============================
+    Starting with the ``Gambia`` release, ``cfg01`` is containerized, so this
+    also works (from ``jumpserver`` only):
 
-To gather information about the cloud, the salt commands can be used. It is based
-around a master-minion idea where the salt-master pushes config to the minions to
-execute actions.
+.. code-block:: console
 
-For example tell salt to execute a ping to ``8.8.8.8`` on all the nodes.
+    jenkins@jumpserver:~$ docker exec -it fuel bash
+    root@cfg01:~$
 
-.. figure:: img/saltstack.png
+Accessing Cluster Nodes
+=======================
 
-Complex filters can be done to the target like compound queries or node roles.
-For more information about Salt see the :ref:`fuel_userguide_references` section.
+Logging in to cluster nodes is possible from the Jumpserver, Salt Master etc.
 
-Some examples are listed below. Note that these commands are issued from Salt master
-as *root* user.
+.. code-block:: console
 
+    jenkins@jumpserver:~$ ssh -i /var/lib/opnfv/mcp.rsa ubuntu@192.168.11.52
 
-#. View the IPs of all the components
-
-    .. code-block:: bash
-
-        root@cfg01:~$ salt "*" network.ip_addrs
-        cfg01.mcp-pike-odl-ha.local:
-           - 10.20.0.2
-           - 172.16.10.100
-        mas01.mcp-pike-odl-ha.local:
-           - 10.20.0.3
-           - 172.16.10.3
-           - 192.168.11.3
-        .........................
+.. TIP::
 
+    ``/etc/hosts`` on ``cfg01`` has all the cluster hostnames, which can be
+    used instead of IP addresses.
 
-#. View the interfaces of all the components and put the output in a file with yaml format
+.. code-block:: console
 
-    .. code-block:: bash
+    root@cfg01:~$ ssh -i ~/fuel/mcp/scripts/mcp.rsa ubuntu@ctl01
 
-        root@cfg01:~$ salt "*" network.interfaces --out yaml --output-file interfaces.yaml
-        root@cfg01:~# cat interfaces.yaml
-        cfg01.mcp-pike-odl-ha.local:
-         enp1s0:
-           hwaddr: 52:54:00:72:77:12
-           inet:
-           - address: 10.20.0.2
-             broadcast: 10.20.0.255
-             label: enp1s0
-             netmask: 255.255.255.0
-           inet6:
-           - address: fe80::5054:ff:fe72:7712
-             prefixlen: '64'
-             scope: link
-           up: true
-        .........................
+Debugging ``MaaS`` Comissioning/Deployment Issues
+=================================================
 
+One of the most common issues when setting up a new POD is ``MaaS`` failing to
+commission/deploy the nodes, usually timing out after a couple of retries.
 
-#. View installed packages in MaaS node
+Such failures might indicate misconfiguration in ``PDF``/``IDF``, ``TOR``
+switch configuration or even faulty hardware.
 
-    .. code-block:: bash
+Here are a couple of pointers for isolating the problem.
 
-        root@cfg01:~# salt "mas*" pkg.list_pkgs
-        mas01.mcp-pike-odl-ha.local:
-            ----------
-            accountsservice:
-                0.6.40-2ubuntu11.3
-            acl:
-                2.2.52-3
-            acpid:
-                1:2.0.26-1ubuntu2
-            adduser:
-                3.113+nmu3ubuntu4
-            anerd:
-                1
-        .........................
+Accessing the ``MaaS`` Dashboard
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+``MaaS`` web-based dashboard is available at
+``http://<mas01 IP address>:5240/MAAS``, e.g.
+``http://172.16.10.12:5240/MAAS``.
 
-#. Execute any linux command on all nodes (list the content of ``/var/log`` in this example)
+The administrator credentials are ``opnfv``/``opnfv_secret``.
 
-    .. code-block:: bash
+.. NOTE::
 
-        root@cfg01:~# salt "*" cmd.run 'ls /var/log'
-        cfg01.mcp-pike-odl-ha.local:
-           alternatives.log
-           apt
-           auth.log
-           boot.log
-           btmp
-           cloud-init-output.log
-           cloud-init.log
-        .........................
+    ``mas01`` VM does not automatically get assigned an IP address in the
+    public network segment. If ``MaaS`` dashboard should be accesiable from
+    the public network, such an address can be manually added to the last
+    VM NIC interface in ``mas01`` (which is already connected to the public
+    network bridge).
 
+Ensure Commission/Deploy Timeouts Are Not Too Small
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-#. Execute any linux command on nodes using compound queries filter
+Some hardware takes longer to boot or to run the initial scripts during
+commissioning/deployment phases. If that's the case, ``MaaS`` will time out
+waiting for the process to finish. ``MaaS`` logs will reflect that, and the
+issue is usually easy to observe on the nodes' serial console - if the node
+seems to PXE-boot the OS live image, starts executing cloud-init/curtin
+hooks without spilling critical errors, then it is powered down/shut off,
+most likely the timeout was hit.
 
-    .. code-block:: bash
+To access the serial console of a node, see your board manufacturer's
+documentation. Some hardware no longer has a physical serial connector these
+days, usually being replaced by a vendor-specific software-based interface.
 
-        root@cfg01:~# salt -C '* and cfg01*' cmd.run 'ls /var/log'
-        cfg01.mcp-pike-odl-ha.local:
-           alternatives.log
-           apt
-           auth.log
-           boot.log
-           btmp
-           cloud-init-output.log
-           cloud-init.log
-        .........................
+If the board supports ``SOL`` (Serial Over LAN) over ``IPMI`` lanplus protocol,
+a simpler solution to hook to the serial console is to use ``ipmitool``.
 
+.. TIP::
 
-#. Execute any linux command on nodes using role filter
+    Early boot stage output might not be shown over ``SOL``, but only over
+    the video console provided by the (vendor-specific) interface.
 
-    .. code-block:: bash
+.. code-block:: console
 
-        root@cfg01:~# salt -I 'nova:compute' cmd.run 'ls /var/log'
-        cmp001.mcp-pike-odl-ha.local:
-           alternatives.log
-           apache2
-           apt
-           auth.log
-           btmp
-           ceilometer
-           cinder
-           cloud-init-output.log
-           cloud-init.log
-        .........................
+    jenkins@jumpserver:~$ ipmitool -H <host BMC IP> -U <user> -P <pass> \
+                                   -I lanplus sol activate
 
+To bypass this, simply set a larger timeout in the ``IDF``.
 
+Check Jumpserver Network Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-===================
-Accessing Openstack
-===================
+.. code-block:: console
 
-Once the deployment is complete, Openstack CLI is accessible from controller VMs (ctl01..03).
-Openstack credentials are at ``/root/keystonercv3``.
+    jenkins@jumpserver:~$ brctl show
+    jenkins@jumpserver:~$ ifconfig -a
 
-    .. code-block:: bash
++-----------------------+------------------------------------------------+
+| Configuration item    | Expected behavior                              |
++=======================+================================================+
+| IP addresses assigned | IP addresses should be assigned to the bridge, |
+| to bridge ports       | and not to individual bridge ports             |
++-----------------------+------------------------------------------------+
 
-        root@ctl01:~# source keystonercv3
-        root@ctl01:~# openstack image list
-        +--------------------------------------+-----------------------------------------------+--------+
-        | ID                                   | Name                                          | Status |
-        +======================================+===============================================+========+
-        | 152930bf-5fd5-49c2-b3a1-cae14973f35f | CirrosImage                                   | active |
-        | 7b99a779-78e4-45f3-9905-64ae453e3dcb | Ubuntu16.04                                   | active |
-        +--------------------------------------+-----------------------------------------------+--------+
+Check Network Connectivity Between Nodes on the Jumpserver
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+``cfg01`` is a Docker container running on the ``jumpserver``, connected to
+Docker networks (created by docker-compose automatically on container up),
+which in turn are connected using veth pairs to their ``libvirt`` managed
+counterparts.
 
-The OpenStack Dashboard, Horizon, is available at ``http://<proxy public VIP>``.
-The administrator credentials are **admin**/**opnfv_secret**.
+For example, the ``mcpcontrol`` network(s) should look like below.
 
-.. figure:: img/horizon_login.png
+.. code-block:: console
 
+    jenkins@jumpserver:~$ brctl show mcpcontrol
+    bridge name   bridge id           STP enabled   interfaces
+    mcpcontrol    8000.525400064f77   yes           mcpcontrol-nic
+                                                    veth_mcp0
+                                                    vnet8
 
-A full list of IPs/services is available at ``<proxy public VIP>:8090`` for baremetal deploys.
+    jenkins@jumpserver:~$ docker network ls
+    NETWORK ID    NAME                              DRIVER   SCOPE
+    81a0fdb3bd78  docker-compose_docker-mcpcontrol  macvlan  local
+    [...]
 
-.. figure:: img/salt_services_ip.png
+    jenkins@jumpserver:~$ docker network inspect docker-compose_mcpcontrol
+    [
+        {
+            "Name": "docker-compose_mcpcontrol",
+            [...]
+            "Options": {
+                "parent": "veth_mcp1"
+            },
+        }
+    ]
 
-==============================
-Guest Operating System Support
-==============================
+Before investigating the rest of the cluster networking configuration, the
+first thing to check is that ``cfg01`` has network connectivity to other
+jumpserver hosted nodes, e.g. ``mas01`` and to the jumpserver itself
+(provided that the jumpserver has an IP address in that particular network
+segment).
 
-There are a number of possibilities regarding the guest operating systems which can be spawned
-on the nodes. The current system spawns virtual machines for VCP VMs on the KVM nodes  and VMs
-requested by users in OpenStack compute nodes. Currently the system supports the following
-UEFI-images for the guests:
-
-+------------------+-------------------+------------------+
-| OS name          | x86_64 status     | aarch64 status   |
-+==================+===================+==================+
-| Ubuntu 17.10     | untested          | Full support     |
-+------------------+-------------------+------------------+
-| Ubuntu 16.04     | Full support      | Full support     |
-+------------------+-------------------+------------------+
-| Ubuntu 14.04     | untested          | Full support     |
-+------------------+-------------------+------------------+
-| Fedora atomic 27 | untested          | Full support     |
-+------------------+-------------------+------------------+
-| Fedora cloud 27  | untested          | Full support     |
-+------------------+-------------------+------------------+
-| Debian           | untested          | Full support     |
-+------------------+-------------------+------------------+
-| Centos 7         | untested          | Not supported    |
-+------------------+-------------------+------------------+
-| Cirros 0.3.5     | Full support      | Full support     |
-+------------------+-------------------+------------------+
-| Cirros 0.4.0     | Full support      | Full support     |
-+------------------+-------------------+------------------+
-
-
-The above table covers only UEFI image and implies OVMF/AAVMF firmware on the host. An x86 deployment
-also supports non-UEFI images, however that choice is up to the underlying hardware and the administrator
-to make.
-
-The images for the above operating systems can be found in their respective websites.
+.. code-block:: console
 
+    jenkins@jumpserver:~$ docker exec -it fuel bash
+    root@cfg01:~# ifconfig -a | grep inet
+        inet addr:10.20.0.2     Bcast:0.0.0.0  Mask:255.255.255.0
+        inet addr:172.16.10.2   Bcast:0.0.0.0  Mask:255.255.255.0
+        inet addr:192.168.11.2  Bcast:0.0.0.0  Mask:255.255.255.0
 
-=================
-OpenStack Storage
-=================
+For each network of interest (``mcpcontrol``, ``mgmt``, ``PXE/admin``), check
+that ``cfg01`` can ping the jumpserver IP in that network segment, as well as
+the ``mas01`` IP in that network.
+
+.. NOTE::
 
-OpenStack Cinder is the project behind block storage in OpenStack and Fuel@OPNFV supports LVM out of the box.
-By default x86 supports 2 additional block storage devices and ARMBand supports only one.
-More devices can be supported if the OS-image created has additional properties allowing block storage devices
-to be spawned as SCSI drives. To do this, add the properties below to the server:
+    ``mcpcontrol`` is set up at VM bringup, so it should always be available,
+    while the other networks are configured by Salt as part of the
+    ``virtual_init`` STATE file.
 
-    .. code-block:: bash
+.. code-block:: console
 
-        $ openstack image set --property hw_disk_bus='scsi' --property hw_scsi_model='virtio-scsi' <image>
+    root@cfg01:~# ping -c1 10.20.0.1  # mcpcontrol jumpserver IP
+    root@cfg01:~# ping -c1 10.20.0.3  # mcpcontrol mas01 IP
 
-The choice regarding which bus to use for the storage drives is an important one. Virtio-blk is the default
-choice for Fuel@OPNFV which attaches the drives in ``/dev/vdX``. However, since we want to be able to attach a
-larger number of volumes to the virtual machines, we recommend the switch to SCSI drives which are attached
-in ``/dev/sdX`` instead. Virtio-scsi is a little worse in terms of performance but the ability to add a larger
-number of drives combined with added features like ZFS, Ceph et al, leads us to suggest the use of virtio-scsi in Fuel@OPNFV for both architectures.
+.. TIP::
 
-More details regarding the differences and performance of virtio-blk vs virtio-scsi are beyond the scope
-of this manual but can be easily found in other sources online like `4`_ or `5`_.
+    ``mcpcontrol`` CIDR is configurable via ``INSTALLER_IP`` env var during
+    deployment. However, IP offsets inside that segment are hard set to ``.1``
+    for the jumpserver, ``.2`` for ``cfg01``, respectively to ``.3`` for
+    ``mas01`` node.
 
-.. _4: https://mpolednik.github.io/2017/01/23/virtio-blk-vs-virtio-scsi/
+.. code-block:: console
 
-.. _5: https://www.ovirt.org/develop/release-management/features/storage/virtio-scsi/
+    root@cfg01:~# salt 'mas*' pillar.item --out yaml \
+                  _param:infra_maas_node01_deploy_address \
+                  _param:infra_maas_node01_address
+    mas01.mcp-ovs-noha.local:
+      _param:infra_maas_node01_address: 172.16.10.12
+      _param:infra_maas_node01_deploy_address: 192.168.11.3
 
-Additional configuration for configuring images in openstack can be found in the OpenStack Glance documentation.
+    root@cfg01:~# ping -c1 192.168.11.1  # PXE/admin jumpserver IP
+    root@cfg01:~# ping -c1 192.168.11.3  # PXE/admin mas01 IP
+    root@cfg01:~# ping -c1 172.16.10.1   # mgmt jumpserver IP
+    root@cfg01:~# ping -c1 172.16.10.12  # mgmt mas01 IP
 
+.. TIP::
 
+    Jumpserver IP addresses for ``PXE/admin``, ``mgmt`` and ``public`` bridges
+    are user-chosen and manually set, so above snippets should be adjusted
+    accordingly if the user chose a different IP, other than ``.1`` in each
+    CIDR.
 
-===================
-Openstack Endpoints
-===================
+Alternatively, a quick ``nmap`` scan would work just as well.
 
-For each Openstack service three endpoints are created: ``admin``, ``internal`` and ``public``.
+.. code-block:: console
 
-    .. code-block:: bash
+    root@cfg01:~# apt update && apt install -y nmap
+    root@cfg01:~# nmap -sn 10.20.0.0/24     # expected: cfg01, mas01, jumpserver
+    root@cfg01:~# nmap -sn 192.168.11.0/24  # expected: cfg01, mas01, jumpserver
+    root@cfg01:~# nmap -sn 172.16.10.0/24   # expected: cfg01, mas01, jumpserver
 
-        ubuntu@ctl01:~$ openstack endpoint list --service keystone
-        +----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------------+
-        | ID                               | Region    | Service Name | Service Type | Enabled | Interface | URL                          |
-        +----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------------+
-        | 008fec57922b4e9e8bf02c770039ae77 | RegionOne | keystone     | identity     | True    | internal  | http://172.16.10.26:5000/v3  |
-        | 1a1f3c3340484bda9ef7e193f50599e6 | RegionOne | keystone     | identity     | True    | admin     | http://172.16.10.26:35357/v3 |
-        | b0a47d42d0b6491b995d7e6230395de8 | RegionOne | keystone     | identity     | True    | public    | https://10.0.15.2:5000/v3    |
-        +----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------------+
+Check ``DHCP`` Reaches Cluster Nodes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-MCP sets up all Openstack services to talk to each other over unencrypted
-connections on the internal management network. All admin/internal endpoints use
-plain http, while the public endpoints are https connections terminated via nginx
-at the VCP proxy VMs.
+One common symptom observed during failed commissioning is that ``DHCP`` does
+not work as expected between cluster nodes (baremetal nodes in the cluster; or
+virtual machines on the jumpserver in case of ``hybrid`` deployments) and
+the ``MaaS`` node.
 
-To access the public endpoints an SSL certificate has to be provided. For
-convenience, the installation script will copy the required certificate into
-to the cfg01 node at ``/etc/ssl/certs/os_cacert``.
+To confirm or rule out this possibility, monitor the serial console output of
+one (or more) cluster nodes during ``MaaS`` commissioning. If the node is
+properly configured to attempt PXE boot, yet it times out waiting for an IP
+address from ``mas01`` ``DHCP``, it's worth checking that ``DHCP`` packets
+reach the ``jumpserver``, respectively the ``mas01`` VM.
 
-Copy the certificate from the cfg01 node to the client that will access the https
-endpoints and place it under ``/etc/ssl/certs/``. The SSL connection will be established
-automatically after.
+.. code-block:: console
 
-    .. code-block:: bash
+    jenkins@jumpserver:~$ sudo apt update && sudo apt install -y dhcpdump
+    jenkins@jumpserver:~$ sudo dhcpdump -i admin_br
 
-        $ ssh -o StrictHostKeyChecking=no -i /var/lib/opnfv/mcp.rsa -l ubuntu 10.20.0.2 \
-          "cat /etc/ssl/certs/os_cacert" | sudo tee /etc/ssl/certs/os_cacert
+.. TIP::
 
+    If ``DHCP`` requests are present, but no replies are sent, ``iptables``
+    might be interfering on the jumpserver.
 
+Check ``MaaS`` Logs
+~~~~~~~~~~~~~~~~~~~
+
+If networking looks fine, yet nodes still fail to commission and/or deploy,
+``MaaS`` logs might offer more details about the failure:
+
+* ``/var/log/maas/maas.log``
+* ``/var/log/maas/rackd.log``
+* ``/var/log/maas/regiond.log``
+
+.. TIP::
+
+    If the problem is with the cluster node and not on the ``MaaS`` server,
+    node's kernel logs usually contain useful information.
+    These are saved via rsyslog on the ``mas01`` node in
+    ``/var/log/maas/rsyslog``.
+
+Recovering Failed Deployments
 =============================
-Reclass model viewer tutorial
-=============================
 
+The first deploy attempt might fail due to various reasons. If the problem
+is not systemic (i.e. fixing it will not introduce incompatible configuration
+changes, like setting a different ``INSTALLER_IP``), the environment is safe
+to be reused and the deployment process can pick up from where it left off.
+
+Leveraging these mechanisms requires a minimum understanding of how the
+deploy process works, at least for manual ``STATE`` runs.
+
+Automatic (re)deploy
+~~~~~~~~~~~~~~~~~~~~
+
+OPNFV Fuel's ``deploy.sh`` script offers a dedicated argument for this, ``-f``,
+which will skip executing the first ``N`` ``STATE`` files, where ``N`` is the
+number of ``-f`` occurrences in the argument list.
+
+.. TIP::
+
+    The list of ``STATE`` files to be executed for a specific environment
+    depends on the OPNFV scenario chosen, deployment type (``virtual``,
+    ``baremetal`` or ``hybrid``) and the presence/absence of a ``VCP``
+    (virtualized control plane).
+
+e.g.: Let's consider a ``baremetal`` enviroment, with ``VCP`` and a simple
+scenario ``os-nosdn-nofeature-ha``, where ``deploy.sh`` failed executing the
+``openstack_ha`` ``STATE`` file.
+
+The simplest redeploy approach (which usually works for **any** combination of
+deployment type/VCP/scenario) is to issue the same deploy command as the
+original attempt used, then adding a single ``-f``:
+
+.. code-block:: console
+
+    jenkins@jumpserver:~/fuel$ ci/deploy.sh -l <lab_name> -p <pod_name> \
+                                            -s <scenario> [...] \
+                                            -f # skips running the virtual_init STATE file
 
-In order to get a better understanding on the reclass model Fuel uses, the `reclass-doc
-<https://github.com/jirihybek/reclass-doc>`_ can be used to visualise the reclass model.
-A simplified installation can be done with the use of a docker ubuntu container. This
-approach will avoid installing packages on the host, which might collide with other packages.
-After the installation is done, a webbrowser on the host can be used to view the results.
+All ``STATE`` files are re-entrant, so the above is equivalent (but a little
+slower) to skipping all ``STATE`` files before the ``openstack_ha`` one, like:
+
+.. code-block:: console
+
+    jenkins@jumpserver:~/fuel$ ci/deploy.sh -l <lab_name> -p <pod_name> \
+                                            -s <scenario> [...] \
+                                            -ffff # skips virtual_init, maas, baremetal_init, virtual_control_plane
+
+.. TIP::
+
+    For fine tuning the infrastructure setup steps executed during deployment,
+    see also the ``-e`` and ``-P`` deploy arguments.
 
 .. NOTE::
 
-    The host can be any device with Docker package already installed.
-    The user which runs the docker needs to have root priviledges.
+    On rare occassions, the cluster cannot idempotently be redeployed (e.g.
+    broken MySQL/Galera cluster), in which case some cleanup is due before
+    (re)running the ``STATE`` files. See ``-E`` deploy arg, which allows
+    either forcing a ``MaaS`` node deletion, then redeployment of all
+    baremetal nodes, if used twice (``-EE``); or only erasing the ``VCP`` VMs
+    if used only once (``-E``).
+
+Manual ``STATE`` Run
+~~~~~~~~~~~~~~~~~~~~
+
+Instead of leveraging the full ``deploy.sh``, one could execute the ``STATE``
+files one by one (or partially) from the ``cfg01``.
+
+However, this requires a better understanding of how the list of ``STATE``
+files to be executed is constructed for a specific scenario, depending on the
+deployment type and the cluster having baremetal nodes, implemented in:
 
+* ``mcp/config/scenario/defaults.yaml.j2``
+* ``mcp/config/scenario/<scenario-name>.yaml``
 
-**Instructions**
+e.g.: For the example presented above (baremetal with ``VCP``,
+``os-nosdn-nofeature-ha``), the list of ``STATE`` files would be:
 
+* ``virtual_init``
+* ``maas``
+* ``baremetal_init``
+* ``virtual_control_plane``
+* ``openstack_ha``
+* ``networks``
 
-#. Create a new directory at any location
+To execute one (or more) of the remaining ``STATE`` files after a failure:
 
-    .. code-block:: bash
+.. code-block:: console
 
-        $ mkdir -p modeler
+    jenkins@jumpserver:~$ docker exec -it fuel bash
+    root@cfg01:~$ cd ~/fuel/mcp/config/states
+    root@cfg01:~/fuel/mcp/config/states$ ./openstack_ha
+    root@cfg01:~/fuel/mcp/config/states$ CI_DEBUG=true ./networks
 
+For even finer granularity, one can also run the commands in a ``STATE`` file
+one by one manually, e.g. if the execution failed applying the ``rabbitmq``
+sls:
 
-#. Place fuel repo in the above directory
+.. code-block:: console
 
-    .. code-block:: bash
+    root@cfg01:~$ salt -I 'rabbitmq:server' state.sls rabbitmq
 
-        $ cd modeler
-        $ git clone https://gerrit.opnfv.org/gerrit/fuel && cd fuel
+Exploring the Cloud with Salt
+=============================
 
+To gather information about the cloud, the salt commands can be used.
+It is based around a master-minion idea where the salt-master pushes config to
+the minions to execute actions.
 
-#. Create a container and mount the above host directory
+For example tell salt to execute a ping to ``8.8.8.8`` on all the nodes.
 
-    .. code-block:: bash
+.. code-block:: console
+
+    root@cfg01:~$ salt "*" network.ping 8.8.8.8
+                       ^^^                       target
+                           ^^^^^^^^^^^^          function to execute
+                                        ^^^^^^^  argument passed to the function
+
+.. TIP::
+
+    Complex filters can be done to the target like compound queries or node roles.
+
+For more information about Salt see the :ref:`fuel_userguide_references`
+section.
+
+Some examples are listed below. Note that these commands are issued from Salt
+master as ``root`` user.
+
+View the IPs of All the Components
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: console
+
+    root@cfg01:~$ salt "*" network.ip_addrs
+    cfg01.mcp-odl-ha.local:
+       - 10.20.0.2
+       - 172.16.10.100
+    mas01.mcp-odl-ha.local:
+       - 10.20.0.3
+       - 172.16.10.3
+       - 192.168.11.3
+    .........................
+
+View the Interfaces of All the Components and Put the Output in a ``yaml`` File
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: console
+
+    root@cfg01:~$ salt "*" network.interfaces --out yaml --output-file interfaces.yaml
+    root@cfg01:~# cat interfaces.yaml
+    cfg01.mcp-odl-ha.local:
+     enp1s0:
+       hwaddr: 52:54:00:72:77:12
+       inet:
+       - address: 10.20.0.2
+         broadcast: 10.20.0.255
+         label: enp1s0
+         netmask: 255.255.255.0
+       inet6:
+       - address: fe80::5054:ff:fe72:7712
+         prefixlen: '64'
+         scope: link
+       up: true
+    .........................
+
+View Installed Packages on MaaS Node
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: console
+
+    root@cfg01:~# salt "mas*" pkg.list_pkgs
+    mas01.mcp-odl-ha.local:
+        ----------
+        accountsservice:
+            0.6.40-2ubuntu11.3
+        acl:
+            2.2.52-3
+        acpid:
+            1:2.0.26-1ubuntu2
+        adduser:
+            3.113+nmu3ubuntu4
+        anerd:
+            1
+    .........................
+
+Execute Any Linux Command on All Nodes (e.g. ``ls /var/log``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: console
+
+    root@cfg01:~# salt "*" cmd.run 'ls /var/log'
+    cfg01.mcp-odl-ha.local:
+       alternatives.log
+       apt
+       auth.log
+       boot.log
+       btmp
+       cloud-init-output.log
+       cloud-init.log
+    .........................
+
+Execute Any Linux Command on Nodes Using Compound Queries Filter
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: console
+
+    root@cfg01:~# salt -C '* and cfg01*' cmd.run 'ls /var/log'
+    cfg01.mcp-odl-ha.local:
+       alternatives.log
+       apt
+       auth.log
+       boot.log
+       btmp
+       cloud-init-output.log
+       cloud-init.log
+    .........................
+
+Execute Any Linux Command on Nodes Using Role Filter
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: console
+
+    root@cfg01:~# salt -I 'nova:compute' cmd.run 'ls /var/log'
+    cmp001.mcp-odl-ha.local:
+       alternatives.log
+       apache2
+       apt
+       auth.log
+       btmp
+       ceilometer
+       cinder
+       cloud-init-output.log
+       cloud-init.log
+    .........................
 
-        $ docker run --privileged -it -v <absolute_path>/modeler:/host ubuntu bash
+Accessing Openstack
+===================
+
+Once the deployment is complete, Openstack CLI is accessible from controller
+VMs (``ctl01`` ... ``ctl03``).
 
+Openstack credentials are at ``/root/keystonercv3``.
 
-#. Install all the required packages inside the container.
+.. code-block:: console
 
-    .. code-block:: bash
+    root@ctl01:~# source keystonercv3
+    root@ctl01:~# openstack image list
+    +--------------------------------------+-----------------------------------------------+--------+
+    | ID                                   | Name                                          | Status |
+    +======================================+===============================================+========+
+    | 152930bf-5fd5-49c2-b3a1-cae14973f35f | CirrosImage                                   | active |
+    | 7b99a779-78e4-45f3-9905-64ae453e3dcb | Ubuntu16.04                                   | active |
+    +--------------------------------------+-----------------------------------------------+--------+
 
-        $ apt-get update
-        $ apt-get install -y npm nodejs
-        $ npm install -g reclass-doc
-        $ cd /host/fuel/mcp/reclass
-        $ ln -s /usr/bin/nodejs /usr/bin/node
-        $ reclass-doc --output /host /host/fuel/mcp/reclass
+The OpenStack Dashboard, Horizon, is available at ``http://<proxy public VIP>``.
+The administrator credentials are ``admin``/``opnfv_secret``.
+
+.. figure:: img/horizon_login.png
+    :width: 60%
+    :align: center
+
+A full list of IPs/services is available at ``<proxy public VIP>:8090`` for
+``baremetal`` deploys.
+
+.. figure:: img/salt_services_ip.png
+    :width: 60%
+    :align: center
+
+Guest Operating System Support
+==============================
+
+There are a number of possibilities regarding the guest operating systems
+which can be spawned on the nodes.
+The current system spawns virtual machines for VCP VMs on the KVM nodes and VMs
+requested by users in OpenStack compute nodes. Currently the system supports
+the following ``UEFI``-images for the guests:
+
++------------------+-------------------+--------------------+
+| OS name          | ``x86_64`` status | ``aarch64`` status |
++==================+===================+====================+
+| Ubuntu 17.10     | untested          | Full support       |
++------------------+-------------------+--------------------+
+| Ubuntu 16.04     | Full support      | Full support       |
++------------------+-------------------+--------------------+
+| Ubuntu 14.04     | untested          | Full support       |
++------------------+-------------------+--------------------+
+| Fedora atomic 27 | untested          | Full support       |
++------------------+-------------------+--------------------+
+| Fedora cloud 27  | untested          | Full support       |
++------------------+-------------------+--------------------+
+| Debian           | untested          | Full support       |
++------------------+-------------------+--------------------+
+| Centos 7         | untested          | Not supported      |
++------------------+-------------------+--------------------+
+| Cirros 0.3.5     | Full support      | Full support       |
++------------------+-------------------+--------------------+
+| Cirros 0.4.0     | Full support      | Full support       |
++------------------+-------------------+--------------------+
+
+The above table covers only ``UEFI`` images and implies ``OVMF``/``AAVMF``
+firmware on the host. An ``x86_64`` deployment also supports ``non-UEFI``
+images, however that choice is up to the underlying hardware and the
+administrator to make.
+
+The images for the above operating systems can be found in their respective
+websites.
+
+OpenStack Storage
+=================
 
+OpenStack Cinder is the project behind block storage in OpenStack and OPNFV
+Fuel supports LVM out of the box.
 
-#. View the results from the host by using a browser. The file to open should be now at modeler/index.html
+By default ``x86_64`` supports 2 additional block storage devices, while
+``aarch64`` supports only one.
 
-   .. figure:: img/reclass_doc.png
+More devices can be supported if the OS-image created has additional
+properties allowing block storage devices to be spawned as ``SCSI`` drives.
+To do this, add the properties below to the server:
 
+.. code-block:: console
+
+    root@ctl01:~$ openstack image set --property hw_disk_bus='scsi' \
+                                      --property hw_scsi_model='virtio-scsi' \
+                                      <image>
+
+The choice regarding which bus to use for the storage drives is an important
+one. ``virtio-blk`` is the default choice for OPNFV Fuel, which attaches the
+drives in ``/dev/vdX``. However, since we want to be able to attach a
+larger number of volumes to the virtual machines, we recommend the switch to
+``SCSI`` drives which are attached in ``/dev/sdX`` instead.
+
+``virtio-scsi`` is a little worse in terms of performance but the ability to
+add a larger number of drives combined with added features like ZFS, Ceph et
+al, leads us to suggest the use of ``virtio-scsi`` in OPNFV Fuel for both
+architectures.
+
+More details regarding the differences and performance of ``virtio-blk`` vs
+``virtio-scsi`` are beyond the scope of this manual but can be easily found
+in other sources online like `VirtIO SCSI`_ or `VirtIO performance`_.
+
+Additional configuration for configuring images in OpenStack can be found in
+the OpenStack Glance documentation.
+
+OpenStack Endpoints
+===================
+
+For each OpenStack service three endpoints are created: ``admin``, ``internal``
+and ``public``.
+
+.. code-block:: console
+
+    ubuntu@ctl01:~$ openstack endpoint list --service keystone
+    +----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------------+
+    | ID                               | Region    | Service Name | Service Type | Enabled | Interface | URL                          |
+    +----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------------+
+    | 008fec57922b4e9e8bf02c770039ae77 | RegionOne | keystone     | identity     | True    | internal  | http://172.16.10.26:5000/v3  |
+    | 1a1f3c3340484bda9ef7e193f50599e6 | RegionOne | keystone     | identity     | True    | admin     | http://172.16.10.26:35357/v3 |
+    | b0a47d42d0b6491b995d7e6230395de8 | RegionOne | keystone     | identity     | True    | public    | https://10.0.15.2:5000/v3    |
+    +----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------------+
+
+MCP sets up all Openstack services to talk to each other over unencrypted
+connections on the internal management network. All admin/internal endpoints
+use plain http, while the public endpoints are https connections terminated
+via nginx at the ``VCP`` proxy VMs.
+
+To access the public endpoints an SSL certificate has to be provided. For
+convenience, the installation script will copy the required certificate
+to the ``cfg01`` node at ``/etc/ssl/certs/os_cacert``.
+
+Copy the certificate from the ``cfg01`` node to the client that will access
+the https endpoints and place it under ``/etc/ssl/certs/``.
+The SSL connection will be established automatically after.
+
+.. code-block:: console
+
+    jenkins@jumpserver:~$ ssh -o StrictHostKeyChecking=no -i /var/lib/opnfv/mcp.rsa -l ubuntu 10.20.0.2 \
+      "cat /etc/ssl/certs/os_cacert" | sudo tee /etc/ssl/certs/os_cacert
+
+Reclass Model Viewer Tutorial
+=============================
+
+In order to get a better understanding of the ``reclass`` model Fuel uses, the
+`reclass-doc`_ tool can be used to visualise the ``reclass`` model.
+
+To avoid installing packages on the ``jumpserver`` or another host, the
+``cfg01`` Docker container can be used. Since the ``fuel`` git repository
+located on the ``jumpserver`` is already mounted inside ``cfg01`` container,
+the results can be visualized using a web browser on the ``jumpserver`` at the
+end of the procedure.
+
+.. code-block:: console
+
+    jenkins@jumpserver:~$ docker exec -it fuel bash
+    root@cfg01:~$ apt-get update
+    root@cfg01:~$ apt-get install -y npm nodejs
+    root@cfg01:~$ npm install -g reclass-doc
+    root@cfg01:~$ ln -s /usr/bin/nodejs /usr/bin/node
+    root@cfg01:~$ reclass-doc --output ~/fuel/mcp/reclass/modeler \
+                                       ~/fuel/mcp/reclass
+
+The generated documentation should be available on the ``jumpserver`` inside
+``fuel`` git repo subpath ``mcp/reclass/modeler/index.html``.
+
+.. figure:: img/reclass_doc.png
+    :width: 60%
+    :align: center
 
 .. _fuel_userguide_references:
 
-==========
 References
 ==========
 
-1) :ref:`fuel-release-installation-label`
-2) `Saltstack Documentation <https://docs.saltstack.com/en/latest/topics/>`_
-3) `Saltstack Formulas <https://salt-formulas.readthedocs.io/en/latest/>`_
-4) `Virtio performance <https://mpolednik.github.io/2017/01/23/virtio-blk-vs-virtio-scsi/>`_
-5) `Virtio SCSI <https://www.ovirt.org/develop/release-management/features/storage/virtio-scsi/>`_
+#. :ref:`OPNFV Fuel Installation Instruction <fuel-installation>`
+#. `Saltstack Documentation`_
+#. `Saltstack Formulas`_
+#. `VirtIO performance`_
+#. `VirtIO SCSI`_
+
+.. _`Saltstack Documentation`: https://docs.saltstack.com/en/latest/topics/
+.. _`Saltstack Formulas`: https://salt-formulas.readthedocs.io/en/latest/
+.. _`VirtIO performance`: https://mpolednik.github.io/2017/01/23/virtio-blk-vs-virtio-scsi/
+.. _`VirtIO SCSI`: https://www.ovirt.org/develop/release-management/features/storage/virtio-scsi/
+.. _`reclass-doc`: https://github.com/jirihybek/reclass-doc