docker: new hybrid deployment architecture and configuration

This patch implements a new docker deployment architecture that
should us to install docker services in a stepwise manner alongside
of baremetal puppet services. This works by using Yaql to select
docker specific services (docker/services/*.yaml) vs the puppet
specific ones and then applying the selected Json to relevant Heat
software deployments for docker and baremetal puppet in a stepwise
fashion.

Additionally the new architecture
leverages new composable services interfaces from Newton to
allow configuration of per-service container configuration
sets (directories that are bind mounted into kolla containers) by
using the Kolla containers themselves. It does this by spinning up
a throw away "configuration only" version of the container being
configured itself, then running the puppet apply in that container and
copying the generated config files into /var/lib/config-data. This
avoids having to install all of the OpenStack dependency packages
in the heat-agent-container itself (our previous approach) and should
allow us to configure a much wider variety of container config files
that would otherwise be impossible with the previous shared approach.

The new approach (combined) should allow us to configure containers in
both the undercloud and overcloud and incrementally add CI coverage to
services as we containerize them.

Co-Authored-By: Martin André <m.andre@redhat.com>
Co-Authored-By: Ian Main <imain@redhat.com>
Co-Authored-By: Flavio Percoco <flavio@redhat.com>

Change-Id: Ibcff99f03e6751fbf3197adefd5d344178b71fc2

5 files changed, 154 insertions, 99 deletions

diff --git a/docker/services/README.rst b/docker/services/README.rst
index 60719bfc..edaa5ee9 100644
--- a/docker/services/README.rst
+++ b/docker/services/README.rst
@@ -1,65 +1,104 @@
-========
-services
-========
+===============
+Docker Services
+===============
 
-A TripleO nested stack Heat template that encapsulates generic configuration
-data to configure a specific service. This generally includes everything
-needed to configure the service excluding the local bind ports which
-are still managed in the per-node role templates directly (controller.yaml,
-compute.yaml, etc.). All other (global) service settings go into
-the puppet/service templates.
+TripleO docker services are currently built on top of the puppet services.
+To do this each of the docker services includes the output of the
+t-h-t puppet/service templates where appropriate.
 
-Input Parameters
-----------------
+In general global docker specific service settings should reside in these
+templates (templates in the docker/services directory.) The required and
+optional items are specified in the docker settings section below.
 
-Each service may define its own input parameters and defaults.
-Operators will use the parameter_defaults section of any Heat
-environment to set per service parameters.
+If you are adding a config setting that applies to both docker and
+baremetal that setting should (so long as we use puppet) go into the
+puppet/services templates themselves.
 
-Config Settings
----------------
+Building Kolla Images
+---------------------
+
+TripleO currently relies on Kolla docker containers. Kolla supports container
+customization and we are making use of this feature within TripleO to inject
+puppet (our configuration tool of choice) into the Kolla base images. To
+build Kolla images for TripleO adjust your kolla config to build your
+centos base image with puppet using the example below:
+
+.. code-block::
+
+$ cat template-overrides.j2
+{% extends parent_template %}
+{% set base_centos_binary_packages_append = ['puppet'] %}
 
-Each service may define a config_settings output variable which returns
-Hiera settings to be configured.
+kolla-build --base centos --template-override template-overrides.j2
 
-Steps
------
+..
 
+
+Docker settings
+---------------
 Each service may define an output variable which returns a puppet manifest
 snippet that will run at each of the following steps. Earlier manifests
 are re-asserted when applying latter ones.
 
- * config_settings: Custom hiera settings for this service. These are
-   used to generate configs.
+ * config_settings: This setting is generally inherited from the
+   puppet/services templates and only need to be appended
+   to on accasion if docker specific config settings are required.
+
+ * step_config: This setting controls the manifest that is used to
+   create docker config files via puppet. The puppet tags below are
+   used along with this manifest to generate a config directory for
+   this container.
 
  * kolla_config: Contains YAML that represents how to map config files
    into the kolla container. This config file is typically mapped into
    the container itself at the /var/lib/kolla/config_files/config.json
    location and drives how kolla's external config mechanisms work.
 
- * step_config: A puppet manifest that is used to step through the deployment
-   sequence. Each sequence is given a "step" (via hiera('step') that provides
-   information for when puppet classes should activate themselves.
+ * docker_image: The full name of the docker image that will be used.
 
- * docker_compose:
+ * docker_config: Data that is passed to the docker-cmd hook to configure
+   a container, or step of containers at each step. See the available steps
+   below and the related docker-cmd hook documentation in the heat-agents
+   project.
 
- * container_name:
+ * puppet_tags: Puppet resource tag names that are used to generate config
+   files with puppet. Only the named config resources are used to generate
+   a config file. Any service that specifies tags will have the default
+   tags of 'file,concat,file_line' appended to the setting.
+   Example: keystone_config
 
- * volumes:
+ * config_volume: The name of the volume (directory) where config files
+   will be generated for this service. Use this as the location to
+   bind mount into the running Kolla container for configuration.
 
-Steps correlate to the following:
-
-   1) Service configuration generation with puppet.
-
-   2) Early Openstack Service setup (database init?)
-
-   3) Early containerized networking services startup (OVS)
+ * config_image: The name of the docker image that will be used for
+   generating configuration files. This is often the same value as
+   'docker_image' above but some containers share a common set of
+   config files which are generated in a common base container.
 
-   4) Network configuration
+Docker steps
+------------
+Similar to baremetal docker containers are brought up in a stepwise manner.
+The current architecture supports bringing up baremetal services alongside
+of containers. For each step the baremetal puppet manifests are executed
+first and then any docker containers are brought up afterwards.
 
-   5) General OpenStack Services
-
-   6) Service activation (Pacemaker)
-
-   7) Fencing (Pacemaker)
+Steps correlate to the following:
 
+   Pre) Containers config files generated per hiera settings.
+   1) Load Balancer configuration baremetal
+     a) step 1 baremetal
+     b) step 1 containers
+   2) Core Services (Database/Rabbit/NTP/etc.)
+     a) step 2 baremetal
+     b) step 2 containers
+   3) Early Openstack Service setup (Ringbuilder, etc.)
+     a) step 3 baremetal
+     b) step 3 containers
+   4) General OpenStack Services
+     a) step 4 baremetal
+     b) step 4 containers
+     c) Keystone containers post initialization (tenant,service,endpoint creation)
+   5) Service activation (Pacemaker)
+     a) step 5 baremetal
+     b) step 5 containers
diff --git a/docker/services/neutron-ovs-agent.yaml b/docker/services/neutron-ovs-agent.yaml
index 0a061f6c..ab99da5e 100644
--- a/docker/services/neutron-ovs-agent.yaml
+++ b/docker/services/neutron-ovs-agent.yaml
@@ -10,7 +10,7 @@ parameters:
     type: string
   DockerOpenvswitchImage:
     description: image
-    default: 'centos-binary-neutron-openvswitch-agent'
+    default: 'centos-binary-neutron-openvswitch-agent:latest'
     type: string
   ServiceNetMap:
     default: {}
@@ -32,53 +32,53 @@ resources:
   NeutronOvsAgentBase:
     type: ../../puppet/services/neutron-ovs-agent.yaml
     properties:
+      EndpointMap: {get_param: EndpointMap}
       ServiceNetMap: {get_param: ServiceNetMap}
       DefaultPasswords: {get_param: DefaultPasswords}
-      EndpointMap: {get_param: EndpointMap}
 
 outputs:
   role_data:
     description: Role data for Neutron openvswitch service
     value:
+      service_name: {get_attr: [NeutronOvsAgentBase, role_data, service_name]}
       config_settings: {get_attr: [NeutronOvsAgentBase, role_data, config_settings]}
       step_config: {get_attr: [NeutronOvsAgentBase, role_data, step_config]}
+      docker_image: &neutron_ovs_agent_image
+        list_join:
+        - '/'
+        - [ {get_param: DockerNamespace}, {get_param: DockerOpenvswitchImage} ]
       puppet_tags: neutron_config,neutron_agent_ovs,neutron_plugin_ml2
+      config_volume: neutron
+      config_image: *neutron_ovs_agent_image
       kolla_config:
-        /var/lib/etc-data/json-config/neutron-openvswitch-agent.json:
-           command: /usr/bin/neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-file /etc/neutron/plugins/ml2/ml2_conf.ini
+        /var/lib/kolla/config_files/neutron-openvswitch-agent.json:
+           command: /usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-file /etc/neutron/plugins/ml2/ml2_conf.ini
            config_files:
            - dest: /etc/neutron/neutron.conf
              owner: neutron
              perm: '0600'
-             source: /var/lib/kolla/config_files/neutron.conf
+             source: /var/lib/kolla/config_files/src/etc/neutron/neutron.conf
            - dest: /etc/neutron/plugins/ml2/openvswitch_agent.ini
              owner: neutron
              perm: '0600'
-             source: /var/lib/kolla/config_files/openvswitch_agent.ini
+             source: /var/lib/kolla/config_files/src/etc/neutron/plugins/ml2/openvswitch_agent.ini
            - dest: /etc/neutron/plugins/ml2/ml2_conf.ini
              owner: neutron
              perm: '0600'
-             source: /var/lib/kolla/config_files/ml2_conf.ini
+             source: /var/lib/kolla/config_files/src/etc/neutron/plugins/ml2/ml2_conf.ini
       docker_config:
-        step_1:
+        step_4:
           neutronovsagent:
-            image:
-              list_join:
-              - '/'
-              - [ {get_param: DockerNamespace}, {get_param: DockerOpenvswitchImage} ]
+            image: *neutron_ovs_agent_image
             net: host
             pid: host
             privileged: true
             restart: always
             volumes:
-              - /var/lib/etc-data/json-config/neutron-openvswitch-agent.json:/var/lib/kolla/config_files/config.json
-              - /var/lib/etc-data/neutron/neutron.conf:/var/lib/kolla/config_files/neutron.conf:ro
-              - /var/lib/etc-data/neutron/plugins/ml2/ml2_conf.ini:/var/lib/kolla/config_files/ml2_conf.ini:ro
-              - /var/lib/etc-data/neutron/plugins/ml2/openvswitch_agent.ini:/var/lib/kolla/config_files/openvswitch_agent.ini:ro
+              - /var/lib/kolla/config_files/neutron-openvswitch-agent.json:/var/lib/kolla/config_files/config.json:ro
+              - /var/lib/config-data/neutron:/var/lib/kolla/config_files/src:ro
               - /etc/localtime:/etc/localtime:ro
               - /lib/modules:/lib/modules:ro
               - /run:/run
-              - logs:/var/log/kolla/
             environment:
               - KOLLA_CONFIG_STRATEGY=COPY_ALWAYS
-        step_2: {}
diff --git a/docker/services/nova-compute.yaml b/docker/services/nova-compute.yaml
index e765609e..8eebc397 100644
--- a/docker/services/nova-compute.yaml
+++ b/docker/services/nova-compute.yaml
@@ -10,7 +10,7 @@ parameters:
     type: string
   DockerNovaComputeImage:
     description: image
-    default: 'centos-binary-nova-compute'
+    default: 'centos-binary-nova-compute:latest'
     type: string
   ServiceNetMap:
     default: {}
@@ -29,53 +29,57 @@ parameters:
 
 resources:
 
+
   NovaComputeBase:
     type: ../../puppet/services/nova-compute.yaml
     properties:
       EndpointMap: {get_param: EndpointMap}
+      ServiceNetMap: {get_param: ServiceNetMap}
+      DefaultPasswords: {get_param: DefaultPasswords}
 
 outputs:
   role_data:
     description: Role data for the Nova Compute service.
     value:
+      service_name: {get_attr: [NovaComputeBase, role_data, service_name]}
       config_settings: {get_attr: [NovaComputeBase, role_data, config_settings]}
       step_config: {get_attr: [NovaComputeBase, role_data, step_config]}
       puppet_tags: nova_config,nova_paste_api_ini
+      docker_image: &nova_compute_image
+        list_join:
+        - '/'
+        - [ {get_param: DockerNamespace}, {get_param: DockerNovaComputeImage} ]
+      config_volume: nova_libvirt
+      config_image: *nova_compute_image
       kolla_config:
-        /var/lib/etc-data/json-config/nova-compute.json:
+        /var/lib/kolla/config_files/nova-compute.json:
            command: /usr/bin/nova-compute --config-file /etc/nova/nova.conf --config-file /etc/nova/rootwrap.conf
            config_files:
            - dest: /etc/nova/nova.conf
              owner: nova
              perm: '0600'
-             source: /var/lib/kolla/config_files/nova.conf
+             source: /var/lib/kolla/config_files/src/etc/nova/nova.conf
            - dest: /etc/nova/rootwrap.conf
              owner: nova
              perm: '0600'
-             source: /var/lib/kolla/config_files/rootwrap.conf
+             source: /var/lib/kolla/config_files/src/etc/nova/rootwrap.conf
       docker_config:
-        step_1:
+        step_4:
           novacompute:
-            image:
-              list_join:
-              - '/'
-              - [ {get_param: DockerNamespace}, {get_param: DockerNovaComputeImage} ]
+            image: *nova_compute_image
             net: host
             privileged: true
             user: root
             restart: always
             volumes:
-              - /var/lib/etc-data/json-config/nova-compute.json:/var/lib/kolla/config_files/config.json
-              - /var/lib/etc-data/nova/nova.conf:/var/lib/kolla/config_files/nova.conf:ro
-              - /var/lib/etc-data/nova/rootwrap.conf:/var/lib/kolla/config_files/rootwrap.conf:ro
+              - /var/lib/kolla/config_files/nova-compute.json:/var/lib/kolla/config_files/config.json:ro
+              - /var/lib/config-data/nova_libvirt:/var/lib/kolla/config_files/src:ro
+              - /dev:/dev
+              - /etc/iscsi:/etc/iscsi
               - /etc/localtime:/etc/localtime:ro
               - /lib/modules:/lib/modules:ro
               - /run:/run
-              - /dev:/dev
-              - logs:/var/log/kolla/
-              - /etc/iscsi:/etc/iscsi
+              - /var/lib/nova:/var/lib/nova
               - libvirtd:/var/lib/libvirt
-              - nova_compute:/var/lib/nova/
             environment:
              - KOLLA_CONFIG_STRATEGY=COPY_ALWAYS
-        step_2: {}
diff --git a/docker/services/nova-libvirt.yaml b/docker/services/nova-libvirt.yaml
index 004d624a..d6e7dc76 100644
--- a/docker/services/nova-libvirt.yaml
+++ b/docker/services/nova-libvirt.yaml
@@ -10,7 +10,13 @@ parameters:
     type: string
   DockerLibvirtImage:
     description: image
-    default: 'centos-binary-libvirt'
+    default: 'centos-binary-nova-libvirt:latest'
+    type: string
+  # we configure libvirt via the nova-compute container due to coupling
+  # in the puppet modules
+  DockerNovaComputeImage:
+    description: image
+    default: 'centos-binary-nova-compute:latest'
     type: string
   ServiceNetMap:
     default: {}
@@ -33,50 +39,54 @@ resources:
     type: ../../puppet/services/nova-libvirt.yaml
     properties:
       EndpointMap: {get_param: EndpointMap}
+      ServiceNetMap: {get_param: ServiceNetMap}
+      DefaultPasswords: {get_param: DefaultPasswords}
 
 outputs:
   role_data:
     description: Role data for the Libvirt service.
     value:
+      service_name: {get_attr: [NovaLibvirtBase, role_data, service_name]}
       config_settings: {get_attr: [NovaLibvirtBase, role_data, config_settings]}
       step_config: {get_attr: [NovaLibvirtBase, role_data, step_config]}
+      docker_image: &libvirt_image
+        list_join:
+        - '/'
+        - [ {get_param: DockerNamespace}, {get_param: DockerLibvirtImage} ]
       puppet_tags: nova_config
+      config_volume: nova_libvirt
+      config_image:
+        list_join:
+        - '/'
+        - [ {get_param: DockerNamespace}, {get_param: DockerNovaComputeImage} ]
       kolla_config:
-        /var/lib/etc-data/json-config/nova-libvirt.json:
+        /var/lib/kolla/config_files/nova-libvirt.json:
            command: /usr/sbin/libvirtd --config /etc/libvirt/libvirtd.conf
            config_files:
            - dest: /etc/libvirt/libvirtd.conf
              owner: root
              perm: '0644'
-             source: /var/lib/kolla/config_files/libvirtd.conf
+             source: /var/lib/kolla/config_files/src/etc/libvirt/libvirtd.conf
       docker_config:
-        step_1:
+        step_3:
           nova_libvirt:
-            image:
-              list_join:
-                - '/'
-                - [ {get_param: DockerNamespace}, {get_param: DockerLibvirtImage} ]
+            image: *libvirt_image
             net: host
             pid: host
             privileged: true
             restart: always
             volumes:
-              - /var/lib/etc-data/json-config/nova-libvirt.json:/var/lib/kolla/config_files/config.json
-              - /var/lib/etc-data/libvirt/libvirtd.conf:/var/lib/kolla/config_files/libvirtd.conf
-              # NOTE(mandre) Ideally the qemu.conf file is mounted in
-              # /var/lib/kolla/config_files and copied to the right place but
-              # copy-json.py doesn't allow us to do that without appending the
-              # file as an additional config on the CLI
-              - /var/lib/etc-data/libvirt/qemu.conf:/etc/libvirt/qemu.conf:ro
+              - /var/lib/kolla/config_files/nova-libvirt.json:/var/lib/kolla/config_files/config.json:ro
+              - /var/lib/config-data/nova_libvirt:/var/lib/kolla/config_files/src:ro
+              - /dev:/dev
               - /etc/localtime:/etc/localtime:ro
               - /lib/modules:/lib/modules:ro
               - /run:/run
-              - /dev:/dev
               - /sys/fs/cgroup:/sys/fs/cgroup
-              - logs:/var/log/kolla/
+              - /var/lib/nova:/var/lib/nova
+              # Needed to use host's virtlogd
+              - /var/run/libvirt:/var/run/libvirt
               - libvirtd:/var/lib/libvirt
-              - nova_compute:/var/lib/nova/
               - nova_libvirt_qemu:/etc/libvirt/qemu
             environment:
               - KOLLA_CONFIG_STRATEGY=COPY_ALWAYS
-        step_2: {}
diff --git a/docker/services/services.yaml b/docker/services/services.yaml
index 8c31107f..8e899024 100644
--- a/docker/services/services.yaml
+++ b/docker/services/services.yaml
@@ -66,10 +66,12 @@ outputs:
       global_config_settings:
         {get_attr: [PuppetServices, role_data, global_config_settings]}
       step_config:
-        {get_attr: [PuppetServices, role_data, step_config]}
-      puppet_tags: {list_join: [",", {get_attr: [ServiceChain, role_data, puppet_tags]}]}
+        {get_attr: [ServiceChain, role_data, step_config]}
+      docker_image: {get_attr: [ServiceChain, role_data, docker_image]}
+      puppet_tags: {get_attr: [ServiceChain, role_data, puppet_tags]}
+      config_volume: {get_attr: [ServiceChain, role_data, config_volume]}
+      config_image: {get_attr: [ServiceChain, role_data, config_image]}
       kolla_config:
         map_merge: {get_attr: [ServiceChain, role_data, kolla_config]}
       docker_config:
-        step_1: {map_merge: {get_attr: [ServiceChain, role_data, docker_config, step_1]}}
-        step_2: {map_merge: {get_attr: [ServiceChain, role_data, docker_config, step_2]}}
+        {get_attr: [ServiceChain, role_data, docker_config]}