docker: new hybrid deployment architecture and configuration

This patch implements a new docker deployment architecture that
should us to install docker services in a stepwise manner alongside
of baremetal puppet services. This works by using Yaql to select
docker specific services (docker/services/*.yaml) vs the puppet
specific ones and then applying the selected Json to relevant Heat
software deployments for docker and baremetal puppet in a stepwise
fashion.

Additionally the new architecture
leverages new composable services interfaces from Newton to
allow configuration of per-service container configuration
sets (directories that are bind mounted into kolla containers) by
using the Kolla containers themselves. It does this by spinning up
a throw away "configuration only" version of the container being
configured itself, then running the puppet apply in that container and
copying the generated config files into /var/lib/config-data. This
avoids having to install all of the OpenStack dependency packages
in the heat-agent-container itself (our previous approach) and should
allow us to configure a much wider variety of container config files
that would otherwise be impossible with the previous shared approach.

The new approach (combined) should allow us to configure containers in
both the undercloud and overcloud and incrementally add CI coverage to
services as we containerize them.

Co-Authored-By: Martin André <m.andre@redhat.com>
Co-Authored-By: Ian Main <imain@redhat.com>
Co-Authored-By: Flavio Percoco <flavio@redhat.com>

Change-Id: Ibcff99f03e6751fbf3197adefd5d344178b71fc2

12 files changed, 552 insertions, 294 deletions

diff --git a/docker/copy-etc.sh b/docker/copy-etc.sh
deleted file mode 100644
index 1a6cd520..00000000
--- a/docker/copy-etc.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/bash
-echo "Copying agent container /etc to /var/lib/etc-data"
-cp -a /etc/* /var/lib/etc-data/
diff --git a/docker/create-config-dir.sh b/docker/create-config-dir.sh
new file mode 100644
index 00000000..1be1a56f
--- /dev/null
+++ b/docker/create-config-dir.sh
@@ -0,0 +1,6 @@
+#!/bin/bash
+# This is where we stack puppet configuration (for now)...
+mkdir -p /var/lib/config-data
+
+# This is the docker-puppet configs end in
+mkdir -p /var/lib/docker-puppet
diff --git a/docker/docker-puppet.py b/docker/docker-puppet.py
new file mode 100755
index 00000000..2d560819
--- /dev/null
+++ b/docker/docker-puppet.py
@@ -0,0 +1,210 @@
+#!/usr/bin/env python
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+# Shell script tool to run puppet inside of the given docker container image.
+# Uses the config file at /var/lib/docker-puppet/docker-puppet.json as a source for a JSON
+# array of [config_volume, puppet_tags, manifest, config_image, [volumes]] settings
+# that can be used to generate config files or run ad-hoc puppet modules
+# inside of a container.
+
+import json
+import os
+import subprocess
+import sys
+import tempfile
+
+
+# this is to match what we do in deployed-server
+def short_hostname():
+    subproc = subprocess.Popen(['hostname', '-s'],
+                               stdout=subprocess.PIPE,
+                               stderr=subprocess.PIPE)
+    cmd_stdout, cmd_stderr = subproc.communicate()
+    return cmd_stdout.rstrip()
+
+
+def pull_image(name):
+    print('Pulling image: %s' % name)
+    subproc = subprocess.Popen(['/usr/bin/docker', 'pull', name],
+                               stdout=subprocess.PIPE,
+                               stderr=subprocess.PIPE)
+    cmd_stdout, cmd_stderr = subproc.communicate()
+    print(cmd_stdout)
+    print(cmd_stderr)
+
+
+def rm_container(name):
+    print('Removing container: %s' % name)
+    subproc = subprocess.Popen(['/usr/bin/docker', 'rm', name],
+                               stdout=subprocess.PIPE,
+                               stderr=subprocess.PIPE)
+    cmd_stdout, cmd_stderr = subproc.communicate()
+    print(cmd_stdout)
+    print(cmd_stderr)
+
+
+config_file = os.environ.get('CONFIG', '/var/lib/docker-puppet/docker-puppet.json')
+print('docker-puppet')
+print('CONFIG: %s' % config_file)
+with open(config_file) as f:
+    json_data = json.load(f)
+
+# To save time we support configuring 'shared' services at the same
+# time. For example configuring all of the heat services
+# in a single container pass makes sense and will save some time.
+# To support this we merge shared settings together here.
+#
+# We key off of config_volume as this should be the same for a
+# given group of services.  We are also now specifying the container
+# in which the services should be configured.  This should match
+# in all instances where the volume name is also the same.
+
+configs = {}
+
+for service in json_data:
+    config_volume = service[0] or ''
+    puppet_tags = service[1] or ''
+    manifest = service[2] or ''
+    config_image = service[3] or ''
+    volumes = service[4] if len(service) > 4 else []
+
+    print('---------')
+    print('config_volume %s' % config_volume)
+    print('puppet_tags %s' % puppet_tags)
+    print('manifest %s' % manifest)
+    print('config_image %s' % config_image)
+    print('volumes %s' % volumes)
+    # We key off of config volume for all configs.
+    if config_volume in configs:
+        # Append puppet tags and manifest.
+        print("Existing service, appending puppet tags and manifest\n")
+        if puppet_tags:
+            configs[config_volume][1] = '%s,%s' % (configs[config_volume][1],
+                                                   puppet_tags)
+        if manifest:
+            configs[config_volume][2] = '%s\n%s' % (configs[config_volume][2],
+                                                    manifest)
+        if configs[config_volume][3] != config_image:
+            print("WARNING: Config containers do not match even though"
+                  " shared volumes are the same!\n")
+    else:
+        print("Adding new service\n")
+        configs[config_volume] = service
+
+print('Service compilation completed.\n')
+
+for config_volume in configs:
+
+    service = configs[config_volume]
+    puppet_tags = service[1] or ''
+    manifest = service[2] or ''
+    config_image = service[3] or ''
+    volumes = service[4] if len(service) > 4 else []
+
+    if puppet_tags:
+        puppet_tags = "file,file_line,concat,%s" % puppet_tags
+    else:
+        puppet_tags = "file,file_line,concat"
+
+    print('---------')
+    print('config_volume %s' % config_volume)
+    print('puppet_tags %s' % puppet_tags)
+    print('manifest %s' % manifest)
+    print('config_image %s' % config_image)
+    hostname = short_hostname()
+
+    with open('/var/lib/docker-puppet/docker-puppet.sh', 'w') as script_file:
+        os.chmod(script_file.name, 0755)
+        script_file.write("""#!/bin/bash
+        set -ex
+        mkdir -p /etc/puppet
+        cp -a /tmp/puppet-etc/* /etc/puppet
+        rm -Rf /etc/puppet/ssl # not in use and causes permission errors
+        echo '{"step": 6}' > /etc/puppet/hieradata/docker.json
+        TAGS=""
+        if [ -n "%(puppet_tags)s" ]; then
+            TAGS='--tags "%(puppet_tags)s"'
+        fi
+        FACTER_hostname=%(hostname)s FACTER_uuid=docker /usr/bin/puppet apply --verbose $TAGS /etc/config.pp
+
+        # Disables archiving
+        if [ -z "%(no_archive)s" ]; then
+            rm -Rf /var/lib/config-data/%(name)s
+
+            # copying etc should be enough for most services
+            mkdir -p /var/lib/config-data/%(name)s/etc
+            cp -a /etc/* /var/lib/config-data/%(name)s/etc/
+
+            if [ -d /root/ ]; then
+              cp -a /root/ /var/lib/config-data/%(name)s/root/
+            fi
+            if [ -d /var/lib/ironic/tftpboot/ ]; then
+              mkdir -p /var/lib/config-data/%(name)s/var/lib/ironic/
+              cp -a /var/lib/ironic/tftpboot/ /var/lib/config-data/%(name)s/var/lib/ironic/tftpboot/
+            fi
+            if [ -d /var/lib/ironic/httpboot/ ]; then
+              mkdir -p /var/lib/config-data/%(name)s/var/lib/ironic/
+              cp -a /var/lib/ironic/httpboot/ /var/lib/config-data/%(name)s/var/lib/ironic/httpboot/
+            fi
+
+            # apache services may files placed in /var/www/
+            if [ -d /var/www/ ]; then
+             mkdir -p /var/lib/config-data/%(name)s/var/www
+             cp -a /var/www/* /var/lib/config-data/%(name)s/var/www/
+            fi
+        fi
+        """ % {'puppet_tags': puppet_tags, 'name': config_volume,
+               'hostname': hostname,
+               'no_archive': os.environ.get('NO_ARCHIVE', '')})
+
+    with tempfile.NamedTemporaryFile() as tmp_man:
+        with open(tmp_man.name, 'w') as man_file:
+            man_file.write('include ::tripleo::packages\n')
+            man_file.write(manifest)
+
+        rm_container('docker-puppet-%s' % config_volume)
+        pull_image(config_image)
+
+        dcmd = ['/usr/bin/docker', 'run',
+                '--user', 'root',
+                '--name', 'docker-puppet-%s' % config_volume,
+                '--volume', '%s:/etc/config.pp:ro' % tmp_man.name,
+                '--volume', '/etc/puppet/:/tmp/puppet-etc/:ro',
+                '--volume', '/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro',
+                '--volume', '/var/lib/config-data/:/var/lib/config-data/:rw',
+                '--volume', 'tripleo_logs:/var/log/tripleo/',
+                '--volume', '/var/lib/docker-puppet/docker-puppet.sh:/var/lib/docker-puppet/docker-puppet.sh:ro']
+
+        for volume in volumes:
+            dcmd.extend(['--volume', volume])
+
+        dcmd.extend(['--entrypoint', '/var/lib/docker-puppet/docker-puppet.sh'])
+
+        env = {}
+        if os.environ.get('NET_HOST', 'false') == 'true':
+            print('NET_HOST enabled')
+            dcmd.extend(['--net', 'host', '--volume',
+                         '/etc/hosts:/etc/hosts:ro'])
+        dcmd.append(config_image)
+
+        subproc = subprocess.Popen(dcmd, stdout=subprocess.PIPE,
+                                   stderr=subprocess.PIPE, env=env)
+        cmd_stdout, cmd_stderr = subproc.communicate()
+        print(cmd_stdout)
+        print(cmd_stderr)
+        if subproc.returncode != 0:
+            print('Failed running docker-puppet.py for %s' % config_volume)
+            sys.exit(subproc.returncode)
+        else:
+            rm_container('docker-puppet-%s' % config_volume)
diff --git a/docker/firstboot/setup_docker_host.sh b/docker/firstboot/setup_docker_host.sh
new file mode 100755
index 00000000..b2287e91
--- /dev/null
+++ b/docker/firstboot/setup_docker_host.sh
@@ -0,0 +1,26 @@
+#!/bin/bash
+set -eux
+# TODO This would be better in puppet
+
+# TODO remove this when built image includes docker
+if [ ! -f "/usr/bin/docker" ]; then
+    yum -y install docker
+fi
+
+# NOTE(mandre) $docker_namespace_is_registry is not a bash variable but is
+# a place holder for text replacement done via heat
+if [ "$docker_namespace_is_registry" = "True" ]; then
+    /usr/bin/systemctl stop docker.service
+    # if namespace is used with local registry, trim all namespacing
+    trim_var=$docker_registry
+    registry_host="${trim_var%%/*}"
+    /bin/sed -i -r "s/^[# ]*INSECURE_REGISTRY *=.+$/INSECURE_REGISTRY='--insecure-registry $registry_host'/" /etc/sysconfig/docker
+fi
+
+# enable and start docker
+/usr/bin/systemctl enable docker.service
+/usr/bin/systemctl start docker.service
+
+# Disable libvirtd
+/usr/bin/systemctl disable libvirtd.service
+/usr/bin/systemctl stop libvirtd.service
diff --git a/docker/firstboot/install_docker_agents.yaml b/docker/firstboot/setup_docker_host.yaml
index 41a87406..2f258987 100644
--- a/docker/firstboot/install_docker_agents.yaml
+++ b/docker/firstboot/setup_docker_host.yaml
@@ -1,9 +1,6 @@
 heat_template_version: ocata
 
 parameters:
-  DockerAgentImage:
-    type: string
-    default: heat-docker-agents
   DockerNamespace:
     type: string
     default: tripleoupstream
@@ -17,22 +14,18 @@ resources:
     type: OS::Heat::MultipartMime
     properties:
       parts:
-      - config: {get_resource: install_docker_agents}
+      - config: {get_resource: setup_docker_host}
 
-  install_docker_agents:
+  setup_docker_host:
     type: OS::Heat::SoftwareConfig
     properties:
       group: script
       config:
         str_replace:
           params:
-            $agent_image:
-              list_join:
-              - '/'
-              - [ {get_param: DockerNamespace}, {get_param: DockerAgentImage} ]
             $docker_registry: {get_param: DockerNamespace}
             $docker_namespace_is_registry: {get_param: DockerNamespaceIsRegistry}
-          template: {get_file: ./start_docker_agents.sh}
+          template: {get_file: ./setup_docker_host.sh}
 
 outputs:
   OS::stack_id:
diff --git a/docker/firstboot/start_docker_agents.sh b/docker/firstboot/start_docker_agents.sh
deleted file mode 100755
index 1c5cc18d..00000000
--- a/docker/firstboot/start_docker_agents.sh
+++ /dev/null
@@ -1,69 +0,0 @@
-#!/bin/bash
-set -eux
-
-# TODO remove this when built image includes docker
-if [ ! -f "/usr/bin/docker" ]; then
-    yum -y install docker
-fi
-
-# Local docker registry 1.8
-# NOTE(mandre) $docker_namespace_is_registry is not a bash variable but is
-# a place holder for text replacement done via heat
-if [ "$docker_namespace_is_registry" = "True" ]; then
-    /usr/bin/systemctl stop docker.service
-    # if namespace is used with local registry, trim all namespacing
-    trim_var=$docker_registry
-    registry_host="${trim_var%%/*}"
-    /bin/sed -i -r "s/^[# ]*INSECURE_REGISTRY *=.+$/INSECURE_REGISTRY='--insecure-registry $registry_host'/" /etc/sysconfig/docker
-fi
-
-mkdir -p /var/lib/etc-data/json-config #FIXME: this should be a docker data container
-
-# NOTE(flaper87): Heat Agent required mounts
-AGENT_COMMAND_MOUNTS="\
--v /var/lib/etc-data:/var/lib/etc-data \
--v /run:/run \
--v /etc/hosts:/etc/hosts \
--v /etc:/host/etc \
--v /var/lib/dhclient:/var/lib/dhclient \
--v /var/lib/cloud:/var/lib/cloud \
--v /var/lib/heat-cfntools:/var/lib/heat-cfntools \
--v /var/lib/os-collect-config:/var/lib/os-collect-config \
--v /var/lib/os-apply-config-deployments:/var/lib/os-apply-config-deployments \
--v /var/lib/heat-config:/var/lib/heat-config \
--v /etc/sysconfig/docker:/etc/sysconfig/docker \
--v /etc/sysconfig/network-scripts:/etc/sysconfig/network-scripts \
--v /usr/lib64/libseccomp.so.2:/usr/lib64/libseccomp.so.2 \
--v /usr/bin/docker:/usr/bin/docker \
--v /usr/bin/docker-current:/usr/bin/docker-current \
--v /var/lib/os-collect-config:/var/lib/os-collect-config"
-
-# heat-docker-agents service
-cat <<EOF > /etc/systemd/system/heat-docker-agents.service
-[Unit]
-Description=Heat Docker Agent Container
-After=docker.service
-Requires=docker.service
-Before=os-collect-config.service
-Conflicts=os-collect-config.service
-
-[Service]
-User=root
-Restart=always
-ExecStartPre=-/usr/bin/docker rm -f heat-agents
-ExecStart=/usr/bin/docker run --name heat-agents --privileged --net=host \
-    $AGENT_COMMAND_MOUNTS \
-    --entrypoint=/usr/bin/os-collect-config $agent_image
-ExecStop=/usr/bin/docker stop heat-agents
-
-[Install]
-WantedBy=multi-user.target
-EOF
-
-# enable and start heat-docker-agents
-/usr/bin/systemctl enable heat-docker-agents.service
-/usr/bin/systemctl start --no-block heat-docker-agents.service
-
-# Disable libvirtd
-/usr/bin/systemctl disable libvirtd.service
-/usr/bin/systemctl stop libvirtd.service
diff --git a/docker/post.j2.yaml b/docker/post.j2.yaml
index dfa8ac2e..c125423d 100644
--- a/docker/post.j2.yaml
+++ b/docker/post.j2.yaml
@@ -1,3 +1,7 @@
+# certain initialization steps (run in a container) will occur
+# on the first role listed in the roles file
+{% set primary_role_name = roles[0].name -%}
+
 heat_template_version: ocata
 
 description: >
@@ -11,7 +15,6 @@ parameters:
   role_data:
     type: json
     description: Mapping of Role name e.g Controller to the per-role data
-
   DeployIdentifier:
     default: ''
     type: string
@@ -46,81 +49,171 @@ resources:
       input_values:
         update_identifier: {get_param: DeployIdentifier}
 
-  {{role.name}}Config:
-    type: OS::TripleO::{{role.name}}Config
+  {{role.name}}CreateConfigDir:
+    type: OS::Heat::SoftwareConfig
     properties:
-      StepConfig: {get_param: [role_data, {{role.name}}, step_config]}
-  {% if role.name.lower() == 'compute' %}
-      PuppetTags: {get_param: [role_data, {{role.name}}, puppet_tags]}
-  {% endif %}
+      group: script
+      config: {get_file: create-config-dir.sh}
 
-  # Step through a series of configuration steps
-  {{role.name}}Deployment_Step1:
-    type: OS::Heat::StructuredDeploymentGroup
-    depends_on: [{{role.name}}PreConfig, {{role.name}}ArtifactsDeploy]
+  {{role.name}}CreateConfigDirDeployment:
+    type: OS::Heat::SoftwareDeploymentGroup
     properties:
-      name: {{role.name}}Deployment_Step1
       servers: {get_param: [servers, {{role.name}}]}
-      config: {get_resource: {{role.name}}Config}
-      input_values:
-        step: 1
-        update_identifier: {get_param: DeployIdentifier}
+      config: {get_resource: {{role.name}}CreateConfigDir}
 
-  {{role.name}}Deployment_Step2:
-    type: OS::Heat::StructuredDeploymentGroup
-    depends_on:
-  {% for dep in roles %}
-      - {{dep.name}}Deployment_Step1
-  {% endfor %}
+  # this creates a JSON config file for our docker-puppet.py script
+  {{role.name}}GenPuppetConfig:
+    type: OS::Heat::StructuredConfig
+    properties:
+      group: json-file
+      config:
+        /var/lib/docker-puppet/docker-puppet.json:
+          yaql:
+            # select only services that have a non-null config_image with
+            # a step_config as well
+            expression:
+              $.data.config_volume.zip($.data.puppet_tags, $.data.step_config, $.data.config_image).where($[3] != null and $[1] != null)
+            data:
+              config_volume: {get_param: [role_data, {{role.name}}, config_volume]}
+              step_config: {get_param: [role_data, {{role.name}}, step_config]}
+              puppet_tags: {get_param: [role_data, {{role.name}}, puppet_tags]}
+              config_image: {get_param: [role_data, {{role.name}}, config_image]}
+
+  {{role.name}}GenPuppetDeployment:
+    type: OS::Heat::SoftwareDeploymentGroup
     properties:
-      name: {{role.name}}Deployment_Step2
       servers: {get_param: [servers, {{role.name}}]}
-      config: {get_resource: {{role.name}}Config}
-      input_values:
-        step: 2
-        update_identifier: {get_param: DeployIdentifier}
+      config: {get_resource: {{role.name}}GenPuppetConfig}
 
-  {{role.name}}Deployment_Step3:
-    type: OS::Heat::StructuredDeploymentGroup
-    depends_on:
-  {% for dep in roles %}
-      - {{dep.name}}Deployment_Step2
-  {% endfor %}
+  {{role.name}}GenerateConfig:
+    type: OS::Heat::SoftwareConfig
     properties:
-      name: {{role.name}}Deployment_Step3
+      group: script
+      config: {get_file: docker-puppet.py}
+
+  {{role.name}}GenerateConfigDeployment:
+    type: OS::Heat::SoftwareDeploymentGroup
+    depends_on: [{{role.name}}GenPuppetDeployment, {{role.name}}ArtifactsDeploy, {{role.name}}CreateConfigDirDeployment]
+    properties:
+      name: {{role.name}}GenerateConfigDeployment
       servers: {get_param: [servers, {{role.name}}]}
-      config: {get_resource: {{role.name}}Config}
+      config: {get_resource: {{role.name}}GenerateConfig}
+
+  {{role.name}}PuppetStepConfig:
+    type: OS::Heat::Value
+    properties:
+      type: string
+      value:
+        yaql:
+          expression:
+            # select 'step_config' only from services that do not have a docker_image
+            $.data.service_names.zip($.data.step_config, $.data.docker_image).where($[2] = null).where($[1] != null).select($[1]).join("\n")
+          data:
+            service_names: {get_param: [role_data, {{role.name}}, service_names]}
+            step_config: {get_param: [role_data, {{role.name}}, step_config]}
+            docker_image: {get_param: [role_data, {{role.name}}, docker_image]}
+
+  {{role.name}}DockerConfig:
+    type: OS::Heat::Value
+    properties:
+      type: json
+      value:
+        yaql:
+          expression:
+            # select 'docker_config' only from services that have a docker_image
+            $.data.service_names.zip($.data.docker_config, $.data.docker_image).where($[2] != null).select($[1]).reduce($1.mergeWith($2), {})
+          data:
+            service_names: {get_param: [role_data, {{role.name}}, service_names]}
+            docker_config: {get_param: [role_data, {{role.name}}, docker_config]}
+            docker_image: {get_param: [role_data, {{role.name}}, docker_image]}
+
+  {{role.name}}KollaJsonConfig:
+    type: OS::Heat::StructuredConfig
+    properties:
+      group: json-file
+      config:
+        {get_param: [role_data, {{role.name}}, kolla_config]}
+
+  {{role.name}}KollaJsonDeployment:
+    type: OS::Heat::SoftwareDeploymentGroup
+    properties:
+      name: {{role.name}}KollaJsonDeployment
+      config: {get_resource: {{role.name}}KollaJsonConfig}
+      servers: {get_param: [servers, {{role.name}}]}
+
+  # BEGIN BAREMETAL CONFIG STEPS
+
+  {% if role.name == 'Controller' %}
+  ControllerPrePuppet:
+    type: OS::TripleO::Tasks::ControllerPrePuppet
+    properties:
+      servers: {get_param: [servers, Controller]}
       input_values:
-        step: 3
         update_identifier: {get_param: DeployIdentifier}
+  {% endif %}
 
-  {{role.name}}Deployment_Step4:
+  {{role.name}}Config:
+    type: OS::TripleO::{{role.name}}Config
+    properties:
+      StepConfig: {get_attr: [{{role.name}}PuppetStepConfig, value]}
+
+  {% for step in range(1, 6) %}
+
+  {{role.name}}Deployment_Step{{step}}:
     type: OS::Heat::StructuredDeploymentGroup
+  {% if step == 1 %}
+    depends_on: [{{role.name}}PreConfig, {{role.name}}ArtifactsDeploy]
+  {% else %}
     depends_on:
-  {% for dep in roles %}
-      - {{dep.name}}Deployment_Step3
-  {% endfor %}
+    {% for dep in roles %}
+      - {{dep.name}}Deployment_Step{{step -1}}
+      - {{dep.name}}ContainersDeployment_Step{{step -1}}
+    {% endfor %}
+  {% endif %}
     properties:
-      name: {{role.name}}Deployment_Step4
+      name: {{role.name}}Deployment_Step{{step}}
       servers: {get_param: [servers, {{role.name}}]}
       config: {get_resource: {{role.name}}Config}
       input_values:
-        step: 4
+        step: {{step}}
         update_identifier: {get_param: DeployIdentifier}
 
-  {{role.name}}Deployment_Step5:
+  {% endfor %}
+  # END BAREMETAL CONFIG STEPS
+
+  # BEGIN CONTAINER CONFIG STEPS
+  {% for step in range(1, 6) %}
+
+  {{role.name}}ContainersConfig_Step{{step}}:
+    type: OS::Heat::StructuredConfig
+    properties:
+      group: docker-cmd
+      config:
+        {get_attr: [{{role.name}}DockerConfig, value, step_{{step}}]}
+
+  {{role.name}}ContainersDeployment_Step{{step}}:
     type: OS::Heat::StructuredDeploymentGroup
+  {% if step == 1 %}
     depends_on:
-  {% for dep in roles %}
-      - {{dep.name}}Deployment_Step4
-  {% endfor %}
+      - {{role.name}}PreConfig
+      - {{role.name}}KollaJsonDeployment
+      - {{role.name}}GenPuppetDeployment
+      - {{role.name}}GenerateConfigDeployment
+  {% else %}
+    depends_on:
+      {% for dep in roles %}
+        - {{dep.name}}ContainersDeployment_Step{{step -1}}
+        - {{dep.name}}Deployment_Step{{step}} # baremetal steps of the same level run first
+        - {{dep.name}}Deployment_Step{{step -1}}
+      {% endfor %}
+  {% endif %}
     properties:
-      name: {{role.name}}Deployment_Step5
+      name: {{role.name}}ContainersDeployment_Step{{step}}
       servers: {get_param: [servers, {{role.name}}]}
-      config: {get_resource: {{role.name}}Config}
-      input_values:
-        step: 5
-        update_identifier: {get_param: DeployIdentifier}
+      config: {get_resource: {{role.name}}ContainersConfig_Step{{step}}}
+
+  {% endfor %}
+  # END CONTAINER CONFIG STEPS
 
   {{role.name}}PostConfig:
     type: OS::TripleO::Tasks::{{role.name}}PostConfig
@@ -144,68 +237,15 @@ resources:
     properties:
         servers: {get_param: [servers, {{role.name}}]}
 
-  {% if role.name.lower() == 'compute' %}
-  CopyEtcConfig:
-    type: OS::Heat::SoftwareConfig
-    depends_on: {{role.name}}PostConfig
-    properties:
-      group: script
-      outputs:
-        - name: result
-      config: {get_file: ../docker/copy-etc.sh}
-
-  CopyEtcDeployment:
-    type: OS::Heat::SoftwareDeploymentGroup
-    properties:
-      name: CopyEtcDeployment
-      servers: {get_param: [servers, {{role.name}}]}
-      config: {get_resource: CopyEtcConfig}
-
-  {{role.name}}KollaJsonConfig:
-    type: OS::Heat::StructuredConfig
-    depends_on: CopyEtcDeployment
-    properties:
-      group: json-file
-      config:
-        {get_param: [role_data, {{role.name}}, kolla_config]}
-
-  {{role.name}}KollaJsonDeployment:
-    type: OS::Heat::SoftwareDeploymentGroup
-    properties:
-      name: {{role.name}}KollaJsonDeployment
-      config: {get_resource: {{role.name}}KollaJsonConfig}
-      servers: {get_param: [servers, {{role.name}}]}
-
-  {{role.name}}ContainersConfig_Step1:
-    type: OS::Heat::StructuredConfig
-    depends_on: {{role.name}}KollaJsonDeployment
-    properties:
-      group: docker-cmd
-      config:
-        {get_param: [role_data, {{role.name}}, docker_config, step_1]}
-
-  {{role.name}}ContainersConfig_Step2:
-    type: OS::Heat::StructuredConfig
-    depends_on: {{role.name}}KollaJsonDeployment
-    properties:
-      group: docker-cmd
-      config:
-        {get_param: [role_data, {{role.name}}, docker_config, step_2]}
-
-  {{role.name}}ContainersDeployment_Step1:
-    type: OS::Heat::StructuredDeploymentGroup
-    depends_on: [{{role.name}}PreConfig, {{role.name}}ArtifactsDeploy]
-    properties:
-      name: {{role.name}}ContainersDeployment_Step1
-      servers: {get_param: [servers, {{role.name}}]}
-      config: {get_resource: {{role.name}}ContainersConfig_Step1}
-
-  {{role.name}}ContainersDeployment_Step2:
-    type: OS::Heat::StructuredDeploymentGroup
-    depends_on: {{role.name}}ContainersDeployment_Step1
+  {% if role.name == 'Controller' %}
+  ControllerPostPuppet:
+    depends_on:
+      - ControllerExtraConfigPost
+    type: OS::TripleO::Tasks::ControllerPostPuppet
     properties:
-      name: {{role.name}}ContainersDeployment_Step2
-      servers: {get_param: [servers, {{role.name}}]}
-      config: {get_resource: {{role.name}}ContainersConfig_Step2}
+      servers: {get_param: [servers, Controller]}
+      input_values:
+        update_identifier: {get_param: DeployIdentifier}
   {% endif %}
+
 {% endfor %}
diff --git a/docker/services/README.rst b/docker/services/README.rst
index 60719bfc..edaa5ee9 100644
--- a/docker/services/README.rst
+++ b/docker/services/README.rst
@@ -1,65 +1,104 @@
-========
-services
-========
+===============
+Docker Services
+===============
 
-A TripleO nested stack Heat template that encapsulates generic configuration
-data to configure a specific service. This generally includes everything
-needed to configure the service excluding the local bind ports which
-are still managed in the per-node role templates directly (controller.yaml,
-compute.yaml, etc.). All other (global) service settings go into
-the puppet/service templates.
+TripleO docker services are currently built on top of the puppet services.
+To do this each of the docker services includes the output of the
+t-h-t puppet/service templates where appropriate.
 
-Input Parameters
-----------------
+In general global docker specific service settings should reside in these
+templates (templates in the docker/services directory.) The required and
+optional items are specified in the docker settings section below.
 
-Each service may define its own input parameters and defaults.
-Operators will use the parameter_defaults section of any Heat
-environment to set per service parameters.
+If you are adding a config setting that applies to both docker and
+baremetal that setting should (so long as we use puppet) go into the
+puppet/services templates themselves.
 
-Config Settings
----------------
+Building Kolla Images
+---------------------
+
+TripleO currently relies on Kolla docker containers. Kolla supports container
+customization and we are making use of this feature within TripleO to inject
+puppet (our configuration tool of choice) into the Kolla base images. To
+build Kolla images for TripleO adjust your kolla config to build your
+centos base image with puppet using the example below:
+
+.. code-block::
+
+$ cat template-overrides.j2
+{% extends parent_template %}
+{% set base_centos_binary_packages_append = ['puppet'] %}
 
-Each service may define a config_settings output variable which returns
-Hiera settings to be configured.
+kolla-build --base centos --template-override template-overrides.j2
 
-Steps
------
+..
 
+
+Docker settings
+---------------
 Each service may define an output variable which returns a puppet manifest
 snippet that will run at each of the following steps. Earlier manifests
 are re-asserted when applying latter ones.
 
- * config_settings: Custom hiera settings for this service. These are
-   used to generate configs.
+ * config_settings: This setting is generally inherited from the
+   puppet/services templates and only need to be appended
+   to on accasion if docker specific config settings are required.
+
+ * step_config: This setting controls the manifest that is used to
+   create docker config files via puppet. The puppet tags below are
+   used along with this manifest to generate a config directory for
+   this container.
 
  * kolla_config: Contains YAML that represents how to map config files
    into the kolla container. This config file is typically mapped into
    the container itself at the /var/lib/kolla/config_files/config.json
    location and drives how kolla's external config mechanisms work.
 
- * step_config: A puppet manifest that is used to step through the deployment
-   sequence. Each sequence is given a "step" (via hiera('step') that provides
-   information for when puppet classes should activate themselves.
+ * docker_image: The full name of the docker image that will be used.
 
- * docker_compose:
+ * docker_config: Data that is passed to the docker-cmd hook to configure
+   a container, or step of containers at each step. See the available steps
+   below and the related docker-cmd hook documentation in the heat-agents
+   project.
 
- * container_name:
+ * puppet_tags: Puppet resource tag names that are used to generate config
+   files with puppet. Only the named config resources are used to generate
+   a config file. Any service that specifies tags will have the default
+   tags of 'file,concat,file_line' appended to the setting.
+   Example: keystone_config
 
- * volumes:
+ * config_volume: The name of the volume (directory) where config files
+   will be generated for this service. Use this as the location to
+   bind mount into the running Kolla container for configuration.
 
-Steps correlate to the following:
-
-   1) Service configuration generation with puppet.
-
-   2) Early Openstack Service setup (database init?)
-
-   3) Early containerized networking services startup (OVS)
+ * config_image: The name of the docker image that will be used for
+   generating configuration files. This is often the same value as
+   'docker_image' above but some containers share a common set of
+   config files which are generated in a common base container.
 
-   4) Network configuration
+Docker steps
+------------
+Similar to baremetal docker containers are brought up in a stepwise manner.
+The current architecture supports bringing up baremetal services alongside
+of containers. For each step the baremetal puppet manifests are executed
+first and then any docker containers are brought up afterwards.
 
-   5) General OpenStack Services
-
-   6) Service activation (Pacemaker)
-
-   7) Fencing (Pacemaker)
+Steps correlate to the following:
 
+   Pre) Containers config files generated per hiera settings.
+   1) Load Balancer configuration baremetal
+     a) step 1 baremetal
+     b) step 1 containers
+   2) Core Services (Database/Rabbit/NTP/etc.)
+     a) step 2 baremetal
+     b) step 2 containers
+   3) Early Openstack Service setup (Ringbuilder, etc.)
+     a) step 3 baremetal
+     b) step 3 containers
+   4) General OpenStack Services
+     a) step 4 baremetal
+     b) step 4 containers
+     c) Keystone containers post initialization (tenant,service,endpoint creation)
+   5) Service activation (Pacemaker)
+     a) step 5 baremetal
+     b) step 5 containers
diff --git a/docker/services/neutron-ovs-agent.yaml b/docker/services/neutron-ovs-agent.yaml
index 0a061f6c..ab99da5e 100644
--- a/docker/services/neutron-ovs-agent.yaml
+++ b/docker/services/neutron-ovs-agent.yaml
@@ -10,7 +10,7 @@ parameters:
     type: string
   DockerOpenvswitchImage:
     description: image
-    default: 'centos-binary-neutron-openvswitch-agent'
+    default: 'centos-binary-neutron-openvswitch-agent:latest'
     type: string
   ServiceNetMap:
     default: {}
@@ -32,53 +32,53 @@ resources:
   NeutronOvsAgentBase:
     type: ../../puppet/services/neutron-ovs-agent.yaml
     properties:
+      EndpointMap: {get_param: EndpointMap}
       ServiceNetMap: {get_param: ServiceNetMap}
       DefaultPasswords: {get_param: DefaultPasswords}
-      EndpointMap: {get_param: EndpointMap}
 
 outputs:
   role_data:
     description: Role data for Neutron openvswitch service
     value:
+      service_name: {get_attr: [NeutronOvsAgentBase, role_data, service_name]}
       config_settings: {get_attr: [NeutronOvsAgentBase, role_data, config_settings]}
       step_config: {get_attr: [NeutronOvsAgentBase, role_data, step_config]}
+      docker_image: &neutron_ovs_agent_image
+        list_join:
+        - '/'
+        - [ {get_param: DockerNamespace}, {get_param: DockerOpenvswitchImage} ]
       puppet_tags: neutron_config,neutron_agent_ovs,neutron_plugin_ml2
+      config_volume: neutron
+      config_image: *neutron_ovs_agent_image
       kolla_config:
-        /var/lib/etc-data/json-config/neutron-openvswitch-agent.json:
-           command: /usr/bin/neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-file /etc/neutron/plugins/ml2/ml2_conf.ini
+        /var/lib/kolla/config_files/neutron-openvswitch-agent.json:
+           command: /usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-file /etc/neutron/plugins/ml2/ml2_conf.ini
            config_files:
            - dest: /etc/neutron/neutron.conf
              owner: neutron
              perm: '0600'
-             source: /var/lib/kolla/config_files/neutron.conf
+             source: /var/lib/kolla/config_files/src/etc/neutron/neutron.conf
            - dest: /etc/neutron/plugins/ml2/openvswitch_agent.ini
              owner: neutron
              perm: '0600'
-             source: /var/lib/kolla/config_files/openvswitch_agent.ini
+             source: /var/lib/kolla/config_files/src/etc/neutron/plugins/ml2/openvswitch_agent.ini
            - dest: /etc/neutron/plugins/ml2/ml2_conf.ini
              owner: neutron
              perm: '0600'
-             source: /var/lib/kolla/config_files/ml2_conf.ini
+             source: /var/lib/kolla/config_files/src/etc/neutron/plugins/ml2/ml2_conf.ini
       docker_config:
-        step_1:
+        step_4:
           neutronovsagent:
-            image:
-              list_join:
-              - '/'
-              - [ {get_param: DockerNamespace}, {get_param: DockerOpenvswitchImage} ]
+            image: *neutron_ovs_agent_image
             net: host
             pid: host
             privileged: true
             restart: always
             volumes:
-              - /var/lib/etc-data/json-config/neutron-openvswitch-agent.json:/var/lib/kolla/config_files/config.json
-              - /var/lib/etc-data/neutron/neutron.conf:/var/lib/kolla/config_files/neutron.conf:ro
-              - /var/lib/etc-data/neutron/plugins/ml2/ml2_conf.ini:/var/lib/kolla/config_files/ml2_conf.ini:ro
-              - /var/lib/etc-data/neutron/plugins/ml2/openvswitch_agent.ini:/var/lib/kolla/config_files/openvswitch_agent.ini:ro
+              - /var/lib/kolla/config_files/neutron-openvswitch-agent.json:/var/lib/kolla/config_files/config.json:ro
+              - /var/lib/config-data/neutron:/var/lib/kolla/config_files/src:ro
               - /etc/localtime:/etc/localtime:ro
               - /lib/modules:/lib/modules:ro
               - /run:/run
-              - logs:/var/log/kolla/
             environment:
               - KOLLA_CONFIG_STRATEGY=COPY_ALWAYS
-        step_2: {}
diff --git a/docker/services/nova-compute.yaml b/docker/services/nova-compute.yaml
index e765609e..8eebc397 100644
--- a/docker/services/nova-compute.yaml
+++ b/docker/services/nova-compute.yaml
@@ -10,7 +10,7 @@ parameters:
     type: string
   DockerNovaComputeImage:
     description: image
-    default: 'centos-binary-nova-compute'
+    default: 'centos-binary-nova-compute:latest'
     type: string
   ServiceNetMap:
     default: {}
@@ -29,53 +29,57 @@ parameters:
 
 resources:
 
+
   NovaComputeBase:
     type: ../../puppet/services/nova-compute.yaml
     properties:
       EndpointMap: {get_param: EndpointMap}
+      ServiceNetMap: {get_param: ServiceNetMap}
+      DefaultPasswords: {get_param: DefaultPasswords}
 
 outputs:
   role_data:
     description: Role data for the Nova Compute service.
     value:
+      service_name: {get_attr: [NovaComputeBase, role_data, service_name]}
       config_settings: {get_attr: [NovaComputeBase, role_data, config_settings]}
       step_config: {get_attr: [NovaComputeBase, role_data, step_config]}
       puppet_tags: nova_config,nova_paste_api_ini
+      docker_image: &nova_compute_image
+        list_join:
+        - '/'
+        - [ {get_param: DockerNamespace}, {get_param: DockerNovaComputeImage} ]
+      config_volume: nova_libvirt
+      config_image: *nova_compute_image
       kolla_config:
-        /var/lib/etc-data/json-config/nova-compute.json:
+        /var/lib/kolla/config_files/nova-compute.json:
            command: /usr/bin/nova-compute --config-file /etc/nova/nova.conf --config-file /etc/nova/rootwrap.conf
            config_files:
            - dest: /etc/nova/nova.conf
              owner: nova
              perm: '0600'
-             source: /var/lib/kolla/config_files/nova.conf
+             source: /var/lib/kolla/config_files/src/etc/nova/nova.conf
            - dest: /etc/nova/rootwrap.conf
              owner: nova
              perm: '0600'
-             source: /var/lib/kolla/config_files/rootwrap.conf
+             source: /var/lib/kolla/config_files/src/etc/nova/rootwrap.conf
       docker_config:
-        step_1:
+        step_4:
           novacompute:
-            image:
-              list_join:
-              - '/'
-              - [ {get_param: DockerNamespace}, {get_param: DockerNovaComputeImage} ]
+            image: *nova_compute_image
             net: host
             privileged: true
             user: root
             restart: always
             volumes:
-              - /var/lib/etc-data/json-config/nova-compute.json:/var/lib/kolla/config_files/config.json
-              - /var/lib/etc-data/nova/nova.conf:/var/lib/kolla/config_files/nova.conf:ro
-              - /var/lib/etc-data/nova/rootwrap.conf:/var/lib/kolla/config_files/rootwrap.conf:ro
+              - /var/lib/kolla/config_files/nova-compute.json:/var/lib/kolla/config_files/config.json:ro
+              - /var/lib/config-data/nova_libvirt:/var/lib/kolla/config_files/src:ro
+              - /dev:/dev
+              - /etc/iscsi:/etc/iscsi
               - /etc/localtime:/etc/localtime:ro
               - /lib/modules:/lib/modules:ro
               - /run:/run
-              - /dev:/dev
-              - logs:/var/log/kolla/
-              - /etc/iscsi:/etc/iscsi
+              - /var/lib/nova:/var/lib/nova
               - libvirtd:/var/lib/libvirt
-              - nova_compute:/var/lib/nova/
             environment:
              - KOLLA_CONFIG_STRATEGY=COPY_ALWAYS
-        step_2: {}
diff --git a/docker/services/nova-libvirt.yaml b/docker/services/nova-libvirt.yaml
index 004d624a..d6e7dc76 100644
--- a/docker/services/nova-libvirt.yaml
+++ b/docker/services/nova-libvirt.yaml
@@ -10,7 +10,13 @@ parameters:
     type: string
   DockerLibvirtImage:
     description: image
-    default: 'centos-binary-libvirt'
+    default: 'centos-binary-nova-libvirt:latest'
+    type: string
+  # we configure libvirt via the nova-compute container due to coupling
+  # in the puppet modules
+  DockerNovaComputeImage:
+    description: image
+    default: 'centos-binary-nova-compute:latest'
     type: string
   ServiceNetMap:
     default: {}
@@ -33,50 +39,54 @@ resources:
     type: ../../puppet/services/nova-libvirt.yaml
     properties:
       EndpointMap: {get_param: EndpointMap}
+      ServiceNetMap: {get_param: ServiceNetMap}
+      DefaultPasswords: {get_param: DefaultPasswords}
 
 outputs:
   role_data:
     description: Role data for the Libvirt service.
     value:
+      service_name: {get_attr: [NovaLibvirtBase, role_data, service_name]}
       config_settings: {get_attr: [NovaLibvirtBase, role_data, config_settings]}
       step_config: {get_attr: [NovaLibvirtBase, role_data, step_config]}
+      docker_image: &libvirt_image
+        list_join:
+        - '/'
+        - [ {get_param: DockerNamespace}, {get_param: DockerLibvirtImage} ]
       puppet_tags: nova_config
+      config_volume: nova_libvirt
+      config_image:
+        list_join:
+        - '/'
+        - [ {get_param: DockerNamespace}, {get_param: DockerNovaComputeImage} ]
       kolla_config:
-        /var/lib/etc-data/json-config/nova-libvirt.json:
+        /var/lib/kolla/config_files/nova-libvirt.json:
            command: /usr/sbin/libvirtd --config /etc/libvirt/libvirtd.conf
            config_files:
            - dest: /etc/libvirt/libvirtd.conf
              owner: root
              perm: '0644'
-             source: /var/lib/kolla/config_files/libvirtd.conf
+             source: /var/lib/kolla/config_files/src/etc/libvirt/libvirtd.conf
       docker_config:
-        step_1:
+        step_3:
           nova_libvirt:
-            image:
-              list_join:
-                - '/'
-                - [ {get_param: DockerNamespace}, {get_param: DockerLibvirtImage} ]
+            image: *libvirt_image
             net: host
             pid: host
             privileged: true
             restart: always
             volumes:
-              - /var/lib/etc-data/json-config/nova-libvirt.json:/var/lib/kolla/config_files/config.json
-              - /var/lib/etc-data/libvirt/libvirtd.conf:/var/lib/kolla/config_files/libvirtd.conf
-              # NOTE(mandre) Ideally the qemu.conf file is mounted in
-              # /var/lib/kolla/config_files and copied to the right place but
-              # copy-json.py doesn't allow us to do that without appending the
-              # file as an additional config on the CLI
-              - /var/lib/etc-data/libvirt/qemu.conf:/etc/libvirt/qemu.conf:ro
+              - /var/lib/kolla/config_files/nova-libvirt.json:/var/lib/kolla/config_files/config.json:ro
+              - /var/lib/config-data/nova_libvirt:/var/lib/kolla/config_files/src:ro
+              - /dev:/dev
               - /etc/localtime:/etc/localtime:ro
               - /lib/modules:/lib/modules:ro
               - /run:/run
-              - /dev:/dev
               - /sys/fs/cgroup:/sys/fs/cgroup
-              - logs:/var/log/kolla/
+              - /var/lib/nova:/var/lib/nova
+              # Needed to use host's virtlogd
+              - /var/run/libvirt:/var/run/libvirt
               - libvirtd:/var/lib/libvirt
-              - nova_compute:/var/lib/nova/
               - nova_libvirt_qemu:/etc/libvirt/qemu
             environment:
               - KOLLA_CONFIG_STRATEGY=COPY_ALWAYS
-        step_2: {}
diff --git a/docker/services/services.yaml b/docker/services/services.yaml
index 8c31107f..8e899024 100644
--- a/docker/services/services.yaml
+++ b/docker/services/services.yaml
@@ -66,10 +66,12 @@ outputs:
       global_config_settings:
         {get_attr: [PuppetServices, role_data, global_config_settings]}
       step_config:
-        {get_attr: [PuppetServices, role_data, step_config]}
-      puppet_tags: {list_join: [",", {get_attr: [ServiceChain, role_data, puppet_tags]}]}
+        {get_attr: [ServiceChain, role_data, step_config]}
+      docker_image: {get_attr: [ServiceChain, role_data, docker_image]}
+      puppet_tags: {get_attr: [ServiceChain, role_data, puppet_tags]}
+      config_volume: {get_attr: [ServiceChain, role_data, config_volume]}
+      config_image: {get_attr: [ServiceChain, role_data, config_image]}
       kolla_config:
         map_merge: {get_attr: [ServiceChain, role_data, kolla_config]}
       docker_config:
-        step_1: {map_merge: {get_attr: [ServiceChain, role_data, docker_config, step_1]}}
-        step_2: {map_merge: {get_attr: [ServiceChain, role_data, docker_config, step_2]}}
+        {get_attr: [ServiceChain, role_data, docker_config]}