aboutsummaryrefslogtreecommitdiffstats
path: root/deployed-server/README.rst
blob: ce74e77b5a39ebc071a0cc4b0d856f21f848aa68 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
TripleO with Deployed Servers
=============================

The deployed-server set of templates can be used to deploy TripleO via
tripleo-heat-templates to servers that are already installed with a base
operating system.

When OS::TripleO::Server is mapped to the deployed-server.yaml template via the
provided deployed-server-environment.yaml resource registry, Nova and Ironic
are not used to create any server instances. Heat continues to create the
SoftwareDeployment resources, and they are made available to the already
deployed and running servers.

Template Usage
--------------
To use these templates pass the included environment file to the deployment
command::

    -e deployed-server/deployed-server-environment.yaml

Deployed Server configuration
-----------------------------
It is currently assumed that the deployed servers being used have the required
set of software and packages already installed on them. These exact
requirements must match how such a server would look if it were deployed the
standard way via Ironic using the TripleO overcloud-full image.

An easy way to help get this setup for development is to use an overcloud-full
image from an already existing TripleO setup. Create the vm's for the already
deployed server, and use the overcloud-full image as their disk.

Each server must have a fqdn set that resolves to an IP address on a routable
network (e.g., the hostname should not resolve to 127.0.0.1).  The hostname
will be detected on each server via the hostnamectl --static command.

Each server also must have a route to the configured IP address on the
undercloud where the OpenStack services are listening. This is the value for
local_ip in the undercloud.conf.

It's recommended that each server have at least 2 nic's. One used for external
management such as ssh, and one used for the OpenStack deployment itself. Since
the overcloud deployment will reconfigure networking on the configured nic to
be used by OpenStack, the external management nic is needed as a fallback so
that all connectivity is not lost in case of a configuration error. Be sure to
use correct nic config templates as needed, since the nodes will not receive
dhcp from the undercloud neutron-dhcp-agent service.

For example, the net-config-static-bridge.yaml template could be used for
controllers, and the net-config-static.yaml template could be used for computes
by specifying:

resource_registry:
  OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/deployed-server/tripleo-heat-templates/net-config-static-bridge.yaml
  OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/deployed-server/tripleo-heat-templates/net-config-static.yaml

In a setup where the first nic on the servers is used for external management,
set the nic's to be used for OpenStack to nic2:

parameter_defaults:
  NeutronPublicInterface: nic2
  HypervisorNeutronPublicInterface: nic2

The above nic config templates also require a route to the ctlplane network to
be defined. Define the needed parameters as necessary for your environment, for
example:

parameter_defaults:
  ControlPlaneDefaultRoute: 192.168.122.130
  ControlPlaneSubnetCidr: "24"
  EC2MetadataIp: "192.0.2.1"

In this example, 192.168.122.130 is the external management IP of an
undercloud, thus it is the default route for the configured local_ip value of
192.0.2.1.


os-collect-config
-----------------
os-collect-config on each deployed server must be manually configured to poll
the Heat API for the available SoftwareDeployments. An example configuration
for /etc/os-collect-config.conf looks like:

    [DEFAULT]
    collectors=heat
    command=os-refresh-config

    [heat]
    # you can get these values from stackrc on the undercloud
    user_id=<a user that can connect to heat> # note this must be the ID, not the username
    password=<a password>
    auth_url=<keystone url>
    project_id=<project_id> # note, this must be the ID, not project name
    stack_id=<stack_id>
    resource_name=<resource_name>

Note that the stack_id value is the id of the nested stack containing the
resource (identified by resource_name) implemented by the deployed-server.yaml
templates.

Once the configuration for os-collect-config has been defined, the service
needs to be restarted. Once restarted, it will start polling Heat and applying
the SoftwareDeployments.

A sample script at deployed-server/scripts/get-occ-config.sh is included that
will automatically generate the os-collect-config configuration needed on each
server, ssh to each server, copy the configuration, and restart the
os-collect-config service.

.. warning::
   The get-occ-config.sh script is not intended for production use, as it
   copies admin credentials to each of the deployed nodes.

The script can only be used once the stack id's of the nested deployed-server
stacks have been created via Heat. This usually only takes a couple of minutes
once the deployment command has been started. Once the following output is seen
from the deployment command, the script should be ready to run:

    [Controller]: CREATE_IN_PROGRESS state changed
    [NovaCompute]: CREATE_IN_PROGRESS state changed

The user running the script must be able to ssh as root to each server.  Define
the hostnames of the deployed servers you intend to use for each role type::

    export controller_hosts="controller0 controller1 controller2"
    export compute_hosts="compute0"

Then run the script on the undercloud with a stackrc file sourced, and
the script will copy the needed os-collect-config.conf configuration to each
server and restart the os-collect-config service.
.ni { color: #f8f8f2 } /* Name.Entity */ .highlight .ne { color: #a6e22e } /* Name.Exception */ .highlight .nf { color: #a6e22e } /* Name.Function */ .highlight .nl { color: #f8f8f2 } /* Name.Label */ .highlight .nn { color: #f8f8f2 } /* Name.Namespace */ .highlight .nx { color: #a6e22e } /* Name.Other */ .highlight .py { color: #f8f8f2 } /* Name.Property */ .highlight .nt { color: #f92672 } /* Name.Tag */ .highlight .nv { color: #f8f8f2 } /* Name.Variable */ .highlight .ow { color: #f92672 } /* Operator.Word */ .highlight .w { color: #f8f8f2 } /* Text.Whitespace */ .highlight .mb { color: #ae81ff } /* Literal.Number.Bin */ .highlight .mf { color: #ae81ff } /* Literal.Number.Float */ .highlight .mh { color: #ae81ff } /* Literal.Number.Hex */ .highlight .mi { color: #ae81ff } /* Literal.Number.Integer */ .highlight .mo { color: #ae81ff } /* Literal.Number.Oct */ .highlight .sa { color: #e6db74 } /* Literal.String.Affix */ .highlight .sb { color: #e6db74 } /* Literal.String.Backtick */ .highlight .sc { color: #e6db74 } /* Literal.String.Char */ .highlight .dl { color: #e6db74 } /* Literal.String.Delimiter */ .highlight .sd { color: #e6db74 } /* Literal.String.Doc */ .highlight .s2 { color: #e6db74 } /* Literal.String.Double */ .highlight .se { color: #ae81ff } /* Literal.String.Escape */ .highlight .sh { color: #e6db74 } /* Literal.String.Heredoc */ .highlight .si { color: #e6db74 } /* Literal.String.Interpol */ .highlight .sx { color: #e6db74 } /* Literal.String.Other */ .highlight .sr { color: #e6db74 } /* Literal.String.Regex */ .highlight .s1 { color: #e6db74 } /* Literal.String.Single */ .highlight .ss { color: #e6db74 } /* Literal.String.Symbol */ .highlight .bp { color: #f8f8f2 } /* Name.Builtin.Pseudo */ .highlight .fm { color: #a6e22e } /* Name.Function.Magic */ .highlight .vc { color: #f8f8f2 } /* Name.Variable.Class */ .highlight .vg { color: #f8f8f2 } /* Name.Variable.Global */ .highlight .vi { color: #f8f8f2 } /* Name.Variable.Instance */ .highlight .vm { color: #f8f8f2 } /* Name.Variable.Magic */ .highlight .il { color: #ae81ff } /* Literal.Number.Integer.Long */ } @media (prefers-color-scheme: light) { .highlight .hll { background-color: #ffffcc } .highlight .c { color: #888888 } /* Comment */ .highlight .err { color: #a61717; background-color: #e3d2d2 } /* Error */ .highlight .k { color: #008800; font-weight: bold } /* Keyword */ .highlight .ch { color: #888888 } /* Comment.Hashbang */ .highlight .cm { color: #888888 } /* Comment.Multiline */ .highlight .cp { color: #cc0000; font-weight: bold } /* Comment.Preproc */ .highlight .cpf { color: #888888 } /* Comment.PreprocFile */ .highlight .c1 { color: #888888 } /* Comment.Single */ .highlight .cs { color: #cc0000; font-weight: bold; background-color: #fff0f0 } /* Comment.Special */ .highlight .gd { color: #000000; background-color: #ffdddd } /* Generic.Deleted */ .highlight .ge { font-style: italic } /* Generic.Emph */ .highlight .gr { color: #aa0000 } /* Generic.Error */ .highlight .gh { color: #333333 } /* Generic.Heading */ .highlight .gi { color: #000000; background-color: #ddffdd } /* Generic.Inserted */ .highlight .go { color: #888888 } /* Generic.Output */ .highlight .gp { color: #555555 } /* Generic.Prompt */ .highlight .gs { font-weight: bold } /* Generic.Strong */ .highlight .gu { color: #666666 } /* Generic.Subheading */ .highlight .gt { color: #aa0000 } /* Generic.Traceback */ .highlight .kc { color: #008800; font-weight: bold } /* Keyword.Constant */ .highlight .kd { color: #008800; font-weight: bold } /* Keyword.Declaration */ .highlight .kn { color: #008800; font-weight: bold } /* Keyword.Namespace */ .highlight .kp { color: #008800 } /* Keyword.Pseudo */ .highlight .kr { color: #008800; font-weight: bold } /* Keyword.Reserved */ .highlight .kt { color: #888888; font-weight: bold } /* Keyword.Type */ .highlight .m { color: #0000DD; font-weight: bold } /* Literal.Number */ .highlight .s { color: #dd2200; background-color: #fff0f0 } /* Literal.String */ .highlight .na { color: #336699 } /* Name.Attribute */ .highlight .nb { color: #003388 } /* Name.Builtin */ .highlight .nc { color: #bb0066; font-weight: bold } /* Name.Class */ .highlight .no { color: #003366; font-weight: bold } /* Name.Constant */ .highlight .nd { color: #555555 } /* Name.Decorator */ .highlight .ne { color: #bb0066; font-weight: bold } /* Name.Exception */ .highlight .nf { color: #0066bb; font-weight: bold } /* Name.Function */ .highlight .nl { color: #336699; font-style: italic } /* Name.Label */ .highlight .nn { color: #bb0066; font-weight: bold } /* Name.Namespace */ .highlight .py { color: #336699; font-weight: bold } /* Name.Property */ .highlight .nt { color: #bb0066; font-weight: bold } /* Name.Tag */ .highlight .nv { color: #336699 } /* Name.Variable */ .highlight .ow { color: #008800 } /* Operator.Word */ .highlight .w { color: #bbbbbb } /* Text.Whitespace */ .highlight .mb { color: #0000DD; font-weight: bold } /* Literal.Number.Bin */ .highlight .mf { color: #0000DD; font-weight: bold } /* Literal.Number.Float */ .highlight .mh { color: #0000DD; font-weight: bold } /* Literal.Number.Hex */ .highlight .mi { color: #0000DD; font-weight: bold } /* Literal.Number.Integer */ .highlight .mo { color: #0000DD; font-weight: bold } /* Literal.Number.Oct */ .highlight .sa { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Affix */ .highlight .sb { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Backtick */ .highlight .sc { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Char */ .highlight .dl { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Delimiter */ .highlight .sd { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Doc */ .highlight .s2 { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Double */ .highlight .se { color: #0044dd; background-color: #fff0f0 } /* Literal.String.Escape */ .highlight .sh { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Heredoc */ .highlight .si { color: #3333bb; background-color: #fff0f0 } /* Literal.String.Interpol */ .highlight .sx { color: #22bb22; background-color: #f0fff0 } /* Literal.String.Other */ .highlight .sr { color: #008800; background-color: #fff0ff } /* Literal.String.Regex */ .highlight .s1 { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Single */ .highlight .ss { color: #aa6600; background-color: #fff0f0 } /* Literal.String.Symbol */ .highlight .bp { color: #003388 } /* Name.Builtin.Pseudo */ .highlight .fm { color: #0066bb; font-weight: bold } /* Name.Function.Magic */ .highlight .vc { color: #336699 } /* Name.Variable.Class */ .highlight .vg { color: #dd7700 } /* Name.Variable.Global */ .highlight .vi { color: #3333bb } /* Name.Variable.Instance */ .highlight .vm { color: #336699 } /* Name.Variable.Magic */ .highlight .il { color: #0000DD; font-weight: bold } /* Literal.Number.Integer.Long */ }
/*
 * Copyright 2014-2015 Open Networking Laboratory
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.onosproject.net.intent.impl;

import com.google.common.collect.ImmutableList;
import org.apache.felix.scr.annotations.Activate;
import org.apache.felix.scr.annotations.Component;
import org.apache.felix.scr.annotations.Deactivate;
import org.apache.felix.scr.annotations.Reference;
import org.apache.felix.scr.annotations.ReferenceCardinality;
import org.apache.felix.scr.annotations.Service;
import org.onosproject.event.AbstractListenerManager;
import org.onosproject.core.CoreService;
import org.onosproject.core.IdGenerator;
import org.onosproject.net.flow.FlowRule;
import org.onosproject.net.flow.FlowRuleOperations;
import org.onosproject.net.flow.FlowRuleOperationsContext;
import org.onosproject.net.flow.FlowRuleService;
import org.onosproject.net.intent.FlowRuleIntent;
import org.onosproject.net.intent.Intent;
import org.onosproject.net.intent.IntentBatchDelegate;
import org.onosproject.net.intent.IntentCompiler;
import org.onosproject.net.intent.IntentData;
import org.onosproject.net.intent.IntentEvent;
import org.onosproject.net.intent.IntentExtensionService;
import org.onosproject.net.intent.IntentListener;
import org.onosproject.net.intent.IntentService;
import org.onosproject.net.intent.IntentState;
import org.onosproject.net.intent.IntentStore;
import org.onosproject.net.intent.IntentStoreDelegate;
import org.onosproject.net.intent.Key;
import org.onosproject.net.intent.impl.phase.FinalIntentProcessPhase;
import org.onosproject.net.intent.impl.phase.IntentProcessPhase;
import org.onosproject.net.intent.impl.phase.IntentWorker;
import org.slf4j.Logger;

import java.util.Collection;
import java.util.EnumSet;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
import java.util.stream.Collectors;

import static com.google.common.base.Preconditions.checkNotNull;
import static java.util.concurrent.Executors.newFixedThreadPool;
import static java.util.concurrent.Executors.newSingleThreadExecutor;
import static org.onlab.util.Tools.groupedThreads;
import static org.onosproject.net.intent.IntentState.*;
import static org.onosproject.net.intent.constraint.PartialFailureConstraint.intentAllowsPartialFailure;
import static org.onosproject.net.intent.impl.phase.IntentProcessPhase.newInitialPhase;
import static org.onosproject.security.AppGuard.checkPermission;
import static org.slf4j.LoggerFactory.getLogger;
import static org.onosproject.security.AppPermission.Type.*;


/**
 * An implementation of intent service.
 */
@Component(immediate = true)
@Service
public class IntentManager
        extends AbstractListenerManager<IntentEvent, IntentListener>
        implements IntentService, IntentExtensionService {

    private static final Logger log = getLogger(IntentManager.class);

    public static final String INTENT_NULL = "Intent cannot be null";
    public static final String INTENT_ID_NULL = "Intent key cannot be null";

    private static final int NUM_THREADS = 12;

    private static final EnumSet<IntentState> RECOMPILE
            = EnumSet.of(INSTALL_REQ, FAILED, WITHDRAW_REQ);
    private static final EnumSet<IntentState> WITHDRAW
            = EnumSet.of(WITHDRAW_REQ, WITHDRAWING, WITHDRAWN);

    @Reference(cardinality = ReferenceCardinality.MANDATORY_UNARY)
    protected CoreService coreService;

    @Reference(cardinality = ReferenceCardinality.MANDATORY_UNARY)
    protected IntentStore store;

    @Reference(cardinality = ReferenceCardinality.MANDATORY_UNARY)
    protected ObjectiveTrackerService trackerService;

    @Reference(cardinality = ReferenceCardinality.MANDATORY_UNARY)
    protected FlowRuleService flowRuleService;

    private ExecutorService batchExecutor;
    private ExecutorService workerExecutor;

    private final CompilerRegistry compilerRegistry = new CompilerRegistry();
    private final InternalIntentProcessor processor = new InternalIntentProcessor();
    private final IntentStoreDelegate delegate = new InternalStoreDelegate();
    private final TopologyChangeDelegate topoDelegate = new InternalTopoChangeDelegate();
    private final IntentBatchDelegate batchDelegate = new InternalBatchDelegate();
    private IdGenerator idGenerator;

    private final IntentAccumulator accumulator = new IntentAccumulator(batchDelegate);

    @Activate
    public void activate() {
        store.setDelegate(delegate);
        trackerService.setDelegate(topoDelegate);
        eventDispatcher.addSink(IntentEvent.class, listenerRegistry);
        batchExecutor = newSingleThreadExecutor(groupedThreads("onos/intent", "batch"));
        workerExecutor = newFixedThreadPool(NUM_THREADS, groupedThreads("onos/intent", "worker-%d"));
        idGenerator = coreService.getIdGenerator("intent-ids");
        Intent.bindIdGenerator(idGenerator);
        log.info("Started");
    }

    @Deactivate
    public void deactivate() {
        store.unsetDelegate(delegate);
        trackerService.unsetDelegate(topoDelegate);
        eventDispatcher.removeSink(IntentEvent.class);
        batchExecutor.shutdown();
        workerExecutor.shutdown();
        Intent.unbindIdGenerator(idGenerator);
        log.info("Stopped");
    }

    @Override
    public void submit(Intent intent) {
        checkPermission(INTENT_WRITE);
        checkNotNull(intent, INTENT_NULL);
        IntentData data = new IntentData(intent, IntentState.INSTALL_REQ, null);
        store.addPending(data);
    }

    @Override
    public void withdraw(Intent intent) {
        checkPermission(INTENT_WRITE);
        checkNotNull(intent, INTENT_NULL);
        IntentData data = new IntentData(intent, IntentState.WITHDRAW_REQ, null);
        store.addPending(data);
    }

    @Override
    public void purge(Intent intent) {
        checkPermission(INTENT_WRITE);
        checkNotNull(intent, INTENT_NULL);
        IntentData data = new IntentData(intent, IntentState.PURGE_REQ, null);
        store.addPending(data);
    }

    @Override
    public Intent getIntent(Key key) {
        checkPermission(INTENT_READ);
        return store.getIntent(key);
    }

    @Override
    public Iterable<Intent> getIntents() {
        checkPermission(INTENT_READ);
        return store.getIntents();
    }

    @Override
    public Iterable<IntentData> getIntentData() {
        checkPermission(INTENT_READ);
        return store.getIntentData(false, 0);
    }

    @Override
    public long getIntentCount() {
        checkPermission(INTENT_READ);
        return store.getIntentCount();
    }

    @Override
    public IntentState getIntentState(Key intentKey) {
        checkPermission(INTENT_READ);
        checkNotNull(intentKey, INTENT_ID_NULL);
        return store.getIntentState(intentKey);
    }

    @Override
    public List<Intent> getInstallableIntents(Key intentKey) {
        checkPermission(INTENT_READ);
        checkNotNull(intentKey, INTENT_ID_NULL);
        return store.getInstallableIntents(intentKey);
    }

    @Override
    public boolean isLocal(Key intentKey) {
        checkPermission(INTENT_READ);
        return store.isMaster(intentKey);
    }

    @Override
    public <T extends Intent> void registerCompiler(Class<T> cls, IntentCompiler<T> compiler) {
        compilerRegistry.registerCompiler(cls, compiler);
    }

    @Override
    public <T extends Intent> void unregisterCompiler(Class<T> cls) {
        compilerRegistry.unregisterCompiler(cls);
    }

    @Override
    public Map<Class<? extends Intent>, IntentCompiler<? extends Intent>> getCompilers() {
        return compilerRegistry.getCompilers();
    }

    @Override
    public Iterable<Intent> getPending() {
        checkPermission(INTENT_READ);

        return store.getPending();
    }

    // Store delegate to re-post events emitted from the store.
    private class InternalStoreDelegate implements IntentStoreDelegate {
        @Override
        public void notify(IntentEvent event) {
            post(event);
        }

        @Override
        public void process(IntentData data) {
            accumulator.add(data);
        }

        @Override
        public void onUpdate(IntentData intentData) {
            trackerService.trackIntent(intentData);
        }
    }

    private void buildAndSubmitBatches(Iterable<Key> intentKeys,
                                       boolean compileAllFailed) {
        // Attempt recompilation of the specified intents first.
        for (Key key : intentKeys) {
            Intent intent = store.getIntent(key);
            if (intent == null) {
                continue;
            }
            submit(intent);
        }

        // If required, compile all currently failed intents.
        for (Intent intent : getIntents()) {
            IntentState state = getIntentState(intent.key());
            if ((compileAllFailed && RECOMPILE.contains(state))
                    || intentAllowsPartialFailure(intent)) {
                if (WITHDRAW.contains(state)) {
                    withdraw(intent);
                } else {
                    submit(intent);
                }
            }
        }

        //FIXME
//        for (ApplicationId appId : batches.keySet()) {
//            if (batchService.isLocalLeader(appId)) {
//                execute(batches.get(appId).build());
//            }
//        }
    }

    // Topology change delegate
    private class InternalTopoChangeDelegate implements TopologyChangeDelegate {
        @Override
        public void triggerCompile(Iterable<Key> intentKeys,
                                   boolean compileAllFailed) {
            buildAndSubmitBatches(intentKeys, compileAllFailed);
        }
    }

    private Future<FinalIntentProcessPhase> submitIntentData(IntentData data) {
        IntentData current = store.getIntentData(data.key());
        IntentProcessPhase initial = newInitialPhase(processor, data, current);
        return workerExecutor.submit(new IntentWorker(initial));
    }

    private class IntentBatchProcess implements Runnable {

        protected final Collection<IntentData> data;

        IntentBatchProcess(Collection<IntentData> data) {
            this.data = checkNotNull(data);
        }

        @Override
        public void run() {
            try {
                /*
                 1. wrap each intentdata in a runnable and submit
                 2. wait for completion of all the work
                 3. accumulate results and submit batch write of IntentData to store
                    (we can also try to update these individually)
                 */
                submitUpdates(waitForFutures(createIntentUpdates()));
            } catch (Exception e) {
                log.error("Error submitting batches:", e);
                // FIXME incomplete Intents should be cleaned up
                //       (transition to FAILED, etc.)

                // the batch has failed
                // TODO: maybe we should do more?
                log.error("Walk the plank, matey...");
                //FIXME
//            batchService.removeIntentOperations(data);
            }
            accumulator.ready();
        }

        private List<Future<FinalIntentProcessPhase>> createIntentUpdates() {
            return data.stream()
                    .map(IntentManager.this::submitIntentData)
                    .collect(Collectors.toList());
        }

        private List<FinalIntentProcessPhase> waitForFutures(List<Future<FinalIntentProcessPhase>> futures) {
            ImmutableList.Builder<FinalIntentProcessPhase> updateBuilder = ImmutableList.builder();
            for (Future<FinalIntentProcessPhase> future : futures) {
                try {
                    updateBuilder.add(future.get());
                } catch (InterruptedException | ExecutionException e) {
                    //FIXME
                    log.warn("Future failed: {}", e);
                }
            }
            return updateBuilder.build();
        }

        private void submitUpdates(List<FinalIntentProcessPhase> updates) {
            store.batchWrite(updates.stream()
                                     .map(FinalIntentProcessPhase::data)
                                     .collect(Collectors.toList()));
        }
    }

    private class InternalBatchDelegate implements IntentBatchDelegate {
        @Override
        public void execute(Collection<IntentData> operations) {
            log.debug("Execute {} operation(s).", operations.size());
            log.trace("Execute operations: {}", operations);

            // batchExecutor is single-threaded, so only one batch is in flight at a time
            batchExecutor.execute(new IntentBatchProcess(operations));
        }
    }

    private class InternalIntentProcessor implements IntentProcessor {
        @Override
        public List<Intent> compile(Intent intent, List<Intent> previousInstallables) {
            return compilerRegistry.compile(intent, previousInstallables);
        }

        @Override
        public void apply(Optional<IntentData> toUninstall, Optional<IntentData> toInstall) {
            IntentManager.this.apply(toUninstall, toInstall);
        }
    }

    private enum Direction {
        ADD,
        REMOVE
    }

    private void applyIntentData(Optional<IntentData> intentData,
                                 FlowRuleOperations.Builder builder,
                                 Direction direction) {
        if (!intentData.isPresent()) {
            return;
        }
        IntentData data = intentData.get();

        List<Intent> intentsToApply = data.installables();
        if (!intentsToApply.stream().allMatch(x -> x instanceof FlowRuleIntent)) {
            throw new IllegalStateException("installable intents must be FlowRuleIntent");
        }

        if (direction == Direction.ADD) {
            trackerService.addTrackedResources(data.key(), data.intent().resources());
            intentsToApply.forEach(installable ->
                    trackerService.addTrackedResources(data.key(), installable.resources()));
        } else {
            trackerService.removeTrackedResources(data.key(), data.intent().resources());
            intentsToApply.forEach(installable ->
                    trackerService.removeTrackedResources(data.intent().key(),
                            installable.resources()));
        }

        // FIXME do FlowRuleIntents have stages??? Can we do uninstall work in parallel? I think so.
        builder.newStage();

        List<Collection<FlowRule>> stages = intentsToApply.stream()
                .map(x -> (FlowRuleIntent) x)
                .map(FlowRuleIntent::flowRules)
                .collect(Collectors.toList());

        for (Collection<FlowRule> rules : stages) {
            if (direction == Direction.ADD) {
                rules.forEach(builder::add);
            } else {
                rules.forEach(builder::remove);
            }
        }

    }

    private void apply(Optional<IntentData> toUninstall, Optional<IntentData> toInstall) {
        // need to consider if FlowRuleIntent is only one as installable intent or not

        FlowRuleOperations.Builder builder = FlowRuleOperations.builder();
        applyIntentData(toUninstall, builder, Direction.REMOVE);
        applyIntentData(toInstall, builder, Direction.ADD);

        FlowRuleOperations operations = builder.build(new FlowRuleOperationsContext() {
            @Override
            public void onSuccess(FlowRuleOperations ops) {
                if (toInstall.isPresent()) {
                    IntentData installData = toInstall.get();
                    log.debug("Completed installing: {}", installData.key());
                    installData.setState(INSTALLED);
                    store.write(installData);
                } else if (toUninstall.isPresent()) {
                    IntentData uninstallData = toUninstall.get();
                    log.debug("Completed withdrawing: {}", uninstallData.key());
                    switch (uninstallData.request()) {
                        case INSTALL_REQ:
                            uninstallData.setState(FAILED);
                            break;
                        case WITHDRAW_REQ:
                        default: //TODO "default" case should not happen
                            uninstallData.setState(WITHDRAWN);
                            break;
                    }
                    store.write(uninstallData);
                }
            }

            @Override
            public void onError(FlowRuleOperations ops) {
                // if toInstall was cause of error, then recompile (manage/increment counter, when exceeded -> CORRUPT)
                if (toInstall.isPresent()) {
                    IntentData installData = toInstall.get();
                    log.warn("Failed installation: {} {} on {}",
                             installData.key(), installData.intent(), ops);
                    installData.setState(CORRUPT);
                    installData.incrementErrorCount();
                    store.write(installData);
                }
                // if toUninstall was cause of error, then CORRUPT (another job will clean this up)
                if (toUninstall.isPresent()) {
                    IntentData uninstallData = toUninstall.get();
                    log.warn("Failed withdrawal: {} {} on {}",
                             uninstallData.key(), uninstallData.intent(), ops);
                    uninstallData.setState(CORRUPT);
                    uninstallData.incrementErrorCount();
                    store.write(uninstallData);
                }
            }
        });

        if (log.isTraceEnabled()) {
            log.trace("applying intent {} -> {} with {} rules: {}",
                      toUninstall.isPresent() ? toUninstall.get().key() : "<empty>",
                      toInstall.isPresent() ? toInstall.get().key() : "<empty>",
                      operations.stages().stream().mapToLong(i -> i.size()).sum(),
                      operations.stages());
        }

        flowRuleService.apply(operations);
    }

}