#######
Logging
#######

************
Installation
************

Currently, we use the `sample configuration`_ in Istio to install fluentd::

    cd clover/logging

First, install logging stack Elasticsearch, Fluentd and Kibana::

    kubectl apply -f install/logging-stack.yaml

Note that, it must be done in separated steps. If you run ``kubectl apply -f
install`` instead, the mixer adapter may fail to intialize because the target
service can not be found. You may find an error message from mixer container::

    2018-05-09T02:43:14.435156Z	error	Unable to initialize adapter:
    snapshot='6', handler='handler.fluentd.istio-system', adapter='fluentd',
    err='adapter instantiation error: dial tcp: lookup fluentd-es.logging on
    10.96.0.10:53: no such host'.

Then configure fluentd for istio::

    kubectl apply -f install/fluentd-istio.yaml

Configure fluentd for node level logging::

    kubectl apply -f install/fluentd-daemonset-elasticsearch-rbac.yaml

.. _sample configuration: https://istio.io/docs/tasks/telemetry/fluentd.html

********
Validate
********

The scripts in ``clover/logging`` validates fluentd installation::

    python clover/logging/validate.py

It validates the installation with the following criterias

#. existence of fluented pod
#. fluentd input is configured correctly
#. TBD

**************************
Understanding how it works
**************************

In clover stack, Istio is configured to automatically gather logs for services
in a mesh. More specificly, it is configured in `Mixer`_::

- when to log
- what to log
- where to log

.. _Mixer: https://istio.io/docs/concepts/policy-and-control/mixer.html

When to log
===========

Istio defines when to log by creating a custom resource ``rule``. For example:

.. code-block:: yaml

    apiVersion: "config.istio.io/v1alpha2"
    kind: rule
    metadata:
      name: newlogtofluentd
      namespace: istio-system
    spec:
      match: "true" # match for all requests
      actions:
      - handler: handler.fluentd
        instances:
        - newlog.logentry

This rule specifies that all instances of ``newlog.logentry`` that matches the
expression will be handled by the specified handler ``handler.fluentd``. We
shall explain ``instances`` and ``handler`` later. The expression ``true`` means
whenever a request arrive at Mixer, it will trigger the actions defined belows.

``rule`` is a custom resource definition from `Istio installation`_.

.. code-block:: yaml

    # Rule to send logentry instances to the fluentd handler
    kind: CustomResourceDefinition
    apiVersion: apiextensions.k8s.io/v1beta1
    metadata:
      name: rules.config.istio.io
      labels:
        package: istio.io.mixer
        istio: core
    spec:
      group: config.istio.io
      names:
        kind: rule
        plural: rules
        singular: rule
    scope: Namespaced
    version: v1alpha2

.. _Istio installation: https://github.com/istio/istio/blob/master/install/kubernetes/templates/istio-mixer.yaml.tmpl

What to log
===========

The instance defines what content to be logged.

> A (request) instance is the result of applying request attributes to the
> template mapping. The mapping is specified as an instance configuration.

For example:

.. code-block:: yaml

    # Configuration for logentry instances
    apiVersion: "config.istio.io/v1alpha2"
    kind: logentry
    metadata:
      name: newlog
      namespace: istio-system
    spec:
      severity: '"info"'
      timestamp: request.time
      variables:
        source: source.labels["app"] | source.service | "unknown"
        user: source.user | "unknown"
        destination: destination.labels["app"] | destination.service | "unknown"
        responseCode: response.code | 0
        responseSize: response.size | 0
        latency: response.duration | "0ms"
      monitored_resource_type: '"UNSPECIFIED"'

The keys under ``spec`` should conform to the template. To learn what fields
are available and valid type, you may need to reference the corresponding
template, in this case, `Log Entry template`_.

The values of each field could be either `Istio attributes`_ or an expression.

> A given Istio deployment has a fixed vocabulary of attributes that it
> understands. The specific vocabulary is determined by the set of attribute
> producers being used in the deployment. The primary attribute producer in
> Istio is Envoy, although Mixer and services can also introduce attributes.

Refer to the `Attribute Vocabulary`_ to learn the full set.

By the way, ``logentry`` is also a custom resource definition created by Istio.

.. _Istio attributes: https://istio.io/docs/concepts/policy-and-control/attributes.html
.. _Attribute Vocabulary: https://istio.io/docs/reference/config/mixer/attribute-vocabulary.html
.. _Log Entry template: https://istio.io/docs/reference/config/template/logentry.html

Where to log
============

For log, the handler defines where these information will be handled, in this
example, a fluentd daemon on fluentd-es.logging:24224.

.. code-block:: yaml

    # Configuration for a fluentd handler
    apiVersion: "config.istio.io/v1alpha2"
    kind: fluentd
    metadata:
      name: handler
      namespace: istio-system
    spec:
      address: "fluentd-es.logging:24224"

In this example, handlers (``handler.fluentd``) configure `Adapters`_
(``fluentd``) to handle the data delivered from the created instances
(``newlog.logentry``).

An adapter only accepts instance of specified kind. For example,
`fluentd adapter`_ accepts logentry but not other kinds.

.. _Adapters: https://istio.io/docs/concepts/policy-and-control/mixer.html#adapters
.. _fluentd adapter: https://istio.io/docs/reference/config/adapters/fluentd.html