diff options
Diffstat (limited to 'docs/development/design/logging.rst')
| -rw-r--r-- | docs/development/design/logging.rst | 184 |
1 files changed, 184 insertions, 0 deletions
diff --git a/docs/development/design/logging.rst b/docs/development/design/logging.rst new file mode 100644 index 0000000..2c2dfe6 --- /dev/null +++ b/docs/development/design/logging.rst @@ -0,0 +1,184 @@ +####### +Logging +####### + +************ +Installation +************ + +Currently, we use the `sample configuration`_ in Istio to install fluentd:: + + cd clover/logging + +First, install logging stack Elasticsearch, Fluentd and Kibana:: + + kubectl apply -f install/logging-stack.yaml + +Note that, it must be done in separated steps. If you run ``kubectl apply -f +install`` instead, the mixer adapter may fail to intialize because the target +service can not be found. You may find an error message from mixer container:: + + 2018-05-09T02:43:14.435156Z error Unable to initialize adapter: + snapshot='6', handler='handler.fluentd.istio-system', adapter='fluentd', + err='adapter instantiation error: dial tcp: lookup fluentd-es.logging on + 10.96.0.10:53: no such host'. + +Then configure fluentd for istio:: + + kubectl apply -f install/fluentd-istio.yaml + +Configure fluentd for node level logging:: + + kubectl apply -f install/fluentd-daemonset-elasticsearch-rbac.yaml + +.. _sample configuration: https://istio.io/docs/tasks/telemetry/fluentd.html + +******** +Validate +******** + +The scripts in ``clover/logging`` validates fluentd installation:: + + python clover/logging/validate.py + +It validates the installation with the following criterias + +#. existence of fluented pod +#. fluentd input is configured correctly +#. TBD + +************************** +Understanding how it works +************************** + +In clover stack, Istio is configured to automatically gather logs for services +in a mesh. More specificly, it is configured in `Mixer`_:: + +- when to log +- what to log +- where to log + +.. _Mixer: https://istio.io/docs/concepts/policy-and-control/mixer.html + +When to log +=========== + +Istio defines when to log by creating a custom resource ``rule``. For example: + +.. code-block:: yaml + + apiVersion: "config.istio.io/v1alpha2" + kind: rule + metadata: + name: newlogtofluentd + namespace: istio-system + spec: + match: "true" # match for all requests + actions: + - handler: handler.fluentd + instances: + - newlog.logentry + +This rule specifies that all instances of ``newlog.logentry`` that matches the +expression will be handled by the specified handler ``handler.fluentd``. We +shall explain ``instances`` and ``handler`` later. The expression ``true`` means +whenever a request arrive at Mixer, it will trigger the actions defined belows. + +``rule`` is a custom resource definition from `Istio installation`_. + +.. code-block:: yaml + + # Rule to send logentry instances to the fluentd handler + kind: CustomResourceDefinition + apiVersion: apiextensions.k8s.io/v1beta1 + metadata: + name: rules.config.istio.io + labels: + package: istio.io.mixer + istio: core + spec: + group: config.istio.io + names: + kind: rule + plural: rules + singular: rule + scope: Namespaced + version: v1alpha2 + +.. _Istio installation: https://github.com/istio/istio/blob/master/install/kubernetes/templates/istio-mixer.yaml.tmpl + +What to log +=========== + +The instance defines what content to be logged. + +> A (request) instance is the result of applying request attributes to the +> template mapping. The mapping is specified as an instance configuration. + +For example: + +.. code-block:: yaml + + # Configuration for logentry instances + apiVersion: "config.istio.io/v1alpha2" + kind: logentry + metadata: + name: newlog + namespace: istio-system + spec: + severity: '"info"' + timestamp: request.time + variables: + source: source.labels["app"] | source.service | "unknown" + user: source.user | "unknown" + destination: destination.labels["app"] | destination.service | "unknown" + responseCode: response.code | 0 + responseSize: response.size | 0 + latency: response.duration | "0ms" + monitored_resource_type: '"UNSPECIFIED"' + +The keys under ``spec`` should conform to the template. To learn what fields +are available and valid type, you may need to reference the corresponding +template, in this case, `Log Entry template`_. + +The values of each field could be either `Istio attributes`_ or an expression. + +> A given Istio deployment has a fixed vocabulary of attributes that it +> understands. The specific vocabulary is determined by the set of attribute +> producers being used in the deployment. The primary attribute producer in +> Istio is Envoy, although Mixer and services can also introduce attributes. + +Refer to the `Attribute Vocabulary`_ to learn the full set. + +By the way, ``logentry`` is also a custom resource definition created by Istio. + +.. _Istio attributes: https://istio.io/docs/concepts/policy-and-control/attributes.html +.. _Attribute Vocabulary: https://istio.io/docs/reference/config/mixer/attribute-vocabulary.html +.. _Log Entry template: https://istio.io/docs/reference/config/template/logentry.html + +Where to log +============ + +For log, the handler defines where these information will be handled, in this +example, a fluentd daemon on fluentd-es.logging:24224. + +.. code-block:: yaml + + # Configuration for a fluentd handler + apiVersion: "config.istio.io/v1alpha2" + kind: fluentd + metadata: + name: handler + namespace: istio-system + spec: + address: "fluentd-es.logging:24224" + +In this example, handlers (``handler.fluentd``) configure `Adapters`_ +(``fluentd``) to handle the data delivered from the created instances +(``newlog.logentry``). + +An adapter only accepts instance of specified kind. For example, +`fluentd adapter`_ accepts logentry but not other kinds. + +.. _Adapters: https://istio.io/docs/concepts/policy-and-control/mixer.html#adapters +.. _fluentd adapter: https://istio.io/docs/reference/config/adapters/fluentd.html |