Initial commit for Spark to analyze visibility data

- Add Apache Spark 2.3 with native Kubernetes support.
- Runs self contained within K8s cluster in clover-system
namespace. One container (clover-spark) includes Clover Spark
JAR artifact. This container interacts with the
K8s API to spawn a spark-driver pod. This pod in turn spawns executor
pods to execute Spark jobs.
- Currently JAR is included in source for convenience and must be
built with sbt (install sbt and execute sbt package)
- Includes JAR from DataStax to provide Cassandra connector to analyze
Cassandra schemas as RDDs (Resilient Distributed Dataset).
- Includes Redis interface JAR to write analyzed data back to visibility
(UI, CLI or API).
- Second container (clover-spark-submit) submits Spark jobs
continuously to allow Spark to be operated entirely within the cluster.
- Two Spark jobs (CloverSlow, CloverFast) allows some analytics to be
provided in real-time and other analytics to be provided over longer
horizons.
- Each Spark job spawns two executor pods.
- Includes yaml manifest to deploy clover-spark-submit with the
necessary RBAC permissions to interact with the K8s API.
- Data analyzed includes tracing and metrics schemas obtained by
clover-collector and written to Cassandra.
- Docker builds of clover-spark and clover-spark-submit are provided
and will be pushed as OPNFV DockerHub images in a separate patch.

Change-Id: I2e92c41fd75d4ebba948c0f8cb60face57005e50
Signed-off-by: earrage <eddie.arrage@huawei.com>

1 files changed, 26 insertions, 0 deletions

diff --git a/clover/spark/docker/spark-submit/runner_fast.sh b/clover/spark/docker/spark-submit/runner_fast.sh
new file mode 100755
index 0000000..2381351
--- /dev/null
+++ b/clover/spark/docker/spark-submit/runner_fast.sh
@@ -0,0 +1,26 @@
+#!/bin/bash
+#
+# Copyright (c) Authors of Clover
+#
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Apache License, Version 2.0
+# which accompanies this distribution, and is available at
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+
+IMAGE_NAME=${IMAGE_NAME:-"clover-spark:latest"}
+IMAGE_PATH=${IMAGE_PATH:-"localhost:5000"}
+CLASS_NAME=${CLASS_NAME:-"CloverFast"}
+JAR_NAME=${JAR_NAME:-"clover-spark_2.11-1.0.jar"}
+
+bin/spark-submit \
+  --master k8s://https://kubernetes.default.svc \
+  --deploy-mode cluster \
+  --name "clover-spark-fast" \
+  --class $CLASS_NAME \
+  --conf spark.executor.instances=2 \
+  --conf spark.kubernetes.container.image="$IMAGE_PATH/$IMAGE_NAME" \
+  --conf spark.kubernetes.authenticate.driver.serviceAccountName="clover-spark" \
+  --conf spark.kubernetes.namespace="clover-system" \
+  --jars local:///opt/spark/jars/redisclient_2.11-3.7.jar,local:///opt/spark/jars/datastax_spark-cassandra-connector-2.3.0-s_2.11.jar \
+  local:///opt/spark/jars/$JAR_NAME