Initial commit for Spark to analyze visibility data

- Add Apache Spark 2.3 with native Kubernetes support.
- Runs self contained within K8s cluster in clover-system
namespace. One container (clover-spark) includes Clover Spark
JAR artifact. This container interacts with the
K8s API to spawn a spark-driver pod. This pod in turn spawns executor
pods to execute Spark jobs.
- Currently JAR is included in source for convenience and must be
built with sbt (install sbt and execute sbt package)
- Includes JAR from DataStax to provide Cassandra connector to analyze
Cassandra schemas as RDDs (Resilient Distributed Dataset).
- Includes Redis interface JAR to write analyzed data back to visibility
(UI, CLI or API).
- Second container (clover-spark-submit) submits Spark jobs
continuously to allow Spark to be operated entirely within the cluster.
- Two Spark jobs (CloverSlow, CloverFast) allows some analytics to be
provided in real-time and other analytics to be provided over longer
horizons.
- Each Spark job spawns two executor pods.
- Includes yaml manifest to deploy clover-spark-submit with the
necessary RBAC permissions to interact with the K8s API.
- Data analyzed includes tracing and metrics schemas obtained by
clover-collector and written to Cassandra.
- Docker builds of clover-spark and clover-spark-submit are provided
and will be pushed as OPNFV DockerHub images in a separate patch.

Change-Id: I2e92c41fd75d4ebba948c0f8cb60face57005e50
Signed-off-by: earrage <eddie.arrage@huawei.com>

1 files changed, 55 insertions, 0 deletions

diff --git a/clover/spark/src/main/scala/CloverFast.scala b/clover/spark/src/main/scala/CloverFast.scala
new file mode 100644
index 0000000..b746760
--- /dev/null
+++ b/clover/spark/src/main/scala/CloverFast.scala
@@ -0,0 +1,55 @@
+// Copyright (c) Authors of Clover
+//
+// All rights reserved. This program and the accompanying materials
+// are made available under the terms of the Apache License, Version 2.0
+// which accompanies this distribution, and is available at
+// http://www.apache.org/licenses/LICENSE-2.0
+
+import org.apache.spark.sql.SparkSession
+import com.datastax.spark.connector._
+import org.apache.spark.sql.cassandra._
+import org.apache.spark.SparkContext
+import org.apache.spark.SparkConf
+
+import com.redis._
+
+object CloverFast {
+  def main(args: Array[String]) {
+    val sp = SparkSession.builder.appName("Clover Fast")
+                         .getOrCreate()
+    sp.stop()
+
+    val CassandraConnect = "cassandra.clover-system"
+    val RedisConnect = "redis.default"
+
+    // Cassandra, Redis, Spark Context
+    val scch = "spark.cassandra.connection.host"
+    val conf = new SparkConf(true).set(scch, CassandraConnect)
+    val redis = new RedisClient(RedisConnect, 6379)
+    val sc = new SparkContext(conf)
+
+    for(  x <- 1 to 10000 ) {
+
+        try {
+            val spans = sc.cassandraTable("visibility", "spans")
+                          .select("spanid").cassandraCount()
+            redis.set("span_count", spans)
+
+            val traces = sc.cassandraTable("visibility", "traces")
+                           .select("traceid").cassandraCount()
+            redis.set("trace_count", traces)
+
+            val metrics = sc.cassandraTable("visibility", "metrics")
+                            .select("monitor_time").cassandraCount()
+            redis.set("metric_count", metrics)
+
+        }
+        catch {
+            case unknown : Throwable => println("System counts exception: "
+                                                + unknown)
+        }
+
+    }
+
+  }
+}