From 2139f983fbe71bf6411259e8ddb460a79663dcb8 Mon Sep 17 00:00:00 2001 From: earrage Date: Mon, 22 Oct 2018 13:53:54 -0700 Subject: Initial commit for Spark to analyze visibility data - Add Apache Spark 2.3 with native Kubernetes support. - Runs self contained within K8s cluster in clover-system namespace. One container (clover-spark) includes Clover Spark JAR artifact. This container interacts with the K8s API to spawn a spark-driver pod. This pod in turn spawns executor pods to execute Spark jobs. - Currently JAR is included in source for convenience and must be built with sbt (install sbt and execute sbt package) - Includes JAR from DataStax to provide Cassandra connector to analyze Cassandra schemas as RDDs (Resilient Distributed Dataset). - Includes Redis interface JAR to write analyzed data back to visibility (UI, CLI or API). - Second container (clover-spark-submit) submits Spark jobs continuously to allow Spark to be operated entirely within the cluster. - Two Spark jobs (CloverSlow, CloverFast) allows some analytics to be provided in real-time and other analytics to be provided over longer horizons. - Each Spark job spawns two executor pods. - Includes yaml manifest to deploy clover-spark-submit with the necessary RBAC permissions to interact with the K8s API. - Data analyzed includes tracing and metrics schemas obtained by clover-collector and written to Cassandra. - Docker builds of clover-spark and clover-spark-submit are provided and will be pushed as OPNFV DockerHub images in a separate patch. Change-Id: I2e92c41fd75d4ebba948c0f8cb60face57005e50 Signed-off-by: earrage --- clover/spark/docker/spark-submit/build.sh | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100755 clover/spark/docker/spark-submit/build.sh (limited to 'clover/spark/docker/spark-submit/build.sh') diff --git a/clover/spark/docker/spark-submit/build.sh b/clover/spark/docker/spark-submit/build.sh new file mode 100755 index 0000000..1bcbbc2 --- /dev/null +++ b/clover/spark/docker/spark-submit/build.sh @@ -0,0 +1,17 @@ +#!/bin/bash +# +# Copyright (c) Authors of Clover +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Apache License, Version 2.0 +# which accompanies this distribution, and is available at +# http://www.apache.org/licenses/LICENSE-2.0 +# + +IMAGE_PATH=${IMAGE_PATH:-"localhost:5000"} +IMAGE_NAME=${IMAGE_NAME:-"clover-spark-submit:latest"} + +docker build -f Dockerfile -t $IMAGE_NAME . +docker tag $IMAGE_NAME $IMAGE_PATH/$IMAGE_NAME +docker push $IMAGE_PATH/$IMAGE_NAME + -- cgit 1.2.3-korg