summaryrefslogtreecommitdiffstats
path: root/clover
AgeCommit message (Collapse)AuthorFilesLines
2018-11-07Merge "Add ability to read controller nodeport IP from file"Stephen Wong29-133/+237
2018-11-07Merge "Fix Docker build of Clovisor Docker image on OPNFV Jenkins"Stephen Wong1-1/+2
2018-11-05Add ability to read controller nodeport IP from fileearrage29-133/+237
- Formalize use of cloverctl config file using viper - Default name of config file is '.cloverctl.yaml' - Look for file in home directory or relative to cloverctl executable - Single value in config is for 'ControllerIP: <IP Address>' - Still try and retrieve port value of clover-controller nodeport automatically - Still try and retrieve IP address of clover-controller LB if GKE environment is being used and override config file value if found in current kube context - Improve error messaging if controller connect fails - Clean up the CLI help output - Modify jmeter 'get testresult' command to avoid the use of flags and use arguments instead Change-Id: Id95d1b3d1079ac84996681db62da1f9b6b8a88b5 Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-11-03Fix Docker build of Clovisor Docker image on OPNFV JenkinsStephen Wong1-1/+2
Change-Id: Ied0ef2ccd4eb95cace68826a1494e82f2c18d95c Signed-off-by: Stephen Wong <stephen.kf.wong@gmail.com>
2018-11-02Modify container tag to opnfv in spark-submitearrage4-28/+28
- Change tag from local repo to opnfv for clover-spark container, which spark-submit references - Move update of configured visibility services into loop Change-Id: I9d09a851977c6006ea2cbba45d7a28be6a4660b9 Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-11-02Add minor enhancements to jmeter configurationearrage7-17/+75
- Add user-agent field to jmeter config yaml and jmeter jmx template that can be configured per URL in testplan - Add duration time that can be used in combination with ramp time. Duration is the total time of the test while ramp time governs how much time is allotted to establish all configured requests/connections - Explicitly define clover-jmeter-master service name with default namespace for clover-controller in clover-system namespace to reach - Add missing request_counts.html file Change-Id: I5e2d388db0b73ea6c28482c94c33cc2a01f1b52e Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-11-01Merge "CLOVER-43: Initial commit for Clovisor"Stephen Wong12-0/+1619
2018-11-01Merge "Initial UI dashboard added for visibility"Stephen Wong24-67/+18284
2018-11-01Merge "Initial commit for Spark to analyze visibility data"Stephen Wong13-0/+532
2018-11-01Merge "Add initial Istio validation testcase for CI"Stephen Wong2-2/+50
2018-11-01CLOVER-43: Initial commit for ClovisorStephen Wong12-0/+1619
Part of Clover's initial task (during project proposal) is to explore cloud native network tracing tools. Clover the project proposes Clovisor as a way for Clover to use the IOvisor (https://www.iovisor.org/) to perform network tracing. This is the first commit of this module Please note that the BPF code used in this commit were only tested with Linux kernel 4.14+ --- hence, if ubuntu is used, and one prefers LTS, the requirement is 18.04+ A different patch will track the redis and k8s API server watcher code Also, a design doc will be added in the future (TBD) Signed-off-by: Stephen Wong <stephen.kf.wong@gmail.com> Change-Id: I30d9f9d474b8703097c470d39628e86bc788f9b6
2018-10-31Initial UI dashboard added for visibilityearrage24-67/+18284
- Add foundation, jquery and jquery-ui open source js/css packages for skin, layout and AJAX - Add plotly and candela js open source graphing packages - Build core visibility view and provide toplevel templates, headers, menu navigation, etc. - Update collector/visibility APIs to align to latest Spark analytics and UI requirements - Refresh page elements at 5 second interval. - Visibility controls to stop continual page refresh and clear visibility. - Multi-select and update service set to analyze. - System count updates for tracing, spans, metrics schemas. - Min/max/avg response times per service (aggregate) table. - Graphs for: per service request counts, user-agent percentages, per service/URL counts, per URL/status code counts, per URL counts - Tabs for distinct URLs, status codes, user-agents, upstream cluster, operation names, nodeids (service names) - Table for metrics counters specified in visibility configuration. - Metrics over time scatter plots for tracked metrics. Change-Id: Ie3ea12fd32e0274ac52f78becc4a56c4c4134660 Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-10-22Fix broken jmeter binary linksearrage2-2/+2
- Docker container builds are failing due to expired links for the jmeter binary for jmeter-master and jmeter-slave - Updated link provided Change-Id: Icb364bc234b118e0c00359a04b51461142f832af Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-10-22Initial commit for Spark to analyze visibility dataearrage13-0/+532
- Add Apache Spark 2.3 with native Kubernetes support. - Runs self contained within K8s cluster in clover-system namespace. One container (clover-spark) includes Clover Spark JAR artifact. This container interacts with the K8s API to spawn a spark-driver pod. This pod in turn spawns executor pods to execute Spark jobs. - Currently JAR is included in source for convenience and must be built with sbt (install sbt and execute sbt package) - Includes JAR from DataStax to provide Cassandra connector to analyze Cassandra schemas as RDDs (Resilient Distributed Dataset). - Includes Redis interface JAR to write analyzed data back to visibility (UI, CLI or API). - Second container (clover-spark-submit) submits Spark jobs continuously to allow Spark to be operated entirely within the cluster. - Two Spark jobs (CloverSlow, CloverFast) allows some analytics to be provided in real-time and other analytics to be provided over longer horizons. - Each Spark job spawns two executor pods. - Includes yaml manifest to deploy clover-spark-submit with the necessary RBAC permissions to interact with the K8s API. - Data analyzed includes tracing and metrics schemas obtained by clover-collector and written to Cassandra. - Docker builds of clover-spark and clover-spark-submit are provided and will be pushed as OPNFV DockerHub images in a separate patch. Change-Id: I2e92c41fd75d4ebba948c0f8cb60face57005e50 Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-10-21Merge "Add visibility API in controller and CLI"Stephen Wong9-37/+231
2018-10-21Merge "Add nginx services control functionality"Stephen Wong8-37/+181
2018-10-21Merge "Improve data ingestion reliability and functionality"Stephen Wong8-86/+128
2018-10-21Merge "Remove former check_envs call"Stephen Wong1-2/+0
2018-10-19Add initial Istio validation testcase for CIJingLu52-2/+50
This work adds testcase for validating Istio installation Change-Id: Ie895b27eb4b3fbec23cf9519699f5fe37236153c Signed-off-by: JingLu5 <lvjing5@huawei.com>
2018-10-17Remove former check_envs callCédric Ollivier1-2/+0
Else it raises exceptions in __init__(). Change-Id: I4af1f59fbb9a94da1df132e750525fcdb5d0917b Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-10-17Add spinnaker configguidewutianwei2-1/+15
- Supply manifest for creation PV - Add the spinnaker as a service documentation - set the ci.jenkins.enabled to false Change-Id: I66d91aa133085c6c5910913da4303c05fcba9800 Signed-off-by: wutianwei <wutianwei1@huawei.com>
2018-10-15Add visibility API in controller and CLIearrage9-37/+231
- Modify get visibility to retrieve config and stats - Add visibility REST API in controller to clear, set, and get from redis - Add example yaml to set visibility (service list by name, metric suffixes/prefixes, and custom metrics) from CLI - Modify example yaml to start visibility (collector) for Istio 1.0 from CLI Change-Id: I43304ff6d70bb4b817b345b9c383ce3621fb06c7 Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-10-11Add nginx services control functionalityearrage8-37/+181
- Modify nginx server template to use clover-controller in clover-system namespace for file upload - Add ability to set nginx server paths and move uploaded files in clover-controller and set/get file upload metadata - Add cloverctl commands to set nginx server and lb configurations - Add example yaml files to configure nginx server and lb v1/v2 from cloverctl - Modify service definition for http-lb versions to distinguish the two versions for GRPC messaging from clover-controller in SDC deployment yaml Change-Id: I5c6866c5ff3de358939c58ea8c0bde64a69c6eca Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-10-10Improve data ingestion reliability and functionalityearrage8-86/+128
- Modify deployment namespace to clover-system and account for cassandra moving to the clover-system namespace - Increase k8s compute resource assigned to cassandra to deal with performance issues - Add additional fields (user-agent, request/response size, status codes) to span schema definition and modify primary keys - Improve exception handling to prevent collect process from crashing - Minor changes to support tracing/monitoring with Istio 1.0 - Inhibit logging for debug messages - Increase time back and number of traces to fetch in each sampling interval to deal with Jaeger REST interface returning trace data out of order under load (tested to 300 conn/sec; 12K connections currently) - Move trace insert into batch mode to cassandra - Read visibility services to analyze from redis rather than defaults (cloverctl, UI or clover-controller REST will set) - Remove local directory copies in docker build, as image is based on base clover container Change-Id: Ibae98ef5057e52a6eeddd9ebbcfaeb644caec36c Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-09-05Spinnaker as a Servicewutianwei22-5/+1416
JIRA: CLOVER-52 1. Add mainfest to install the spinnaker in kubernetes cluster 2. after using mainfest to install spinnaker, we can interacte with the halyard daemon with its REST API and we can add/delete/list the dockerRegistry/kubernetes accounts. 3. Add the cloverctl to interate with the halyard daemon Change-Id: I71bc5977f2d65aab88fa55f7d7a53ab75eb6a46b Signed-off-by: wutianwei <wutianwei1@huawei.com>
2018-08-19Merge "Implement initial clover-controller service"Stephen Wong22-0/+771
2018-08-19Merge "Implement initial cloverctl CLI tool"Stephen Wong30-0/+1533
2018-08-19Merge "Implement initial Jmeter master/slave containers"Stephen Wong15-0/+956
2018-08-02Implement initial cloverctl CLI toolEddie Arrage30-0/+1533
- Uses client-go package to interface to k8s API and implement functions as cloverkube package. - Identifies GKE LB IP for clover-controller for user - Identifies NodePort port number for clover-controller for user if environment is local k8s (assumes flannel CNI currently) - Deploys and deletes clover-collector and clover-controller with native client-go constructs (currently images are defined with local registry). Future work will implement other clover services and Istio components. Uses the clover-system namespace. - Uses Cobra go package to implement CLI (used in kubectl and istioctl) using cloverctl <verb> <noun> convention. - Interfaces to clover-controller to configure clover services (visibility, IDS ...) within the cluster via REST messaging - Start visibility (collector) engine using input yaml file or defaults - Init, stop and clear (truncate Cassandra tables) visibility engine or get basic stats. - Add custom rules to IDS from input yaml file and start/stop IDS - Generate jmeter testplan on jmeter-master using input yaml file. Start tests and output log/results from CLI. - Specify number of jmeter slaves to initiate tests on from CLI. Automatically find IP addresses of jmeter slaves within the k8s cluster. - Sample yaml files for adding IDS rules, starting visibility engine and generating jmeter test plans. - Build script to install go and get dependent packages. - Implement a custom Istio inject package for manual sidecar injection (cloverinject). Currently, unused as it is built from Istio 0.8.0/1.0.0 code base. Change-Id: Ibb8d08cb98267bdffb8905c221473f177d51bbb3 Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-08-01Merge "Adding cassandra service with a separate yaml file."Stephen Wong1-0/+97
2018-08-01Merge "[Clover] using elasticsearch to store log"Stephen Wong2-1/+130
2018-07-31Implement initial clover-controller serviceEddie Arrage22-0/+771
- First pass of clover-controller which resides within the k8s cluster and provides interfaces to all Clover services - Only service that should need to be exposed outside of cluster - Docker build of container that uses stack of nginx, gunicorn and flask to provide REST interface - REST interface is intended to serve cloverctl CLI and dashboard browser UI - Implements GRPC messaging to clover-collector and snort - GRPC interfaces files for snort/nginx are added to container from repo. Collector GRPC files will be removed from controller/control/api once patch below is merged https://gerrit.opnfv.org/gerrit/#/c/57245/ and added similarly - Provides first pass callback for file upload from clover-server. - Some REST messages implement JSON for passing params to internal services - Redis interface added to obtain data from services. Currently, a simple interface to retrieve snort event information - YAML manifest renderer to add to k8s. Uses NodePort service currently, defaulting to port 32044. - Removed collector gRPC interface files with merge of collector - Expose tracing and monitoring host/port parameters, as these vary depending on Istio version and Jaeger version - Add logging to flask blueprints - Added jmeter blueprint interface with REST for testplan generation, start test and result retrieval - Added flask Response to REST reply messages - Retrieve some basic stats from collector in json response Change-Id: I59eaeb860445ade4b45bba22747a61fb0cf0bbd4 Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-07-28Implement initial Jmeter master/slave containersEddie Arrage15-0/+956
- Jmeter can be used for L4-7 functional and performance testing - Jmeter master has gRPC server for management - Generates Jmeter test plans from minimal yaml params file (sample to be added with cloverctl) using template - Optionally span tests across slave containers to allow greater loads to be generated - Specify loop/thread/slave count and URL list, which dictates target and number of connections that will be attempted - clover-controller will interface to gRPC interface on Jmeter master - Start tests on master and retrieve log/result files - Render master and slave k8s manifests files Change-Id: Id144c8f551b7d375ff252c8de0611f895b50387c Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-07-23Adding cassandra service with a separate yaml file.iharijono1-0/+97
- Left the file samples/scenarios/service_delivery_controller_opnfv.yaml unchanged. - Added a yaml definition of Cassandra StatefulSet and its service into a separate file under tools directory - Cassandra Service run with 1 replica - Deleted 'data-plane-ns' and use 'default' instead for cassandra containers. - Revoked changes for samples/scenarios/service_delivery_controller_opnfv.yaml. - Added new line (Wutien suggested it) JIRA: CLOVER-000 Change-Id: I2bb4249cf2523f5011d6fefc69dc469a90e20eaf Signed-off-by: iharijono <indra.harijono@huawei.com>
2018-07-07Merge "Initial commit for Clover Collector"Stephen Wong19-2/+1197
2018-06-29[Clover] using elasticsearch to store logwutianwei2-1/+130
Change-Id: I0335fa912a3ca2dff5c989fa06183065216f10e4 Signed-off-by: wutianwei <wutianwei1@huawei.com>
2018-06-20Sleep 1 sec after set test.wutianwei1-0/+2
if we set testid and start test immediately, the first test's result can't be got from jaeger Change-Id: Ia2ab8a91d8c5f9956ea4d3d7c2436fb05490acee Signed-off-by: wutianwei <wutianwei1@huawei.com>
2018-06-15Merge "Streaming logs from application container"Stephen Wong1-0/+32
2018-06-12Initial commit for Clover CollectorEddie Arrage19-2/+1197
- Added a container named clover-collector using clover container as a base with build script - GRPC server to manage collector process - Cassandra DB client interface to initialize visibility keyspace - Init messaging adds table schemas for tracing - traces & spans - Adds table for monitoring - metrics - Does not implement Cassandra server but developed using public Cassandra docker container - Collector process in simple loop that periodically fetches traces and monitoring data and inserts to Cassandra - not optimized for batch retrieval yet for monitoring - CLI interface added to collector process and used by GRPC server for configuration - Simple GRPC client script to test GRPC server and start/stop of collector process - Collector process can be configured with access for tracing, monitoring and Cassandra - Added a return value in monitoring query method - Added ability to truncate tracing, metrics and spans tables in cql - Added cql prepared statements and batch insert for metrics and spans - Align cql connection to cql deployment within k8s - Fix issue with cql host list using ast and collect process args with background argument - Added redis interface to accept service/metric list externally for monitoring (will work in conjunction with clover-controller) - Use k8s DNS names and default ports for monitoring, tracing and cassandra - Added yaml manifest renderer/template for collector Change-Id: I3e4353e28844c4ce9c185ff4638012b66c7fff67 Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-05-31Add support for node level loggingYujun Zhang1-0/+96
Change-Id: Ib5b2240de3276164fe9e272bf36f0d1f89f409c0 Signed-off-by: Yujun Zhang <zhang.yujunz@zte.com.cn>
2018-05-22Streaming logs from application containerYujun Zhang1-0/+32
Change-Id: I6a1e526bec4160bcdac32d4124acb110b9cf6959 Signed-off-by: Yujun Zhang <zhang.yujunz@zte.com.cn>
2018-04-25Merge "Experimental commit for A-B testing with Clover Fraser release and on ↵Stephen Wong15-4/+646
the SDC application"
2018-04-23Experimental commit for A-B testing with Clover Fraser releaseStephen Wong15-4/+646
and on the SDC application Change-Id: I6e1bd84a6d674a2c4c4484722b20415f5402a59c Signed-off-by: Stephen Wong <stephen.kf.wong@gmail.com>
2018-04-17Cover validation of elasticsearch clusterYujun Zhang3-0/+48
- cluster health is not red - indics found - log entry created by istio found - requests in and out http load balance matches pytest is used as the test runner and wrapped in `validate.py` Change-Id: Iad540b69d05118fadc97df679cf3424513c15e38 Signed-off-by: Yujun Zhang <zhang.yujunz@zte.com.cn>
2018-04-11Modified validation script for tracing to support CIEddie Arrage2-11/+30
- Changed default Jaeger ports to 16686 for use with basic kubernetes port-forward and CI scripts - Added CLI to validate script to disable istio service check by default. This requires at least a single http request to istio-ingress after Jaeger deployment. It can be enabled with 'python validate.py -s'. Port and IP address for Jaeger can optionally be specified with '-ip' and '-port' options - Modified tracing doc to add k8s port-forward example in addition to k8s expose Change-Id: I10fb4d3cccfa50370d44ed7446f67a49c538bba9 Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-04-05Add existing public redis pod/serviceEddie Arrage1-0/+45
- Use a community yaml for redis in k8s as simple data store - Redis can be used for tracing and also by the snort-ids to store alerts that can be processed by other services - If flannel is used, the redis CLI can be accessed on the host OS with redis-cli -h <flannel ip> - Within the k8s cluster, the redis service can be accessed with DNS using name 'redis' - The same yaml for redis is also included in toplevel manifest for SDC scenario. Included here if intention is to use separately (tracing only) Change-Id: Ibad283a4cc8938fe01f5de6b7743bdb5511be3af Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-04-04Add clover initial docker image build scriptQiLiang5-14/+12
- install dependent deb/pip packages - install basic tools istioctl, kubectl - install clover source code - build/upload docker image script - update requirements.txt - update module import path - To use this image use need setup kube-config file. e.g. `docker run -v /root/config:/root/.kube/config -it clover bash` Change-Id: I91044bb99ce8e2b785ef03212d961a97b3d42233 Signed-off-by: QiLiang <liangqi1@huawei.com>
2018-03-30Clover initial commit for servicemesh/route_rules,Stephen Wong8-0/+578
orchestration/kube_client, and tools/clover_validate_rr Add an 'orchestration' directory. Please note that 'orchestration' does NOT mean Clover does any orchestration --- similar to how Clover doesn't by itself implement tracing or logging, orchestration is a directory for code related to Docker orchestration client --- such as k8s client kube_client utilizes the Kubernetes python client (a dependency) to perform tasks against Kubernetes API server. For this commit, it is only tested for weighted route rule verification, it does three tasks: (1) get a list of pods under a namespace --- pod dictionary now only contains pod name and label dictionary: used to match pod name with the node name in traces from OpenTracing (2) check to see if a particular pod is up in a particular namespace: used to check if Istio pods are running in istio-system namespace (3) check if a container exists in a list of pods under a namespace: used to check if application pods have istio-proxy container running route_rule directly invokes istioctl as there isn't any Istio Python client yet. Currently it reads and parses routerules from Istio, and validates if a particular trace result matches the routerules Finally, a sample tool clover_validate_rr is provided. This tool assumes a previous test has been ran (with an id with both the route-rule-under-test and corresponding traces are stored --- currently the assumption is tests were ran with redis-master running on system). The tool can be invoked: python clover_validate_rr.py -t <test-id> -s <service name> where test-id is the ID of the test (most likely uuid) and service name is the name of the service running in the Kubernetes cluster upon which test traces should be fetched against Change-Id: Ic8ab6efc23c71ac4643bee796ef986a86f6fc7dd Signed-off-by: Stephen Wong <stephen.kf.wong@gmail.com>
2018-03-30Merge "Initial commit for logging installation and validataion"Stephen Wong3-0/+301
2018-03-30Merge "Initial commit for tracing"Stephen Wong3-0/+314