summaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)AuthorFilesLines
2018-11-07Add create/delete of clover-system services from CLIearrage26-244/+936
- Improve usability when deploying or deleting various Clover components using cloverctl - Add yaml for each of the clover-system services including controller, collector, spark, redis, cassandra, clovisor and jmeter separated by k8s resource and available under a single yaml directory - Deploy/delete yaml for services in various subsets including: - visibility(controller, collector, spark, redis, cassandra) - datastore(redis, cassandra) - validation(jmeter master/slaves(4)) - clovisor (individually) - collector (individually) - controller (individually) Ex. cloverctl create system controller Ex. cloverctl create system visibility - Ability to add nodeport for controller separately (cloverctl create system controller nodeport) - Use native client-go methods to create/delete all k8s resources used by Clover including: - pods, deployments, statefulsets, daemonsets, services - serviceaccounts, clusterrolebindings - Above allows yaml override of values such as image value specified in deployment or daemonset. This is broken down by: - tag (ex. latest, opnfv-7.0.0) or repo'(ex. opnfv, localhost:5000) that can be added with CLI flags as shown below: (cloverctl create system visibility -t latest -r localhost:5000) - defaults to -r opnfv -t latest - Creates/deletes clover-system namespace - Also added ability to create/delete lb service for controller external access when running on environment such as GKE (cloverctl <create/delete> system controller lb) Change-Id: I2a3c6c80035d4663fa38368b3ff13e9a14090a47 Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-11-02Add minor enhancements to jmeter configurationearrage7-17/+75
- Add user-agent field to jmeter config yaml and jmeter jmx template that can be configured per URL in testplan - Add duration time that can be used in combination with ramp time. Duration is the total time of the test while ramp time governs how much time is allotted to establish all configured requests/connections - Explicitly define clover-jmeter-master service name with default namespace for clover-controller in clover-system namespace to reach - Add missing request_counts.html file Change-Id: I5e2d388db0b73ea6c28482c94c33cc2a01f1b52e Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-11-01Merge "CLOVER-43: Initial commit for Clovisor" into stable/gambiaStephen Wong12-0/+1619
2018-11-01Merge "Initial UI dashboard added for visibility" into stable/gambiaStephen Wong24-67/+18284
2018-11-01Merge "Update modsecurity image docker registry" into stable/gambiaStephen Wong1-1/+1
2018-11-01Merge "Initial commit for Spark to analyze visibility data" into stable/gambiaStephen Wong13-0/+532
2018-11-01Merge "Add initial Istio validation testcase for CI" into stable/gambiaStephen Wong5-17/+83
2018-11-01CLOVER-43: Initial commit for ClovisorStephen Wong12-0/+1619
Part of Clover's initial task (during project proposal) is to explore cloud native network tracing tools. Clover the project proposes Clovisor as a way for Clover to use the IOvisor (https://www.iovisor.org/) to perform network tracing. This is the first commit of this module Please note that the BPF code used in this commit were only tested with Linux kernel 4.14+ --- hence, if ubuntu is used, and one prefers LTS, the requirement is 18.04+ A different patch will track the redis and k8s API server watcher code Also, a design doc will be added in the future (TBD) Signed-off-by: Stephen Wong <stephen.kf.wong@gmail.com> Change-Id: I30d9f9d474b8703097c470d39628e86bc788f9b6
2018-10-31Initial UI dashboard added for visibilityearrage24-67/+18284
- Add foundation, jquery and jquery-ui open source js/css packages for skin, layout and AJAX - Add plotly and candela js open source graphing packages - Build core visibility view and provide toplevel templates, headers, menu navigation, etc. - Update collector/visibility APIs to align to latest Spark analytics and UI requirements - Refresh page elements at 5 second interval. - Visibility controls to stop continual page refresh and clear visibility. - Multi-select and update service set to analyze. - System count updates for tracing, spans, metrics schemas. - Min/max/avg response times per service (aggregate) table. - Graphs for: per service request counts, user-agent percentages, per service/URL counts, per URL/status code counts, per URL counts - Tabs for distinct URLs, status codes, user-agents, upstream cluster, operation names, nodeids (service names) - Table for metrics counters specified in visibility configuration. - Metrics over time scatter plots for tracked metrics. Change-Id: Ie3ea12fd32e0274ac52f78becc4a56c4c4134660 Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-10-30Update modsecurity image docker registryJingLu51-1/+1
The clover-ns-modsecurity-crs docker image is now built by Jenkins and uploaded to the OPNFV docker hub. This patch updates the modsecurity deployment kubernetes manifest to pull image from OPNFV docker hub. Change-Id: I7e8afd27f726600de9247fc98fe3f5d491000f1c Signed-off-by: JingLu5 <lvjing5@huawei.com>
2018-10-22Fix broken jmeter binary linksearrage2-2/+2
- Docker container builds are failing due to expired links for the jmeter binary for jmeter-master and jmeter-slave - Updated link provided Change-Id: Icb364bc234b118e0c00359a04b51461142f832af Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-10-22Initial commit for Spark to analyze visibility dataearrage13-0/+532
- Add Apache Spark 2.3 with native Kubernetes support. - Runs self contained within K8s cluster in clover-system namespace. One container (clover-spark) includes Clover Spark JAR artifact. This container interacts with the K8s API to spawn a spark-driver pod. This pod in turn spawns executor pods to execute Spark jobs. - Currently JAR is included in source for convenience and must be built with sbt (install sbt and execute sbt package) - Includes JAR from DataStax to provide Cassandra connector to analyze Cassandra schemas as RDDs (Resilient Distributed Dataset). - Includes Redis interface JAR to write analyzed data back to visibility (UI, CLI or API). - Second container (clover-spark-submit) submits Spark jobs continuously to allow Spark to be operated entirely within the cluster. - Two Spark jobs (CloverSlow, CloverFast) allows some analytics to be provided in real-time and other analytics to be provided over longer horizons. - Each Spark job spawns two executor pods. - Includes yaml manifest to deploy clover-spark-submit with the necessary RBAC permissions to interact with the K8s API. - Data analyzed includes tracing and metrics schemas obtained by clover-collector and written to Cassandra. - Docker builds of clover-spark and clover-spark-submit are provided and will be pushed as OPNFV DockerHub images in a separate patch. Change-Id: I2e92c41fd75d4ebba948c0f8cb60face57005e50 Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-10-22Add initial Istio validation testcase for CIJingLu55-17/+83
This work adds testcase for validating Istio installation Change-Id: Ie895b27eb4b3fbec23cf9519699f5fe37236153c Signed-off-by: JingLu5 <lvjing5@huawei.com>
2018-10-21Merge "Add visibility API in controller and CLI" into stable/gambiaStephen Wong9-37/+231
2018-10-21Merge "Add nginx services control functionality" into stable/gambiaStephen Wong11-42/+230
2018-10-21Merge "Improve data ingestion reliability and functionality" into stable/gambiaStephen Wong8-86/+128
2018-10-21Merge "Remove former check_envs call" into stable/gambiaStephen Wong1-2/+0
2018-10-17Remove former check_envs callCédric Ollivier1-2/+0
Else it raises exceptions in __init__(). Change-Id: I4af1f59fbb9a94da1df132e750525fcdb5d0917b Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-10-16Add visibility API in controller and CLIearrage9-37/+231
- Modify get visibility to retrieve config and stats - Add visibility REST API in controller to clear, set, and get from redis - Add example yaml to set visibility (service list by name, metric suffixes/prefixes, and custom metrics) from CLI - Modify example yaml to start visibility (collector) for Istio 1.0 from CLI Change-Id: I43304ff6d70bb4b817b345b9c383ce3621fb06c7 Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-10-15Add spinnaker configguidewutianwei3-1/+255
- Supply manifest for creation PV - Add the spinnaker as a service documentation - set the ci.jenkins.enabled to false Change-Id: I66d91aa133085c6c5910913da4303c05fcba9800 Signed-off-by: wutianwei <wutianwei1@huawei.com>
2018-10-12Add nginx services control functionalityearrage11-42/+230
- Modify nginx server template to use clover-controller in clover-system namespace for file upload - Add ability to set nginx server paths and move uploaded files in clover-controller and set/get file upload metadata - Add cloverctl commands to set nginx server and lb configurations - Add example yaml files to configure nginx server and lb v1/v2 from cloverctl - Modify service definition for http-lb versions to distinguish the two versions for GRPC messaging from clover-controller in SDC deployment yaml Change-Id: I5c6866c5ff3de358939c58ea8c0bde64a69c6eca Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-10-12Improve data ingestion reliability and functionalityearrage8-86/+128
- Modify deployment namespace to clover-system and account for cassandra moving to the clover-system namespace - Increase k8s compute resource assigned to cassandra to deal with performance issues - Add additional fields (user-agent, request/response size, status codes) to span schema definition and modify primary keys - Improve exception handling to prevent collect process from crashing - Minor changes to support tracing/monitoring with Istio 1.0 - Inhibit logging for debug messages - Increase time back and number of traces to fetch in each sampling interval to deal with Jaeger REST interface returning trace data out of order under load (tested to 300 conn/sec; 12K connections currently) - Move trace insert into batch mode to cassandra - Read visibility services to analyze from redis rather than defaults (cloverctl, UI or clover-controller REST will set) - Remove local directory copies in docker build, as image is based on base clover container Change-Id: Ibae98ef5057e52a6eeddd9ebbcfaeb644caec36c Signed-off-by: earrage <eddie.arrage@huawei.com>
2018-10-06Updated from global requirementsCédric Ollivier1-1/+1
Change-Id: I92fc0937130b0c70dc5888b64b1082e8816969d4 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-09-19Merge "Local Documentation Builds"Stephen Wong5-1/+24
2018-09-19Merge "Add ModSecurity config guide"Stephen Wong8-3/+370
2018-09-19Merge "Spinnaker as a Service"Stephen Wong22-5/+1416
2018-09-18Local Documentation BuildsAric Gardner5-1/+24
This adds configuration for performing local documentation builds with the following simple command: tox -e docs Change-Id: Ifb8fce66e59432863d042a96308e6c3a89fb11a8 Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
2018-09-08Fix missing psutil dependency in nginx servicesEddie Arrage3-3/+3
- A psutil dependency was added in the latest Clover server version that was not added to the docker builds of the lb/proxy Change-Id: Id48c3446a93d6b9cef65d453266d0bf15eddf6cc Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-09-07Add ModSecurity config guideJingLu58-3/+370
This patch adds ModSecurity config guide. This patch also deploy the modsecurity and ext_authz filter to clover-gateway namespace. Change-Id: I5ab21e6337b8f8b839ddd028370df378686bd017 Signed-off-by: JingLu5 <lvjing5@huawei.com>
2018-09-05Merge "Add file upload/download and configure URL paths"Stephen Wong9-87/+393
2018-09-05Merge "Update entry_point to conform with Xtesting"Stephen Wong1-0/+3
2018-09-05Merge "Modsecurity as a service"Stephen Wong10-0/+434
2018-09-05Spinnaker as a Servicewutianwei22-5/+1416
JIRA: CLOVER-52 1. Add mainfest to install the spinnaker in kubernetes cluster 2. after using mainfest to install spinnaker, we can interacte with the halyard daemon with its REST API and we can add/delete/list the dockerRegistry/kubernetes accounts. 3. Add the cloverctl to interate with the halyard daemon Change-Id: I71bc5977f2d65aab88fa55f7d7a53ab75eb6a46b Signed-off-by: wutianwei <wutianwei1@huawei.com>
2018-09-04Merge "Add docker and deployment scripts for exemplar video streaming ↵Wenjing Chu5-0/+164
microservice"
2018-09-02Update entry_point to conform with XtestingCédric Ollivier1-0/+3
Functest is being updated to Xtesting 0.60 which leverages on stevedore [1]. [1] https://docs.openstack.org/stevedore/latest/ Change-Id: I2c162c8e08a580f80b547b3a1cd6cc700d12c858 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-08-31Merge "Add envoy.ext_authz filter"Stephen Wong1-0/+24
2018-08-30Add docker and deployment scripts for exemplar video streaming microserviceadarsh10015-0/+164
JIRA: CLOVER-72 This issue falls under the "Edge Cloud-Native Cluster" intern project and is a part of both Clover and Edge cloud projects. The scripts added are of an exemplar video streaming microservice designed to run on the raspberry pi edge cluster. This commit includes the docker scripts to containerize the UV4L stream and the kubernetes deployment script to correctly run the microservice as a pod on the cluster. The detailed description of the project and the instructions for using the scripts are documented in the README. Change-Id: Ice4e5e7e63c2eb617556eaef1255801ffac851e7 Signed-off-by: adarsh1001 <adarshpalsingh1996@gmail.com>
2018-08-29Bug fix in SDC after istio to 1.0.0Ace Lee1-12/+33
JIRA: CLOVER-84 There will be some Bug in SDC after we upgrade the Istio to 1.0.0 Istio 1.0 have some concept, for example : virtualservice gateway. So we change the yaml file using the 1.0.0 concepts. Add mirror function Change-Id: Id138cfec2c7d94b44eb508a056c91e193ac1b08b Signed-off-by: Ace Lee <liyin11@huawei.com>
2018-08-28Add file upload/download and configure URL pathsEddie Arrage9-87/+393
- Compile nginx from source in order to employ additional modules - Add nginx-upload-module for high performance file upload that avoids the need for file copies with a web application. - File upload allows for placement of files for file download for performance benchmarking. - File upload can also be used directly for bi-directional throughput testing having emulated clients upload files while file downloads simultaneously occur. - Nginx file upload stores files with hash to avoid conflicting file names. Upload block in nginx config is configured to send REST message to clover-controller with file metadata (original filename, size, etc.) clover-controller will be responsible for modifying the hashed filename and placing in a target directory within an nginx server. - Build also adds nginx-rtmp module to act as streaming media server L7 loader will be extended to fetch streaming files from RTMP servers. - Add ability to create directories in server site root and create the location directive(s) in nginx configuration - Separated upload for configuration (download files in various paths) from upload for testing (upload to create bi-directional session throughput) - Upload for testing does not sent upload metadata to clover-controller - Added ability to move upload files to file folders in the nginx site root to use for download - Delete files in upload folder - Fixed issue with 426 Upgrade Required error message when upload module sends upload metadata to clover-controller - Added server name to metadata sent to clover-controller Change-Id: Ib4cf6240f92360b82f378c062675f4fdaa19ca93 Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-08-28Add envoy.ext_authz filterJingLu51-0/+24
JIRA: CLOVER-86 This external authorization HTTP filter calls an external HTTP service (ModSecuruty service) to check if the incoming HTTP request is authorized or not. If the request is deemed unauthorized then the request will be denied normally with 403 (Forbidden) response. Change-Id: I0fe14c73defec027c54f42713cbdf69b0b83e102 Signed-off-by: JingLu5 <lvjing5@huawei.com>
2018-08-28Modsecurity as a serviceJingLu510-0/+434
JIRA: CLOVER-68 1. Add Dockerfile and related files to build clover's modsecurity Docekr container 2. Add mainfest to install the Modsecurity in kubernetes cluster Change-Id: Ia92926e730c04720f931999d7ec30565ce9e54be Signed-off-by: JingLu5 <lvjing5@huawei.com>
2018-08-21Merge "upgrade istio to 1.0.0"Stephen Wong2-2/+2
2018-08-19Merge "Implement initial clover-controller service"Stephen Wong22-0/+771
2018-08-19Merge "Implement initial cloverctl CLI tool"Stephen Wong30-0/+1533
2018-08-19Merge "Implement initial Jmeter master/slave containers"Stephen Wong15-0/+956
2018-08-14Merge "Add ansible scripts for raspberry pi based kubernetes edge cluster"Wenjing Chu10-0/+797
2018-08-10upgrade istio to 1.0.0Ace Lee2-2/+2
JIRA: CLOVER-84 we change the env in clover and change some deploy script. will upgrade the doc later Change-Id: I73a78afb91676efc3278b623c5d263a4a215ccd9 Signed-off-by: Ace Lee <liyin11@huawei.com>
2018-08-02Implement initial cloverctl CLI toolEddie Arrage30-0/+1533
- Uses client-go package to interface to k8s API and implement functions as cloverkube package. - Identifies GKE LB IP for clover-controller for user - Identifies NodePort port number for clover-controller for user if environment is local k8s (assumes flannel CNI currently) - Deploys and deletes clover-collector and clover-controller with native client-go constructs (currently images are defined with local registry). Future work will implement other clover services and Istio components. Uses the clover-system namespace. - Uses Cobra go package to implement CLI (used in kubectl and istioctl) using cloverctl <verb> <noun> convention. - Interfaces to clover-controller to configure clover services (visibility, IDS ...) within the cluster via REST messaging - Start visibility (collector) engine using input yaml file or defaults - Init, stop and clear (truncate Cassandra tables) visibility engine or get basic stats. - Add custom rules to IDS from input yaml file and start/stop IDS - Generate jmeter testplan on jmeter-master using input yaml file. Start tests and output log/results from CLI. - Specify number of jmeter slaves to initiate tests on from CLI. Automatically find IP addresses of jmeter slaves within the k8s cluster. - Sample yaml files for adding IDS rules, starting visibility engine and generating jmeter test plans. - Build script to install go and get dependent packages. - Implement a custom Istio inject package for manual sidecar injection (cloverinject). Currently, unused as it is built from Istio 0.8.0/1.0.0 code base. Change-Id: Ibb8d08cb98267bdffb8905c221473f177d51bbb3 Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-08-01Merge "Adding cassandra service with a separate yaml file."Stephen Wong2-1/+97
2018-08-01Merge "Adding clearwater IMS yaml and clearwater live test dockerfile to ↵Stephen Wong14-0/+363
test istio service-mesh"