diff options
Diffstat (limited to 'docs/testing/user')
10 files changed, 494 insertions, 21 deletions
diff --git a/docs/testing/user/userguide/kubestone/deployment_capacity.rst b/docs/testing/user/userguide/kubestone/deployment_capacity.rst new file mode 100644 index 00000000..06cdd886 --- /dev/null +++ b/docs/testing/user/userguide/kubestone/deployment_capacity.rst @@ -0,0 +1,59 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. + .. http://creativecommons.org/licenses/by/4.0 + .. (c) Huawei Technologies Co.,Ltd and others. + +*************************************************** +Kubenetes Stress Test of Deployment Capacity +*************************************************** + +Test Case +========= + ++-----------------------------------------------------------------------------+ +|Bottlenecks Kubestone Deployment Capacity Test | +| | ++--------------+--------------------------------------------------------------+ +|test case name| kubestone_deployment_capacity | +| | | ++--------------+--------------------------------------------------------------+ +|description | Stress test regarding capacity of deployment | ++--------------+--------------------------------------------------------------+ +|configuration | config file: | +| | testsuite/kubestone/testcases/deployment.yaml | +| | | +| | | ++--------------+--------------------------------------------------------------+ +|test result | Capcity, Life-Cycle Duration, Available Deployments | +| | | ++--------------+--------------------------------------------------------------+ + +Configuration +============ +:: + + apiVersion: apps/v1 + kind: Deployment + namespace: bottlenecks-kubestone + test_type: Horizontal-Scaling + scaling_steps: 10, 50, 100, 200 + template: None + metadata: + name: nginx-deployment + labels: + app: nginx + spec: + replicas: 3 + selector: + matchLabels: + app: nginx + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx:1.7.9 + ports: + - containerPort: 80 + diff --git a/docs/testing/user/userguide/overview.rst b/docs/testing/user/userguide/overview.rst index 697191b0..fc8fff14 100644 --- a/docs/testing/user/userguide/overview.rst +++ b/docs/testing/user/userguide/overview.rst @@ -38,6 +38,16 @@ Euphrates: * Extend Life-cycle test to data-plane to validate the system capability to handle concurrent networks usage. * Testing framework is revised to support installer-agnostic testing. +Gambia: + +* Introduction of kubetenetes stress test for scaling capability of PoD management +* Introduction of test scheduler supporting testing services ochestration while providing monitoring services + +Hunter: + +* Introduction of AI based algorithms to do anomaly detection and correlation analysis of Bottlenecks testing results +# Enhancing test scheduler to supporting testing results graphing + These enhancements and test cases help the end users to gain more comprehensive understanding of the SUT. Graphic reports of the system behavior additional to test cases are provided to indicate the confidence level of SUT. Installer-agnostic testing framework allow end user to do stress testing adaptively over either Open Source or commercial deployments. @@ -55,26 +65,38 @@ Integration Description +-------------+----------------------+----------------------+ | Euphrates | Any | POSCA | +-------------+----------------------+----------------------+ +| Fraser | Any | POSCA | ++-------------+----------------------+----------------------+ +| Gambia | Any | POSCA, kubestone | ++-------------+----------------------+----------------------+ +| Hunter | Any | POSCA, kubestone | ++-------------+----------------------+----------------------+ Test suite & Test case Description ================================== -+--------+-------------------------------------------+ -|POSCA | posca_factor_ping | -| +-------------------------------------------+ -| | posca_factor_system_bandwidth | -| +-------------------------------------------+ -| | posca_facotor_soak_througputs | -| +-------------------------------------------+ -| | posca_feature_vnf_scale_up | -| +-------------------------------------------+ -| | posca_feature_vnf_scale_out | -| +-------------------------------------------+ -| | posca_factor_storperf | -| +-------------------------------------------+ -| | posca_factor_multistack_storage_parallel | -| +-------------------------------------------+ -| | posca_factor_multistack_storage | -+--------+-------------------------------------------+ ++----------+----+-------------------------------------------+ +|POSCA | 1 | posca_factor_ping | +| +----+-------------------------------------------+ +| | 2 | posca_factor_system_bandwidth | +| +----+-------------------------------------------+ +| | 3 | posca_facotor_soak_througputs | +| +----+-------------------------------------------+ +| | 4 | posca_feature_vnf_scale_up | +| +----+-------------------------------------------+ +| | 5 | posca_feature_vnf_scale_out | +| +----+-------------------------------------------+ +| | 6 | posca_factor_storperf | +| +----+-------------------------------------------+ +| | 7 | posca_factor_multistack_storage_parallel | +| +----+-------------------------------------------+ +| | 8 | posca_factor_multistack_storage | +| +----+-------------------------------------------+ +| | 9 | posca_feature_moon_resources | +| +----+-------------------------------------------+ +| | 10 | posca_feature_moon_tenants | ++----------+----+-------------------------------------------+ +|Kubestone | 1 | deployment_capacity | ++----------+----+-------------------------------------------+ As for the abandoned test suite in the previous Bottlenecks releases, please refer to http://docs.opnfv.org/en/stable-danube/submodules/bottlenecks/docs/testing/user/userguide/deprecated.html. diff --git a/docs/testing/user/userguide/posca/posca_factor_ping.rst b/docs/testing/user/userguide/posca/posca_factor_ping.rst index 953ff4b7..e1870d49 100644 --- a/docs/testing/user/userguide/posca/posca_factor_ping.rst +++ b/docs/testing/user/userguide/posca/posca_factor_ping.rst @@ -39,8 +39,8 @@ Configuration tool: ping test_times: 100 package_size: - num_stack: 5, 10, 20 - package_loss: 10% + num_stack: 5, 5 + package_loss: 0 contexts: stack_create: yardstick diff --git a/docs/testing/user/userguide/posca/posca_factor_soak_throughputs.rst b/docs/testing/user/userguide/posca/posca_factor_soak_throughputs.rst new file mode 100644 index 00000000..9e427b15 --- /dev/null +++ b/docs/testing/user/userguide/posca/posca_factor_soak_throughputs.rst @@ -0,0 +1,51 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Tech and others. + +************************************* +POSCA Factor Test of Soak Throughputs +************************************* + +Test Case +======== + ++-----------------------------------------------------------------------------+ +|Bottlenecks POSCA Soak Test Throughputs | +| | ++--------------+--------------------------------------------------------------+ +|test case name| posca_factor_soak_throughputs | +| | | ++--------------+--------------------------------------------------------------+ +|description | Long duration stability tests of data-plane traffic | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | config file: | +| | /testsuite/posca/testcase_cfg/... | +| | posca_factor_soak_throughputs.yaml | +| | | ++--------------+--------------------------------------------------------------+ +|test result | THROUGHPUT,THROUGHPUT_UNITS,MEAN_LATENCY,LOCAL_CPU_UTIL, | +| | REMOTE_CPU_UTIL,LOCAL_BYTES_SENT,REMOTE_BYTES_RECVD | ++--------------+--------------------------------------------------------------+ + +Configuration +============ +:: + + load_manager: + scenarios: + tool: netperf + test_duration_hours: 1 + vim_pair_ttl: 300 + vim_pair_lazy_cre_delay: 2 + package_size: + threshhold: + package_loss: 0% + latency: 300 + + runners: + stack_create: yardstick + flavor: + yardstick_test_dir: "samples" + yardstick_testcase: "netperf_soak" diff --git a/docs/testing/user/userguide/posca/posca_feature_moon_resources.rst b/docs/testing/user/userguide/posca/posca_feature_moon_resources.rst new file mode 100644 index 00000000..5f89bca8 --- /dev/null +++ b/docs/testing/user/userguide/posca/posca_feature_moon_resources.rst @@ -0,0 +1,52 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Tech and others. + +************************************************************ +POSCA feature Test of Moon Security for resources per tenant +************************************************************ + +Test Case +========= + ++-----------------------------------------------------------------------------+ +|Bottlenecks POSCA Soak Test Throughputs | +| | ++--------------+--------------------------------------------------------------+ +|test case name| posca_feature_moon_resources | +| | | ++--------------+--------------------------------------------------------------+ +|description | Moon authentication capability test for maximum number of | +| | authentication operations per tenant | ++--------------+--------------------------------------------------------------+ +|configuration | config file: | +| | /testsuite/posca/testcase_cfg/... | +| | posca_feature_moon_resources.yaml | +| | | ++--------------+--------------------------------------------------------------+ +|test result | number of tenants, max number of users | +| | | ++--------------+--------------------------------------------------------------+ + +Configuration +============ +:: + + load_manager: + scenarios: + tool: https request + # info that the cpus and memes have the same number of data. + pdp_name: pdp + policy_name: "MLS Policy example" + model_name: MLS + tenants: 1,5,10,20 + subject_number: 10 + object_number: 10 + timeout: 0.2 + + runners: + stack_create: yardstick + Debug: False + yardstick_test_dir: "samples" + yardstick_testcase: "moon_resource" diff --git a/docs/testing/user/userguide/posca/posca_feature_moon_tenants.rst b/docs/testing/user/userguide/posca/posca_feature_moon_tenants.rst new file mode 100644 index 00000000..1ac93f8d --- /dev/null +++ b/docs/testing/user/userguide/posca/posca_feature_moon_tenants.rst @@ -0,0 +1,55 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Tech and others. + +*********************************************** +POSCA feature Test of Moon Security for Tenants +*********************************************** + +Test Case +========= + ++-----------------------------------------------------------------------------+ +|Bottlenecks POSCA Soak Test Throughputs | +| | ++--------------+--------------------------------------------------------------+ +|test case name| posca_feature_moon_tenants | +| | | ++--------------+--------------------------------------------------------------+ +|description | Moon authentication capability test for maximum tenants | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | config file: | +| | /testsuite/posca/testcase_cfg/... | +| | posca_feature_moon_tenants.yaml | +| | | ++--------------+--------------------------------------------------------------+ +|test result | Max number of tenants | +| | | ++--------------+--------------------------------------------------------------+ + +Configuration +============ +:: + + load_manager: + scenarios: + tool: https request + # info that the cpus and memes have the same number of data. + pdp_name: pdp + policy_name: "MLS Policy example" + model_name: MLS + subject_number: 20 + object_number: 20 + timeout: 0.003 + initial_tenants: 0 + steps_tenants: 1 + tolerate_time: 20 + SLA: 5 + + runners: + stack_create: yardstick + Debug: False + yardstick_test_dir: "samples" + yardstick_testcase: "moon_tenant" diff --git a/docs/testing/user/userguide/posca/posca_feature_vnf_scale_out.rst b/docs/testing/user/userguide/posca/posca_feature_vnf_scale_out.rst new file mode 100644 index 00000000..baf30578 --- /dev/null +++ b/docs/testing/user/userguide/posca/posca_feature_vnf_scale_out.rst @@ -0,0 +1,46 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Tech and others. + +*********************************** +POSCA feature Test of VNF Scale Out +*********************************** + +Test Case +========= + ++-----------------------------------------------------------------------------+ +|Bottlenecks POSCA Soak Test Throughputs | +| | ++--------------+--------------------------------------------------------------+ +|test case name| posca_feature_nfv_scale_out | +| | | ++--------------+--------------------------------------------------------------+ +|description | SampleVNF Scale Out Test | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | config file: | +| | /testsuite/posca/testcase_cfg/... | +| | posca_feature_nfv_scale_out.yaml | +| | | ++--------------+--------------------------------------------------------------+ +|test result | throughputs, latency, loss rate | +| | | ++--------------+--------------------------------------------------------------+ + +Configuration +============ +:: + + load_manager: + scenarios: + number_vnfs: 1, 2, 4 + iterations: 10 + interval: 35 + + runners: + stack_create: yardstick + flavor: + yardstick_test_dir: "samples/vnf_samples/nsut/acl" + yardstick_testcase: "tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_correlated_traffic_scale_out" diff --git a/docs/testing/user/userguide/posca_guide.rst b/docs/testing/user/userguide/posca_guide.rst index ec623d03..934415f0 100644 --- a/docs/testing/user/userguide/posca_guide.rst +++ b/docs/testing/user/userguide/posca_guide.rst @@ -100,7 +100,7 @@ Edit admin_rc.sh and add the following line export OS_CACERT=/tmp/os_cacert If you have deployed your openstack environment by compass, -you could use the following command to get the required files. For Fuel, Apex and JOID installer, we only provide limited support now +you could use the following command to get the required files. As to Fuel, Apex and JOID installers, we only provide limited support now for retrieving the configuration/description files. If you find that the following command can not do the magic, you should put the required files in /tmp manually. @@ -108,7 +108,7 @@ required files in /tmp manually. bash ./utils/env_prepare/config_prepare.sh -i <installer> [--debug] -Note that if we execute the command above, then admin_rc.sh and pod.yml gets created automatically in /tmp folder along with the line `export OS_CACERT=/tmp/os_cacert` added in admin_rc.sh file. +Note that if we execute the command above, then admin_rc.sh and pod.yml will be created automatically in /tmp folder along with the line `export OS_CACERT=/tmp/os_cacert` added in admin_rc.sh file. Executing Specified Testcase diff --git a/docs/testing/user/userguide/test_cases.rst b/docs/testing/user/userguide/test_cases.rst index 57e338e9..20003414 100644 --- a/docs/testing/user/userguide/test_cases.rst +++ b/docs/testing/user/userguide/test_cases.rst @@ -11,3 +11,11 @@ Bottlenecks - Test Cases ./posca/posca_factor_system_bandwidth.rst ./posca/posca_factor_ping.rst + ./posca/posca_factor_storperf.rst + ./posca/posca_factor_multistack_storage.rst + ./posca/posca_factor_multistack_storage_parallel.rst + ./posca/posca_factor_soak_throughputs.rst + ./posca/posca_feature_moon_resources.rst + ./posca/posca_feature_moon_tenants.rst + ./posca/posca_feature_vnf_scale_out.rst + ./kubestone/deployment_capacity.rst diff --git a/docs/testing/user/userguide/test_scheduler_user_guide.rst b/docs/testing/user/userguide/test_scheduler_user_guide.rst new file mode 100644 index 00000000..9820810e --- /dev/null +++ b/docs/testing/user/userguide/test_scheduler_user_guide.rst @@ -0,0 +1,180 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Huawei Technologies Co.,Ltd and others. + +**************************** +Test-scheduler User Guide +**************************** + + +Test-scheduler Introduction +============================== + +Test scheduler aims to schedule the testing process on NFV(Network +Function Virtualizaion) platform or MSA(Microservice Architecture) +application.By creating a testcase, you can implements a testing +process integrates and schedules the other testing frameworks and tools. +You can also create a testsuite which contains several testcases, and run +all the testcases at a time. + + +Preinstall Packages +=================== + +* Docker: https://docs.docker.com/engine/installation/ + * For Ubuntu, please refer to https://docs.docker.com/engine/installation/linux/ubuntu/ + +* Docker-Compose: https://docs.docker.com/compose/ + +.. code-block:: bash + + if [ -d usr/local/bin/docker-compose ]; then + rm -rf usr/local/bin/docker-compose + fi + curl -L https://github.com/docker/compose/releases/download/1.11.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose + chmod +x /usr/local/bin/docker-compose + + +Run Test-scheduler +===================== + +There are a few steps to do. + + +Download Bottlenecks Software +----------------------------- + +.. code-block:: bash + + mkdir -p /some/dir + git clone https://gerrit.opnfv.org/gerrit/bottlenecks + cd bottlenecks/test-scheduler + + +Build And Run Docker Containers +------------------------------- + +.. code-block:: bash + + cd docker + sh build.sh + sh run.sh + + +build.sh is used to build the images, and run.sh is use to +run the containers based on the images. +If you are not the root user, you need to use 'sudo', like: + +.. code-block:: bash + + cd docker + sudo sh build.sh + sudo sh run.sh + + +Otherwise there will be a problem of "Permission Denied". +The second command takes approximately 1h to finish( +so need some patience :) ), and the latter one just takes a few minutes. + +You can use command the check whether all the containers are in running. + +.. code-block:: bash + + docker ps + +if the output contains 6 containers as below, then the project +runs successfully: + +* t-scheduler-server +* t-scheduler-ui +* conductor_conductor-server_1 +* conductor_conductor-ui_1 +* conductor_dynomite_1 +* conductor_elasticsearch_1 + +Sometimes, the command execution fails, and you need to read the sections: +*(Optional)Commands Explanation* , +*(Optional) Build And Run Containers Seperately* to solve it. + + +Start To Use Via Web +-------------------- +You can visit the web pages via the url: http://your-host-ip:5311/. + +You can do these operations: + +* test suite CRUD +* test case CRUD +* execute a single test case +* execute several chosen test cases +* execute a single test suite +* test service CRUD +* context setting + +Cleaning Up Environment +----------------------- + +.. code-block:: bash + + cd docker + sh clean.sh + + +(Optional)Commands Explanation +------------------------------ + +The directory(**docker**) contains the shell scripts which are used +to build this project(**test-scheduler**) as a dockerized application. +Built by these scripts, the dockerized application will contain +6 containers(1 + 1 + 4). They can be divided as three components: + +* 1 server container: server component of **test-scheduler**. +* 1 webUI container: ui component of **test-scheduler**. +* a group of 4 containers of Conductor. + +Correspondingly, there are three subdirectories in the current +directory(**docker**): + +* server: contains scirpts of running server container. +* ui: contains scirpts of running ui container. +* plugin: contains scirpts of running Conductor containers. + +The three subdirectories contains scripts respectively.The scripts +(in one subdirectory) are used to build image and start container +for the single component. + +Essentially, the **build.sh** and **run.sh** (in the directory(**docker**)) +call the subdirectory scripts to build all three components. + + +(Optional) Build And Run Containers Seperately +---------------------------------------------- + +If you run the containers successfully, you can skip this section. +As said in *Build And Run Docker Containers*, build step will need about 1h to +finish.But it sometime will failed due to the network, and the rebuild will +take a great time cost.So we can build and run the containers seperately +according to the three subdirectories(**server**, **ui**, **plugin**).The +steps are similar to *Build And Run Docker Containers*. + +*IMPORTANT: There are relationships in these components(some need to be +created before other).So you can only build the components below in the +order:* **plugin** -> **server** -> **ui** . + +* enter the subdirectory(**$dir** stands for **server**, **ui**, **plugin**). + +.. code-block:: bash + + cd $dir + +* build the docker images. + +.. code-block:: bash + + sh build.sh + +* run the docker containers. + +.. code-block:: bash + + sh run.sh |