diff options
Diffstat (limited to 'docs/testing/user/testspecification/stress')
| -rw-r--r-- | docs/testing/user/testspecification/stress/index.rst | 159 |
1 files changed, 159 insertions, 0 deletions
diff --git a/docs/testing/user/testspecification/stress/index.rst b/docs/testing/user/testspecification/stress/index.rst new file mode 100644 index 00000000..41c84e7c --- /dev/null +++ b/docs/testing/user/testspecification/stress/index.rst @@ -0,0 +1,159 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV + +========================= +Stress Test Specification +========================= + +.. toctree:: + :maxdepth: 2 + +Scope +===== + +The stress test involves testing and verifying the ability of the SUT to withstand +stress and other challenging factors. Main purpose behind the testing is to make sure +the SUT is able to absorb failures while providing an acceptable level of service. + + +References +================ + +This test area references the following specifications, definitions and reviews: + +- Upstream OpenStack NOVA Resiliency + + - https://wiki.openstack.org/wiki/NovaResiliency + +- Stress Testing over OPNFV Platform + + - https://wiki.opnfv.org/display/bottlenecks/Stress+Testing+over+OPNFV+Platform + + +Definitions and Abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test area + +- iff - if and only if +- NFVI - Network Functions Virtualization Infrastructure +- NaN - Not a Number +- Service Level - Measurable terms that describe the quality of the service provided by the SUT within a given time period +- SUT - System Under Test +- VM - Virtual Machine + + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVI and VIM in operation on a +Pharos compliant infrastructure. + + +Test Area Structure +==================== + +According to the testing goals stated in the test scope section, +preceding test will not affect the subsequent test +as long as the SUT is able to sustain the given stress +while providing an acceptable level of service. +Any FAIL result from a single test case will cause the SUT failing the whole test. + + +Test Descriptions +================= + +---------------------------------------------------------------- +Test Case 1 - Concurrent capacity based on life-cycle ping test +---------------------------------------------------------------- + +Short name +---------- + +dovetail.stress.ping + +Use case specification +---------------------- + +This test case verifies the ability of the SUT concurrently setting up VM pairs +for different tenants (through different OpenStack related components) and +providing acceptable capacity under stressful conditions. The connectivity between +VMs in a VM pair for a tenant is validated through Ping test. A life-cycle event +in this test case is particularly referred to a VM pair life-cycle consisting of +spawning, pinging and destroying. + +Test preconditions +------------------ + +* heat_template_version: 2013-05-23 +* ElasticSearch Port: 9200 +* LogStash Port: 5044 +* Kibana Port: 5601 +* Yardstick Port: 5000 + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for validating capacity of the SUT +''''''''''''''''''''''''''''''''''''''''''''' +Validating capacity of the SUT based on life-cycle ping test generally involves +2 subtests which provides secondary validation for the SUT furnishing users with +reliable capacity without being crushed. + +Let *N1*, *N2*, *N3* and *P1* be certain preset numbers, respectively. +In subtest 1, the SUT concurrently setting up *N1* VM pairs with each VM pair +belonging to a different tenant. Then VM1 in a VM pair pings VM2 for *P1* times +with *P1* packets. The connectivity could be validated iff VM1 successfully pings +VM2 with these *P1* packets. +Subtest 1 is finished iff all the concurrent (*N1*) requests for creating VM pairs +are fulfilled with returned values that indicate the statuses of the VM pairs creations. + +Subtest 2 is executed after subtest 1 as secondary validation of the capacity. +It follows the same workflow as subtest 1 does to set up *N2* VM pairs. + +Assume *S1* and *S2* be the numbers of VM pairs that are successfully created in +subtest 1 and subtest 2, respectively. If *min(S1,S2)>=N3*, then the SUT is considered as PASS. +Otherwise, we denote the SUT with FAIL. + +Note that for subtest 1, if the number of successfully created VM pairs, i.e., *S1*, +is smaller than *N3*. Subtest 2 will not be executed and SUT will be marked with FAIL. + +Test execution +'''''''''''''' +* Test action 1: Install the testing tools by pulling and running the Bottlenecks Docker container +* Test action 2: Prepare the test by sourcing openstack credential file, + eliminating the environment constraints, i.e., Quota setting, setting up + Yardstick docker, pulling and registering OS images and VM flavor +* Test action 3: Call Yardstick to concurrently creating *N1* VM pairs for N1 tenants +* Test action 4: In each VM pair, VM1 pings VM2 for *P1* times with *P1* packets while recording the successful numbers +* Test action 5: Mark the VM pairs with *P1* successful pings as PASS and record the total number of PASS VM pairs as *S1* +* Test action 6: Destroy all the VM pairs +* Test action 7: If *S1<N3*, the SUT is marked with FAIL and the test return. Otherwise go to *Test action 8* +* Test action 8: Go to *Test action 3* and do the test again to create *N2* VM pairs with PASS VM pairs counted as *S2* +* Test action 9: If *S2<N3*, the SUT is marked with FAIL. Otherwise marked with PASS. + +Pass / fail criteria +'''''''''''''''''''' + +Typical setting of *(N1, N2, N3, P1)* is *(5, 5, 5, 10)*. +The reference setting above is acquired based on the results from OPNFV CI jobs +and testing over commercial products. + +The connectivity within a VM pair is validated iff: + +* VM1 successfully pings VM2 for *P1* times with *P1* packets + +The SUT is considered passing the test iff: + +* *min(S1,S2)>=N3* + +Note that after each subtest, the program will check if the successfully created number of VM pairs +is smaller than *N3*. If true, the program will return and the SUT will be marked with FAIL. +Then the passing criteria is equal to the equation above. When subtest 1 returns, *S2* here is denoted +by NaN. + +Post conditions +--------------- + +N/A |