diff options
| -rw-r--r-- | docs/release/release-notes/release_notes.rst | 131 | ||||
| -rw-r--r-- | docs/release/release-notes/requirements_notes.rst | 13 | ||||
| -rw-r--r-- | docs/testing/user/userguide/index.rst | 2 | ||||
| -rw-r--r-- | docs/testing/user/userguide/overview.rst | 74 | ||||
| -rw-r--r-- | docs/testing/user/userguide/posca_guide.rst | 78 |
5 files changed, 176 insertions, 122 deletions
diff --git a/docs/release/release-notes/release_notes.rst b/docs/release/release-notes/release_notes.rst index fae8760a..d8d143ac 100644 --- a/docs/release/release-notes/release_notes.rst +++ b/docs/release/release-notes/release_notes.rst @@ -31,28 +31,31 @@ Apache License, Version 2.0. Version History =============== -+----------------+--------------------+---------------------------------+ -| *Date* | *Version* | *Comment* | -| | | | -+----------------+--------------------+---------------------------------+ -| Sept 22nd, 2016| 1.0 | Bottlenecks Colorado release 1.0| -| | | | -+----------------+--------------------+---------------------------------+ -| Feb 17nd, 2017 | 1.1 | Bottlenecks Danube release 1.0 | -| | | | -+----------------+--------------------+---------------------------------+ -| Mar 24nd, 2017 | 1.2 | Bottlenecks Danube release 1.0 | -| | | | -+----------------+--------------------+---------------------------------+ -| Mar 24nd, 2017 | 1.3 | Bottlenecks Danube release 1.0 | -| | | | -+----------------+--------------------+---------------------------------+ -| Apr 25th, 2017 | 1.4 | Bottlenecks Danube release 2.0 | -| | | | -+----------------+--------------------+---------------------------------+ -| Jun 19th, 2017 | 1.5 | Bottlenecks Danube release 3.0 | -| | | | -+----------------+--------------------+---------------------------------+ ++----------------+--------------------+-----------------------------------+ +| *Date* | *Version* | *Comment* | +| | | | ++----------------+--------------------+-----------------------------------+ +| Sept 22nd, 2016| 1.0 | Bottlenecks Colorado release 1.0 | +| | | | ++----------------+--------------------+-----------------------------------+ +| Feb 17nd, 2017 | 1.1 | Bottlenecks Danube release 1.0 | +| | | | ++----------------+--------------------+-----------------------------------+ +| Mar 24nd, 2017 | 1.2 | Bottlenecks Danube release 1.0 | +| | | | ++----------------+--------------------+-----------------------------------+ +| Mar 24nd, 2017 | 1.3 | Bottlenecks Danube release 1.0 | +| | | | ++----------------+--------------------+-----------------------------------+ +| Apr 25th, 2017 | 1.4 | Bottlenecks Danube release 2.0 | +| | | | ++----------------+--------------------+-----------------------------------+ +| Jun 19th, 2017 | 1.5 | Bottlenecks Danube release 3.0 | +| | | | ++----------------+--------------------+-----------------------------------+ +| Sept 15th, 2017| 1.6 | Bottlenecks Euphrates release 5.0 | +| | | | ++----------------+--------------------+-----------------------------------+ Summary ======= @@ -67,7 +70,7 @@ Summary * User Guide - * Development Guide + * Developer Guide The *Bottlenecks software* is developed in the OPNFV community, by the Bottlenecks_ team. @@ -75,6 +78,33 @@ Bottlenecks_ team. Release Data ============ +Euphrates Release Data +----------------------- + ++--------------------------------------+--------------------------------+ +| **Project** | Bottlenecks | +| | | ++--------------------------------------+--------------------------------+ +| **Repo/tag** | * Bottlenecks/5.0.0 | +| | | +| | | ++--------------------------------------+--------------------------------+ +| **Bottlenecks Docker image tag** | * 5.0.0 | +| | | +| | | ++--------------------------------------+--------------------------------+ +| **Release designation** | * 5.0.0 | +| | | +| | | ++--------------------------------------+--------------------------------+ +| **Release date** | * Sept 15th 2017 | +| | | +| | | ++--------------------------------------+--------------------------------+ +| **Purpose of the delivery** | Euphrates stable release | +| | | ++--------------------------------------+--------------------------------+ + Danube Release Data ----------------------- @@ -155,7 +185,7 @@ Danube Deliverables Software Deliverables --------------------- -**Bottlenecks source code <danube>** +**Bottlenecks source code** * https://gerrit.opnfv.org/gerrit/gitweb?p=bottlenecks.git;a=summary @@ -163,55 +193,52 @@ Software Deliverables Documentatiion Deliverables --------------------------- +**Bottlenecks documentation <euphrates>** + +* Release Notes: http://docs.opnfv.org/en/stable-euphrates/submodules/bottlenecks/docs/release/release-notes/release_notes.html +* User Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/bottlenecks/docs/testing/user/userguide/index.html +* Developer Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/bottlenecks/docs/testing/developer/devguide/index.html + **Bottlenecks documentation <danube>** * Release Notes: http://docs.opnfv.org/en/stable-danube/submodules/bottlenecks/docs/release/release-notes/release_notes.html * User Guide: http://docs.opnfv.org/en/stable-danube/submodules/bottlenecks/docs/testing/user/userguide/index.html -* Development Guide: http://docs.opnfv.org/en/stable-danube/submodules/bottlenecks/docs/testing/developer/devguide/index.html - -Colorado Deliverables -===================== - -Software Deliverables ---------------------- - -**Bottlenecks source code <colorado.1.0>** - -* https://gerrit.opnfv.org/gerrit/gitweb?p=bottlenecks.git;a=summary - - -Documentatiion Deliverables ---------------------------- +* Developer Guide: http://docs.opnfv.org/en/stable-danube/submodules/bottlenecks/docs/testing/developer/devguide/index.html **Bottlenecks documentation <colorado.1.0>** -* Platformation Overview: http://artifacts.opnfv.org/bottlenecks/colorado/1.0/docs/platformoverview/index.html +* Platform Overview: http://artifacts.opnfv.org/bottlenecks/colorado/1.0/docs/platformoverview/index.html * Configuration Guide: http://artifacts.opnfv.org/bottlenecks/colorado/1.0/configurationguide/index.html * Installation Guide: http://artifacts.opnfv.org/bottlenecks/colorado/docs/installationprocedure/index.html * http://artifacts.opnfv.org/bottlenecks/colorado/1.0/releasenotes/index.html * User Guide: http://artifacts.opnfv.org/bottlenecks/colorado/1.0/docs/userguide/index.html -Reason for Version -================== +Reason for Euphrates +==================== + +* Introduction of a simple monitoring module, i.e., Prometheus+Collectd+Node+Grafana to monitor the system behavior when executing stress tests. +* Support VNF scale up/out tests to verify NFVI capability to adapt the resource consuming. +* Extend Life-cycle test to data-plane to validate the system capability to handle concurrent networks usage. +* Testing framework is revised to support installer-agnostic testing. -* In Danube, POSCA testsuite is introduced and two stress tests are implemented. Rubbos and vstf are not supported in this release. Their documentations for former releases will only be kept this release. -* In Colorado, rubbos is refactored by using puppet, which makes it quite flexible to configure with different number of load generator(Client), worker(tomcat). -* In Colorado, vstf is refactored by extracting the test case's configuration information. +These enhancements and test cases help the end users to gain more comprehensive understanding of the SUT. +Graphic reports of the system behavior additional to test cases are provided to indicate the confidence level of SUT. +Installer-agnostic testing framework allow end user to do stress testing adaptively over either Open Source or commercial deployments. Known restrictions/issues ========================= -* In Danube, Bottlenecks use Yardstick to do stress tests by iteratively calling yardstick running test cases and in the meantime increasing load. +* Bottlenecks uses Yardstick to do stress tests by iteratively calling yardstick running test cases and in the meantime increasing load. * Sometimes, Yardstick just waits for the test environment preparing. This cause troubles since it seams to get Yardstick into a loop to keep waiting. For OPNFV CI, this loop will be automatically stopped because of the default timeout setting. However, for local testing, user should stop this loop manually. * Sometimes, Yardstick will return empty test results with test flag indicating test is excuted succefully. - It maybe because of the environment issue or poor internet connection causing testing tools are not installed successfully. + It maybe due to the environment issue or poor internet connection causing testing tools are not installed successfully. -* Sometimes, a process will go to dormancy. In this case, if a tool is installed in the SUT and its process go dormancy, we try to call it twice. Normally, it will response. +* Sometimes, a process will go to sleep state. In this case when a running tool go to sleep state, we try to call it twice. Normally, it will response. Test results @@ -251,10 +278,14 @@ Open JIRA tickets +------------------+----------------------------------------------------+ | JIRA | Description | +==================+====================================================+ -| BOTTLENECK-103 | Refactoring the Bottlenecks tesing workflow | -+------------------+----------------------------------------------------+ | BOTTLENECK-147 | Investigting why calling remote docker client | +------------------+----------------------------------------------------+ +| BOTTLENECK-207 | Yardstick loop when executing test cases | ++------------------+----------------------------------------------------+ +| BOTTLENECK-208 | Yardstick empty results when executing test cases | ++------------------+----------------------------------------------------+ +| BOTTLENECK-209 | Sleep process when executing test cases | ++------------------+----------------------------------------------------+ Useful links diff --git a/docs/release/release-notes/requirements_notes.rst b/docs/release/release-notes/requirements_notes.rst index 9e1c4719..eb5cd743 100644 --- a/docs/release/release-notes/requirements_notes.rst +++ b/docs/release/release-notes/requirements_notes.rst @@ -3,15 +3,18 @@ .. (c) Huawei Technologies Co.,Ltd and others. -******************************************************* -Bottlenecks Requirements Notes for OPNFV Danube Release -******************************************************* +********************************************************** +Bottlenecks Requirements Notes for OPNFV Euphrates Release +********************************************************** ====================== Installer Requirements ====================== -In Danube, test cases are runned defaultly over compass4nfv deployment. +Bottlenecks Euphrates release is installer-agnostic which means that +the test cases could be executed over different deployments. +However, the pod description file and openstack rc file +are required to automatically get the SUT informations. ===================== Hardware Requirements @@ -32,7 +35,7 @@ Ubuntu 14.04 is the default system. However, we do not do any linux core operati Cloud OS -------- -Bottlenecks support OpenStack Newton with compatibility for both openstack V2/V3 API. That is, Bottlenecks could be runned over lower version of OpenStack. +Bottlenecks supports OpenStack with compatibility for both OpenStack Keystone V2/V3 API. That is, Bottlenecks could be run over previous versions of OpenStack. Depending Package ----------------- diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst index 8e959abc..57396da8 100644 --- a/docs/testing/user/userguide/index.rst +++ b/docs/testing/user/userguide/index.rst @@ -3,7 +3,7 @@ .. (c) Huawei Technologies Co.,Ltd and others. *************************** -Bottlenecks - Testing Guide +Bottlenecks User Guide *************************** .. toctree:: diff --git a/docs/testing/user/userguide/overview.rst b/docs/testing/user/userguide/overview.rst index a9302297..5d76c76c 100644 --- a/docs/testing/user/userguide/overview.rst +++ b/docs/testing/user/userguide/overview.rst @@ -2,30 +2,45 @@ .. http://creativecommons.org/licenses/by/4.0 .. (c) Huawei Technologies Co.,Ltd and others. -********************* -Project Testing Guide -********************* +********** +User Guide +********** For each *testsuite*, you can either setup *teststory* or *testcase* to run certain test. *teststory* comprises several *testcases* as a set in one configuration file. You could call *teststory* or *testcase* by using -Bottlencks user interfaces. +Bottlenecks user interfaces. Details will be shown in the following section. Brief Introdcution of the Test suites in Project Releases ============================================================= -Brahmaputra: rubbos is introduced, which is an end2end NFVI perforamnce tool. -Virtual switch test framework(VSTF) is also introduced, -which is an test framework used for vswitch performance test. +Brahmaputra: -Colorado: rubbos is refactored by using puppet, which makes it quite flexible -to configure with different number of load generator (Client), worker (tomcat). -vstf is refactored by extracting the test case's configuration information. +* rubbos is introduced, which is an end2end NFVI perforamnce tool. +* Virtual switch test framework (VSTF) is also introduced, which is an test framework used for vswitch performance test. -Danube: posca testsuite is introduced to implementing stress (factor), scenario and -tuning test in parametric manner. Two testcases are developed and integrated into -community CI pipeline. Rubbos and VSTF are not supported any more. +Colorado: + +* rubbos is refactored by using puppet, and ease the integration with several load generators(Client) and worker(Tomcat). +* VSTF is refactored by extracting the test case's configuration information. + +Danube: + +* posca testsuite is introduced to implement stress (factor), feature and tuning test in parametric manner. +* Two testcases are developed and integrated into community CI pipeline. +* Rubbos and VSTF are not supported any more. + +Euphrates: + +* Introduction of a simple monitoring module, i.e., Prometheus+Collectd+Node+Grafana to monitor the system behavior when executing stress tests. +* Support VNF scale up/out tests to verify NFVI capability to adapt the resource consuming. +* Extend Life-cycle test to data-plane to validate the system capability to handle concurrent networks usage. +* Testing framework is revised to support installer-agnostic testing. + +These enhancements and test cases help the end users to gain more comprehensive understanding of the SUT. +Graphic reports of the system behavior additional to test cases are provided to indicate the confidence level of SUT. +Installer-agnostic testing framework allow end user to do stress testing adaptively over either Open Source or commercial deployments. Integration Description ======================= @@ -38,6 +53,8 @@ Integration Description +-------------+----------------------+----------------------+ | Danube | Compass | POSCA | +-------------+----------------------+----------------------+ +| Euphrates | Any | POSCA | ++-------------+----------------------+----------------------+ Test suite & Test case Description ================================== @@ -45,32 +62,13 @@ Test suite & Test case Description |POSCA | posca_factor_ping | | +-------------------------------+ | | posca_factor_system_bandwidth | -+--------+-------------------------------+ -|Rubbos | rubbos_basic | -| +-------------------------------+ -| | rubbos_TC1101 | -| +-------------------------------+ -| | rubbos_TC1201 | | +-------------------------------+ -| | rubbos_TC1301 | +| | posca_facotor_througputs | | +-------------------------------+ -| | rubbos_TC1401 | +| | posca_feature_scaleup | | +-------------------------------+ -| | rubbos_heavy_TC1101 | -+--------+-------------------------------+ -|vstf | vstf_Ti1 | -| +-------------------------------+ -| | vstf_Ti2 | -| +-------------------------------+ -| | vstf_Ti3 | -| +-------------------------------+ -| | vstf_Tn1 | -| +-------------------------------+ -| | vstf_Tn2 | -| +-------------------------------+ -| | vstf_Tu1 | -| +-------------------------------+ -| | vstf_Tu2 | -| +-------------------------------+ -| | vstf_Tu3 | +| | posca_feature_scaleout | +--------+-------------------------------+ + +As for the abandoned test suite in the previous Bottlenecks releases, please +refer to http://docs.opnfv.org/en/stable-danube/submodules/bottlenecks/docs/testing/user/userguide/deprecated.html. diff --git a/docs/testing/user/userguide/posca_guide.rst b/docs/testing/user/userguide/posca_guide.rst index ae6c2481..009aedb1 100644 --- a/docs/testing/user/userguide/posca_guide.rst +++ b/docs/testing/user/userguide/posca_guide.rst @@ -9,12 +9,12 @@ POSCA Testsuite Guide POSCA Introduction ==================== -The POSCA (Parametric Bottlenecks Testing Catalogue) testsuite +The POSCA (Parametric Bottlenecks Testing Catalogue) test suite classifies the bottlenecks test cases and results into 5 categories. Then the results will be analyzed and bottlenecks will be searched among these categories. -The POSCA testsuite aims to locate the bottlenecks in parmetric +The POSCA testsuite aims to locate the bottlenecks in parametric manner and to decouple the bottlenecks regarding the deployment requirements. The POSCA testsuite provides an user friendly way to profile and @@ -59,10 +59,13 @@ Preinstall Packages Run POSCA Locally ================= -POSCA testsuite is highly automated regarding test environment preparation, installing testing tools, excuting tests and showing the report/analysis. +The test environment preparation, the installation of the testing tools, +the execution of the tests and the reporting/analyisis of POSCA test suite +are highly automated. A few steps are needed to run it locally. -It is presumed that a user is using Compass4nfv to deploy OPNFV Danube and the user logins jumper server as root. +In Euphrates, Bottlenecks has modified its framework to support installer-agnostic +testing which means that test cases could be executed over different deployments. Downloading Bottlenecks Software @@ -84,35 +87,59 @@ Preparing Python Virtual Evnironment . pre_virt_env.sh -Excuting Specified Testcase +Preparing configuration/description files +----------------------------------------- + +Put OpenStack RC file (admin_rc.sh), os_carcert and pod.yaml (pod descrition file) in /tmp directory. +Edit admin_rc.sh and add the following line + +.. code-block:: bash + + export OS_CACERT=/tmp/os_cacert + +If you are using compass, fuel, apex or joid to deploy your openstack +environment, you could use the following command to get the required files. + +.. code-block:: bash + + bash /utils/env_prepare/config_prepare.sh -i <installer> [--debug] + +Note that if we execute the command above, then admin_rc.sh and pod.yml gets created automatically in /tmp folder along with the line `export OS_CACERT=/tmp/os_cacert` added in admin_rc.sh file. + + +Executing Specified Testcase --------------------------- -Bottlencks provide a CLI interface to run the tests, which is one of the most convenient way since it is more close to our natural languge. An GUI interface with rest API will also be provided in later update. +1. Bottlenecks provides a CLI interface to run the tests, which is one of the most convenient way since it is more close to our natural languge. An GUI interface with rest API will also be provided in later update. .. code-block:: bash - bottlenecks [testcase run <testcase>] [teststory run <teststory>] + bottlenecks testcase|teststory run <testname> -For the *testcase* command, testcase name should be as the same name of the test case configuration file located in testsuites/posca/testcase_cfg. -For stress tests in Danube, *testcase* should be replaced by either *posca_factor_ping* or *posca_factor_system_bandwidth*. -For the *teststory* command, a user could specified the test cases to be excuted by defined it in a teststory configuration file located in testsuites/posca/testsuite_story. There is also an example there named *posca_factor_test*. + For the *testcase* command, testname should be as the same name of the test case configuration file located in testsuites/posca/testcase_cfg. + For stress tests in Danube/Euphrates, *testcase* should be replaced by either *posca_factor_ping* or *posca_factor_system_bandwidth*. + For the *teststory* command, a user can specify the test cases to be executed by defining it in a teststory configuration file located in testsuites/posca/testsuite_story. There is also an example there named *posca_factor_test*. -There are also other 2 ways to run test cases and test stories. -The first one is using shell script. +2. There are also other 2 ways to run test cases and test stories. + + The first one is to use shell script. .. code-block:: bash - bash run_tests.sh [-h|--help] [-s <testsuite>] [-c <testcase>] + bash run_tests.sh [-h|--help] -s <testsuite>|-c <testcase> + -The second is using python interpreter. + The second is to use python interpreter. .. code-block:: bash - docker-compose -f docker/bottleneck-compose/docker-compose.yml up -d - docker pull tutum/influxdb:0.13 + $REPORT=False + opts="--privileged=true -id" + docker_volume="-v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/tmp" + docker run $opts --name bottlenecks-load-master $docker_volume opnfv/bottlenecks:latest /bin/bash sleep 5 POSCA_SCRIPT="/home/opnfv/bottlenecks/testsuites/posca" - docker exec bottleneckcompose_bottlenecks_1 python ${POSCA_SCRIPT}/run_posca.py [testcase <testcase>] [teststory <teststory>] + docker exec bottlenecks-load-master python ${POSCA_SCRIPT}/../run_posca.py testcase|teststory <testname> ${REPORT} Showing Report @@ -135,13 +162,6 @@ If you want to clean the dockers that established during the test, you can excut .. code-block:: bash - docker-compose -f docker/bottleneck-compose/docker-compose.yml down -d - docker ps -a | grep 'influxdb' | awk '{print $1}' | xargs docker rm -f >/dev/stdout - -Or you can just run the following command - -.. code-block:: bash - bash run_tests.sh --cleanup Note that you can also add cleanup parameter when you run a test case. Then environment will be automatically cleaned up when @@ -157,18 +177,20 @@ reported automatically to community MongoDB. There are two ways to report the re .. code-block:: bash - bash run_tests.sh [-h|--help] [-s <testsuite>] [-c <testcase>] --report + bash run_tests.sh [-h|--help] -s <testsuite>|-c <testcase> --report 2. Report testing result by python interpreter .. code-block:: bash - docker-compose -f docker/bottleneck-compose/docker-compose.yml up -d - docker pull tutum/influxdb:0.13 + REPORT=True + opts="--privileged=true -id" + docker_volume="-v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/tmp" + docker run $opts --name bottlenecks-load-master $docker_volume opnfv/bottlenecks:latest /bin/bash sleep 5 REPORT="True" POSCA_SCRIPT="/home/opnfv/bottlenecks/testsuites/posca" - docker exec bottleneckcompose_bottlenecks_1 python ${POSCA_SCRIPT}/run_posca.py [testcase <testcase>] [teststory <teststory>] REPORT + docker exec bottlenecks_load-master python ${POSCA_SCRIPT}/../run_posca.py testcase|teststory <testcase> ${REPORT} Test Result Description ======================= |