diff options
Diffstat (limited to 'docs/testing/user/userguide')
| -rw-r--r-- | docs/testing/user/userguide/index.rst | 26 | ||||
| -rw-r--r-- | docs/testing/user/userguide/reporting.rst | 2 | ||||
| -rw-r--r-- | docs/testing/user/userguide/runfunctest.rst | 466 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_details.rst | 71 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_overview.rst | 49 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_results.rst | 5 | ||||
| -rw-r--r-- | docs/testing/user/userguide/troubleshooting.rst | 30 |
7 files changed, 225 insertions, 424 deletions
diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst index 4b66eacd2..c0c763651 100644 --- a/docs/testing/user/userguide/index.rst +++ b/docs/testing/user/userguide/index.rst @@ -3,9 +3,9 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 -========================= -OPNFV FUNCTEST user guide -========================= +=================== +Functest User Guide +=================== .. toctree:: :maxdepth: 2 @@ -29,12 +29,10 @@ Introduction ============ The goal of this document is to describe the OPNFV Functest test cases and to -provide a procedure to execute them. In the OPNFV Danube system release, -a Functest CLI utility is introduced for an easier execution of test procedures. +provide a procedure to execute them. -**IMPORTANT**: It is assumed here that the Functest Docker container is already -properly deployed and that all instructions described in this guide are to be -performed from *inside* the deployed Functest Docker container. +**IMPORTANT**: It is assumed here that Functest has been properly deployed +following the installation guide procedure `[1]`_. .. include:: ./test_overview.rst @@ -86,13 +84,17 @@ References `[15]`_: Testing OpenStack Tempest part 1 +`[16]`_: Running Functest through internal REST API + +`[17]`_: OPNFV Test API + `OPNFV main site`_: OPNFV official web site `Functest page`_: Functest wiki page IRC support chan: #opnfv-functest -.. _`[1]`: http://docs.opnfv.org/en/stable-danube/submodules/functest/docs/testing/user/configguide/index.html +.. _`[1]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html .. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html .. _`[3]`: https://rally.readthedocs.org/en/latest/index.html .. _`[4]`: http://events.linuxfoundation.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf @@ -103,13 +105,15 @@ IRC support chan: #opnfv-functest .. _`[9]`: https://github.com/openstack/defcore .. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst .. _`[11]`: http://robotframework.org/ -.. _`[12]`: http://docs.opnfv.org/en/stable-danube/submodules/functest/docs/testing/user/userguide/index.html +.. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html .. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter .. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/ +.. _`[16]`: https://wiki.opnfv.org/display/functest/Running+test+cases+via+new+Functest+REST+API +.. _`[17]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html .. _`OPNFV main site`: http://www.opnfv.org .. _`Functest page`: https://wiki.opnfv.org/functest .. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html .. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html .. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/functest/ci/config_functest.yaml -.. _`Functest reporting`: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-fuel.html +.. _`Functest reporting`: http://testresults.opnfv.org/reporting/master/functest/status-apex.html diff --git a/docs/testing/user/userguide/reporting.rst b/docs/testing/user/userguide/reporting.rst index 14d22f230..88f5e865a 100644 --- a/docs/testing/user/userguide/reporting.rst +++ b/docs/testing/user/userguide/reporting.rst @@ -87,4 +87,4 @@ See `reporting page`_ for details. For the status, click on the version, Functest then the Status menu. -_`reporting page`: http://testresults.opnfv.org/reporting/ +.. _`reporting page`: http://testresults.opnfv.org/reporting/ diff --git a/docs/testing/user/userguide/runfunctest.rst b/docs/testing/user/userguide/runfunctest.rst index 83e603b3a..db85c18ff 100644 --- a/docs/testing/user/userguide/runfunctest.rst +++ b/docs/testing/user/userguide/runfunctest.rst @@ -1,342 +1,9 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -Executing the functest suites (Ubuntu) -====================================== - -Manual testing --------------- - -This section assumes the following: - * The Functest Docker container is running - * The docker prompt is shown - * The Functest environment is ready (Functest CLI command 'functest env prepare' - has been executed) - -If any of the above steps are missing please refer to the Functest Config Guide -as they are a prerequisite and all the commands explained in this section **must** be -performed **inside the container**. - -The Functest CLI offers two commands (functest tier ...) and (functest testcase ... ) -for the execution of Test Tiers or Test Cases:: - - root@22e436918db0:~/repos/functest/ci# functest tier --help - Usage: functest tier [OPTIONS] COMMAND [ARGS]... - - Options: - -h, --help Show this message and exit. - - Commands: - get-tests Prints the tests in a tier. - list Lists the available tiers. - run Executes all the tests within a tier. - show Shows information about a tier. - root@22e436918db0:~/repos/functest/ci# functest testcase --help - - Usage: functest testcase [OPTIONS] COMMAND [ARGS]... - - Options: - -h, --help Show this message and exit. - - Commands: - list Lists the available testcases. - run Executes a test case. - show Shows information about a test case. - -More details on the existing Tiers and Test Cases can be seen with the 'list' -command:: - - root@22e436918db0:~/repos/functest/ci# functest tier list - - 0. healthcheck: - ['connection_check', 'api_check', 'snaps_health_check',] - - 1. smoke: - ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'odl', 'rally_sanity', 'refstack_defcore', 'snaps_smoke'] - - 2. features: - ['doctor', 'domino', 'promise', security_scan'] - - 3. components: - ['tempest_full_parallel', 'rally_full'] - - 4. vnf: - ['cloudify_ims', 'orchestra_ims', 'vyos_vrouter'] - - and - - root@22e436918db0:~/repos/functest/ci# functest testcase list - api_check - connection_check - snaps_health_check - vping_ssh - vping_userdata - snaps_smoke - refstack_defcore - tempest_smoke_serial - rally_sanity - odl - tempest_full_parallel - rally_full - vyos_vrouter -Note the list of test cases depend on the installer and the scenario. - -More specific details on specific Tiers or Test Cases can be seen wih the -'show' command:: - - root@22e436918db0:~/repos/functest/ci# functest tier show smoke - +======================================================================+ - | Tier: smoke | - +======================================================================+ - | Order: 1 | - | CI Loop: (daily)|(weekly) | - | Description: | - | Set of basic Functional tests to validate the OpenStack | - | deployment. | - | Test cases: | - | - vping_ssh | - | - vping_userdata | - | - tempest_smoke_serial | - | - rally_sanity | - | | - +----------------------------------------------------------------------+ - - and - - root@22e436918db0:~/repos/functest/ci# functest testcase show tempest_smoke_serial - +======================================================================+ - | Testcase: tempest_smoke_serial | - +======================================================================+ - | Description: | - | This test case runs the smoke subset of the OpenStack Tempest | - | suite. The list of test cases is generated by Tempest | - | automatically and depends on the parameters of the OpenStack | - | deplopyment. | - | Dependencies: | - | - Installer: | - | - Scenario : | - | | - +----------------------------------------------------------------------+ - - -To execute a Test Tier or Test Case, the 'run' command is used:: - - root@22e436918db0:~/repos/functest/ci# functest tier run healthcheck - - 2017-08-16 12:35:51,799 - functest.ci.run_tests - INFO - ############################################ - 2017-08-16 12:35:51,799 - functest.ci.run_tests - INFO - Running tier 'healthcheck' - 2017-08-16 12:35:51,800 - functest.ci.run_tests - INFO - ############################################ - 2017-08-16 12:35:51,800 - functest.ci.run_tests - INFO - - 2017-08-16 12:35:51,800 - functest.ci.run_tests - INFO - ============================================ - 2017-08-16 12:35:51,800 - functest.ci.run_tests - INFO - Running test case 'connection_check'... - 2017-08-16 12:35:51,800 - functest.ci.run_tests - INFO - ============================================ - 2017-08-16 12:36:00,278 - functest.core.testcase - INFO - The results were successfully pushed to DB - 2017-08-16 12:36:00,279 - functest.ci.run_tests - INFO - Test result: - +--------------------------+------------------+------------------+----------------+ - | TEST CASE | PROJECT | DURATION | RESULT | - +--------------------------+------------------+------------------+----------------+ - | connection_check | functest | 00:06 | PASS | - +--------------------------+------------------+------------------+----------------+ - 2017-08-16 12:36:00,281 - functest.ci.run_tests - INFO - - 2017-08-16 12:36:00,281 - functest.ci.run_tests - INFO - ============================================ - 2017-08-16 12:36:00,281 - functest.ci.run_tests - INFO - Running test case 'api_check'... - 2017-08-16 12:36:00,281 - functest.ci.run_tests - INFO - ============================================ - 2017-08-16 12:41:04,088 - functest.core.testcase - INFO - The results were successfully pushed to DB - 2017-08-16 12:41:04,088 - functest.ci.run_tests - INFO - Test result: - +-------------------+------------------+------------------+----------------+ - | TEST CASE | PROJECT | DURATION | RESULT | - +-------------------+------------------+------------------+----------------+ - | api_check | functest | 05:03 | PASS | - +-------------------+------------------+------------------+----------------+ - 2017-08-16 12:41:04,092 - functest.ci.run_tests - INFO - - 2017-08-16 12:41:04,092 - functest.ci.run_tests - INFO - ============================================ - 2017-08-16 12:41:04,092 - functest.ci.run_tests - INFO - Running test case 'snaps_health_check'... - 2017-08-16 12:41:04,092 - functest.ci.run_tests - INFO - ============================================ - 2017-08-16 12:41:39,817 - functest.core.testcase - INFO - The results were successfully pushed to DB - 2017-08-16 12:41:39,818 - functest.ci.run_tests - INFO - Test result: - +----------------------------+------------------+------------------+----------------+ - | TEST CASE | PROJECT | DURATION | RESULT | - +----------------------------+------------------+------------------+----------------+ - | snaps_health_check | functest | 00:35 | PASS | - +----------------------------+------------------+------------------+----------------+ - - and - - root@22e436918db0:~/repos/functest/ci# functest testcase run vping_ssh - 2017-08-16 12:41:39,821 - functest.ci.run_tests - INFO - ============================================ - 2017-08-16 12:41:39,821 - functest.ci.run_tests - INFO - Running test case 'vping_ssh'... - 2017-08-16 12:41:39,821 - functest.ci.run_tests - INFO - ============================================ - 2017-08-16 12:42:49,861 - functest.core.testcase - INFO - The results were successfully pushed to DB - 2017-08-16 12:42:49,861 - functest.ci.run_tests - INFO - Test result: - +-------------------+------------------+------------------+----------------+ - | TEST CASE | PROJECT | DURATION | RESULT | - +-------------------+------------------+------------------+----------------+ - | vping_ssh | functest | 00:47 | PASS | - +-------------------+------------------+------------------+----------------+ - : - : - etc. - -To list the test cases which are part of a specific Test Tier, the 'get-tests' -command is used with 'functest tier':: - - root@22e436918db0:~/repos/functest/ci# functest tier get-tests healthcheck - Test cases in tier 'healthcheck': - ['connection_check', 'api_check', 'snaps_health_check'] - - -Please note that for some scenarios some test cases might not be launched. -For example, the last example displayed only the 'odl' testcase for the given -environment. In this particular system the deployment does not support the 'ocl' SDN -Controller Test Case; for example. - -**Important** If you use the command 'functest tier run <tier_name>', then the -Functest CLI utility will call **all valid Test Cases**, which belong to the -specified Test Tier, as relevant to scenarios deployed to the SUT environment. -Thus, the Functest CLI utility calculates automatically which tests can be -executed and which cannot, given the environment variable **DEPLOY_SCENARIO**, -which is passed in to the Functest docker container. - -Currently, the Functest CLI command 'functest testcase run <testcase_name>', supports -two possibilities:: - - * Run a single Test Case, specified by a valid choice of <testcase_name> - * Run ALL test Test Cases (for all Tiers) by specifying <testcase_name> = 'all' - -Functest includes a cleaning mechanism in order to remove all the OpenStack -resources except those present before running any test. The script -*$REPOS_DIR/functest/functest/utils/openstack_snapshot.py* is called once when setting up -the Functest environment (i.e. CLI command 'functest env prepare') to snapshot -all the OpenStack resources (images, networks, volumes, security groups, tenants, -users) so that an eventual cleanup does not remove any of these defaults. - -The script **openstack_clean.py** which is located in -*$REPOS_DIR/functest/functest/utils/* is in charge of cleaning the OpenStack resources -that are not specified in the defaults file generated previously which is stored in -*/home/opnfv/functest/conf/openstack_snapshot.yaml* in the Functest docker container. - -It is important to mention that if there are new OpenStack resources created -manually after the snapshot done before running the tests, they will be removed, -unless you use the special method of invoking the test case with specific -suppression of clean up. (See the `Troubleshooting`_ section). - -The reason to include this cleanup meachanism in Functest is because some -test suites create a lot of resources (users, tenants, networks, volumes etc.) -that are not always properly cleaned, so this function has been set to keep the -system as clean as it was before a full Functest execution. - -Although the Functest CLI provides an easy way to run any test, it is possible to -do a direct call to the desired test script. For example: - - python $REPOS_DIR/functest/functest/opnfv_tests/openstack/vping/vping_ssh.py - - -Automated testing ------------------ - -As mentioned previously, the Functest Docker container preparation as well as -invocation of Test Cases can be called within the container from the Jenkins CI -system. There are 3 jobs that automate the whole process. The first job runs all -the tests referenced in the daily loop (i.e. that must been run daily), the second -job runs the tests referenced in the weekly loop (usually long duration tests run -once a week maximum) and the third job allows testing test suite by test suite specifying -the test suite name. The user may also use either of these Jenkins jobs to execute -the desired test suites. - -One of the most challenging task in the Danube release consists -in dealing with lots of scenarios and installers. Thus, when the tests are -automatically started from CI, a basic algorithm has been created in order to -detect whether a given test is runnable or not on the given scenario. -Some Functest test suites cannot be systematically run (e.g. ODL suite can not -be run on an ONOS scenario). The daily/weekly notion has been introduces in -Colorado in order to save CI time and avoid running systematically -long duration tests. It was not used in Colorado due to CI resource shortage. -The mechanism remains however as part of the CI evolution. - -CI provides some useful information passed to the container as environment -variables: - - * Installer (apex|compass|fuel|joid), stored in INSTALLER_TYPE - * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP - * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with - - * controller = (odl|ocl|nosdn) - * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|ovs_dpdk_bar) - * mode = (ha|noha) - -The constraints per test case are defined in the Functest configuration file -*/usr/local/lib/python2.7/dist-packages/functest/ci/testcases.yaml*:: - - tiers: - - - name: smoke - order: 1 - ci_loop: '(daily)|(weekly)' - description : >- - Set of basic Functional tests to validate the OpenStack deployment. - testcases: - - - name: vping_ssh - criteria: 'status == "PASS"' - blocking: true - description: >- - This test case verifies: 1) SSH to an instance using floating - IPs over the public network. 2) Connectivity between 2 instances - over a private network. - dependencies: - installer: '' - scenario: '^((?!bgpvpn|odl_l3).)*$' - run: - module: 'functest.opnfv_tests.openstack.vping.vping_ssh' - class: 'VPingSSH' - .... - -We may distinguish 2 levels in the test case description: - * Tier level - * Test case level - -At the tier level, we define the following parameters: - - * ci_loop: indicate if in automated mode, the test case must be run in dail and/or weekly jobs - * description: a high level view of the test case - -For a given test case we defined: - * the name of the test case - * the criteria (experimental): a criteria used to declare the test case as PASS or FAIL - * blocking: if set to true, if the test is failed, the execution of the following tests is canceled - * the description of the test case - * the dependencies: a combination of 2 regex on the scenario and the installer name - * run: In Danube we introduced the notion of abstract class in order to harmonize the way to run internal, feature or vnf tests - -For further details on abstraction classes, see developper guide. - -Additional parameters have been added in the desription in the Database. -The target is to use the configuration stored in the Database and consider the -local file as backup if the Database is not reachable. -The additional fields related to a test case are: - * trust: we introduced this notion to put in place a mechanism of scenario promotion. - * Version: it indicates since which version you can run this test - * domains: the main domain covered by the test suite - * tags: a list of tags related to the test suite - -The order of execution is the one defined in the file if all test cases are selected. - -In CI daily job the tests are executed in the following order: - - 1) healthcheck (blocking) - 2) smoke: both vPings are blocking - 3) Feature project tests cases - -In CI weekly job we add 2 tiers: - - 4) VNFs (vIMS) - 5) Components (Rally and Tempest long duration suites) - -As explained before, at the end of an automated execution, the OpenStack resources -might be eventually removed. -Please note that a system snapshot is taken before any test case execution. - -This testcase.yaml file is used for CI, for the CLI and for the automatic reporting. - - -Executing Functest suites (Alpine) -================================== +Executing Functest suites +========================= As mentioned in the configuration guide `[1]`_, Alpine docker containers have been introduced in Euphrates. @@ -362,9 +29,117 @@ You should get:: You can run functest-healcheck, functest-smoke, functest-features, functest-components and functest-vnf. -Please note that you may also use the CLI for manual tests using Alpine -containers. +The result tables show the results by test case, it can be:: + + * PASS + * FAIL + * SKIP: if the scenario/installer does not support the test case + + +Manual run +========== +If you want to run the test step by step, you may add docker option then run the +different commands within the docker. + +Considering the healthcheck example, running functest manaully means:: + + sudo docker run -ti --env-file env \ + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ + -v $(pwd)/images:/home/opnfv/functest/images \ + opnfv/functest-healthcheck /bin/bash + +The docker prompt shall be returned. Then within the docker run the following +commands:: + + $ source /home/opnfv/functest/conf/openstack.creds + +Prepare environment +------------------- +Prior to commencing the Functest environment preparation, we can check +the initial status of the environment. Issue the **functest env status** +command at the prompt:: + + # functest env status + # Functest environment is not installed. + +To prepare the Functest docker container for test case execution, type:: + + # functest env prepare + # ... + # Functest environment ready to run tests. + +you may also type prepare_env instead of functest env prepare. + +To list some basic information about an already prepared Functest +docker container environment, issue the **functest env show** at the +prompt:: + bash-4.3# functest env show + +------------------------------+---------------------------------+ + | FUNCTEST ENVIRONMENT | VALUE | + +------------------------------+---------------------------------+ + | STATUS | ready | + | SCENARIO | os-nosdn-nofeature-noha | + | DEBUG FLAG | false | + | BUILD_TAG | None | + | INSTALLER | compass | + | POD | huawei-pod1 | + +------------------------------+---------------------------------+ + +See configuration guide for details on Functest environnement variables. + +Tier +---- +Each Alpine container provided on the docker hub matches with a tier. +The following commands are available:: + + # functest tier list + - 0. healthcheck: + ['connection_check', 'api_check', 'snaps_health_check'] + # functest tier show healthcheck + +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ + | TIERS | ORDER | CI LOOP | DESCRIPTION | TESTCASES | + +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ + | healthcheck | 0 | (daily)|(weekly) | First tier to be executed to verify the | connection_check api_check | + | | | | basic operations in the VIM. | snaps_health_check | + +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ + +To run all the cases of the tier, type:: + + # functest tier run healthcheck + +Testcase +-------- +Testcases can be listed, shown and run though the CLI:: + + # functest testcase list + connection_check + api_check + snaps_health_check + # functest testcase show api_check + +-------------------+--------------------------------------------------+------------------+---------------------------+ + | TEST CASE | DESCRIPTION | CRITERIA | DEPENDENCY | + +-------------------+--------------------------------------------------+------------------+---------------------------+ + | api_check | This test case verifies the retrieval of | 100 | ^((?!netvirt).)*$ | + | | OpenStack clients: Keystone, Glance, | | | + | | Neutron and Nova and may perform some | | | + | | simple queries. When the config value of | | | + | | snaps.use_keystone is True, functest | | | + | | must have access to the cloud's private | | | + | | network. | | | + +-------------------+--------------------------------------------------+------------------+---------------------------+ + # functest testcase run connection_check + ... + # functest run all + +You can also type run_tests -t all to run all the tests. + +Note the list of test cases depend on the installer and the scenario. + +Reporting results to the test Database +====================================== +In OPNFV CI we collect all the results from CI. A test APi shall be available +as well as a test database `[17]`_. Functest internal API ===================== @@ -377,18 +152,23 @@ other test projects. In Euphrates the main method of the APIs are: - * Show environment - * Prepare Environment * Show credentials + * Update openrc file + * Show environment + * Update hosts info for domain name + * Prepare environment * List all testcases * Show a testcase + * Run a testcase * List all tiers * Show a tier * List all testcases within given tier + * Get the result of the specified task + * Get the log of the specified task -The API can be invoked as follows: - http://<functest_url>:5000/api/v1/functest/envs +See `[16]`_ to get examples on how to use the API. -TODO -.. _`[1]`: http://artifacts.opnfv.org/functest/colorado/docs/configguide/# +.. _`[1]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html +.. _`[16]`: https://wiki.opnfv.org/display/functest/Running+test+cases+via+new+Functest+REST+API +.. _`[17]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html diff --git a/docs/testing/user/userguide/test_details.rst b/docs/testing/user/userguide/test_details.rst index 978614895..c9ef63d38 100644 --- a/docs/testing/user/userguide/test_details.rst +++ b/docs/testing/user/userguide/test_details.rst @@ -9,7 +9,7 @@ VIM (Virtualized Infrastructure Manager) Healthcheck tests ^^^^^^^^^^^^^^^^^ -In Danube, healthcheck tests have been refactored and rely on SNAPS, an +Since Danube, healthcheck tests have been refactored and rely on SNAPS, an OPNFV middleware project. SNAPS stands for "SDN/NFV Application development Platform and Stack". @@ -195,6 +195,7 @@ The OPNFV Rally scenarios are based on the collection of the actual Rally scenar * neutron * nova * quotas + * ceilometer A basic SLA (stop test on errors) has been implemented. @@ -207,35 +208,20 @@ NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test cases. Test case 'rally_full' executes the full defined set of Rally tests. -Refstack-client to run Defcore testcases ------------------------------------------ +Refstack-client to run OpenStack interop testcases +-------------------------------------------------- Refstack-client `[8]`_ is a command line utility that allows you to execute Tempest test runs based on configurations you specify. -It is the official tool to run Defcore `[9]`_ testcases, -which focuses on testing interoperability between OpenStack clouds. +It is the official tool to run Openstack Interop (previously known as Defcore) +testcases `[9]`_, which focus on testing interoperability between OpenStack +clouds. Refstack-client is integrated in Functest, consumed by Dovetail, which intends to define and provide a set of OPNFV related validation criteria that will provide input for the evaluation of the use of OPNFV trademarks. This progress is under the guideline of Compliance Verification Program(CVP). -Defcore testcases -^^^^^^^^^^^^^^^^^^ - -*Danube Release* - -Set of DefCore tempest test cases not flagged and required. -According to `[10]`_, some tests are still flagged due to outstanding bugs -in the Tempest library, particularly tests that require SSH. Refstack developers -are working on correcting these bugs upstream. Please note that although some tests -are flagged because of bugs, there is still an expectation that the capabilities -covered by the tests are available. It only contains Openstack core compute -(no object storage). The approved guidelines (2016.08) are valid for Kilo, -Liberty, Mitaka and Newton releases of OpenStack. -The list can be generated using the Rest API from RefStack project: -https://refstack.openstack.org/api/v1/guidelines/2016.08/tests?target=compute&type=required&alias=true&flag=false - Running methods ^^^^^^^^^^^^^^^ @@ -314,8 +300,8 @@ snaps_smoke This test case contains tests that setup and destroy environments with VMs with and without Floating IPs with a newly created user and project. Set the config -value snaps.use_floating_ips (True|False) to toggle this functionality. When -the config value of snaps.use_keystone is True, Functest must have access +value snaps.use_floating_ips (True|False) to toggle this functionality. +Please note that When the configuration value of snaps.use_keystone is True, Functest must have access the cloud's private network. This suite consists in 38 tests (test duration < 10 minutes) @@ -323,12 +309,6 @@ This suite consists in 38 tests (test duration < 10 minutes) SDN Controllers --------------- -There are currently 3 available controllers: - - * OpenDaylight (ODL) - * ONOS - * OpenContrail (OCL) - OpenDaylight ^^^^^^^^^^^^ @@ -397,7 +377,7 @@ Functest has been supporting several feature projects since Brahpamutra: +-----------------+---------+----------+--------+-----------+ | fds | | | X | X | +-----------------+---------+----------+--------+-----------+ -| moon | | X | | X | +| moon | | X | | | +-----------------+---------+----------+--------+-----------+ | multisite | | X | X | | +-----------------+---------+----------+--------+-----------+ @@ -409,7 +389,7 @@ Functest has been supporting several feature projects since Brahpamutra: +-----------------+---------+----------+--------+-----------+ | orchestra | | | X | X | +-----------------+---------+----------+--------+-----------+ -| parser | | | X | | +| parser | | | X | X | +-----------------+---------+----------+--------+-----------+ | promise | X | X | X | X | +-----------------+---------+----------+--------+-----------+ @@ -482,7 +462,7 @@ orchestrator. parser ^^^^^^ -See parser user guide for details: `[12]`_ +See parser user guide for details. vyos-vrouter @@ -505,13 +485,38 @@ The Workflow is as follows: The vyos-vrouter architecture is described in `[14]`_ +cloudify_ims_perf +^^^^^^^^^^^^^^^^^ + +This test case is available but not declared in testcases.yaml. If you want to +run it you need to get the Ixia loader images and have access to an Ixia license +server. + +The declaration of the testcase is simple, connect to your functest-vnf docker, +add the following section in /usr/lib/python2.7/site-packacges/functest/ci/testcases.yaml:: + + case_name: cloudify_ims_perf + project_name: functest + criteria: 80 + blocking: false + description: >- + Stress tests based on Cloudify. Ixia loader images and access to Ixia + server license. + dependencies: + installer: '' + scenario: 'os-nosdn-nofeature-ha' + run: + module: 'functest.opnfv_tests.vnf.ims.cloudify_ims_perf' + class: 'CloudifyImsPerf' + + .. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html .. _`[3]`: https://rally.readthedocs.org/en/latest/index.html .. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml .. _`[8]`: https://github.com/openstack/refstack-client .. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst .. _`[11]`: http://robotframework.org/ -.. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html +.. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html .. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter .. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/ diff --git a/docs/testing/user/userguide/test_overview.rst b/docs/testing/user/userguide/test_overview.rst index b9faa24a0..2fca325a2 100644 --- a/docs/testing/user/userguide/test_overview.rst +++ b/docs/testing/user/userguide/test_overview.rst @@ -4,14 +4,16 @@ Overview of the Functest suites =============================== -Functest is the OPNFV project primarily targeting function testing. +Functest is the OPNFV project primarily targeting functional testing. In the Continuous Integration pipeline, it is launched after an OPNFV fresh installation to validate and verify the basic functions of the infrastructure. -The current list of test suites can be distributed over 5 main domains: VIM -(Virtualised Infrastructure Manager), Controllers (i.e. SDN Controllers), -Features, VNF (Virtual Network Functions) and MANO stacks. +The current list of test suites can be distributed over 4 main domains: + * VIM (Virtualised Infrastructure Manager) + * Controllers (i.e. SDN Controllers) + * Features + * VNF (Virtual Network Functions) Functest test suites are also distributed in the OPNFV testing categories: healthcheck, smoke, features, components, performance, VNF, Stress tests. @@ -41,20 +43,20 @@ validate the scenario for the release. | | | | VM and then executed via SSH. | | | | | The script will ping another | | | | | VM on a specified IP address over| -| | | | the SUT Private Tenant network. | +| | | | the SUT Private Tenant network | | | +----------------+----------------------------------+ | | | vping_userdata | Uses Ping with given userdata | | | | | to test intra-VM connectivity | | | | | over the SUT Private Tenant | | | | | network. The correct operation | | | | | of the NOVA Metadata service is | -| | | | also verified in this test. | +| | | | also verified in this test | | | +----------------+----------------------------------+ | | | tempest_smoke | Generate and run a relevant | | | | \_serial | Tempest Test Suite in smoke mode.| | | | | The generated test set is | | | | | dependent on the OpenStack | -| | | | deployment environment. | +| | | | deployment environment | | | +----------------+----------------------------------+ | | | rally_sanity | Run a subset of the OpenStack | | | | | Rally Test Suite in smoke mode | @@ -70,11 +72,11 @@ validate the scenario for the release. | | | | See the OpenStack reference test | | | | | suite `[2]`_. The generated | | | | | test set is dependent on the | -| | | | OpenStack deployment environment.| +| | | | OpenStack deployment environment | | | +----------------+----------------------------------+ | | | rally_full | Run the OpenStack testing tool | | | | | benchmarking OpenStack modules | -| | | | See the Rally documents `[3]`_. | +| | | | See the Rally documents `[3]`_ | | | +----------------+----------------------------------+ | | | tempest_custom | Allow to run a customized list | | | | | of Tempest cases | @@ -149,7 +151,7 @@ validate the scenario for the release. | | | | regarding compute, network and | | | | | storage. | | | | | See `Promise User Guide`_ for | -| | | | details. | +| | | | details | +-------------+---------------+----------------+----------------------------------+ | VNF | vnf | cloudify_ims | Example of a real VNF deployment | | | | | to show the NFV capabilities of | @@ -166,6 +168,21 @@ validate the scenario for the release. | | | cleawaterims | Openbaton orchestrator | | | +----------------+----------------------------------+ | | | vyos_vrouter | vRouter testing | +| | +----------------+----------------------------------+ +| | | cloudify_ims | Based on cloudify_ims test case | +| | | perf | cloudify_ims_perf substitutes | +| | | | the signaling test suite by an | +| | | | automatic deployment of an Ixia | +| | | | loader and generic SIP stress | +| | | | tests. | +| | | | This work has been initiated | +| | | | during the plugfest and allows | +| | | | realistic load tests on top of | +| | | | cloudify_ims. Please note that | +| | | | this test is available but not | +| | | | declared in testcases.yaml as it | +| | | | requires access to proprietary | +| | | | resources (Ixia loader) | +-------------+---------------+----------------+----------------------------------+ @@ -199,7 +216,7 @@ NoSQL database. The goal is to populate the database with results from different sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot of a live Functest Dashboard is shown below: -.. figure:: ../../../images/FunctestDashboardDanube.png +.. figure:: ../../../images/FunctestDashboardEuphrates.png :align: center :alt: Functest Dashboard @@ -215,15 +232,15 @@ This parameters as well as possible tags can be used for the Test case catalog. vIMS test case was integrated to demonstrate the capability to deploy a relatively complex NFV scenario on top of the OPNFV infrastructure. -Functest considers OPNFV as a black box. As of Danube release the OPNFV -offers a lot of potential combinations: +Functest considers OPNFV as a black box. OPNFV offers a lot of potential +combinations (which may change from one version to another): * 3 controllers (OpenDaylight, ONOS, OpenContrail) * 5 installers (Apex, Compass, Daisy, Fuel, Joid) Most of the tests are runnable by any combination, but some tests might have restrictions imposed by the utilized installers or due to the available -deployed features. The system uses the environment variables (INSTALLER_IP and +deployed features. The system uses the environment variables (INSTALLER_TYPE and DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given environment. @@ -233,8 +250,8 @@ clean-up) and for executing tests. The Functest CLI organised the testcase into logical Tiers, which contain in turn one or more testcases. The CLI allows execution of a single specified testcase, all test cases in a specified Tier, or the special case of execution -of **ALL** testcases. The Functest CLI is introduced in more detail in the -section `Executing the functest suites`_ of this document. +of **ALL** testcases. The Functest CLI is introduced in more details in next +section. .. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html .. _`[3]`: https://rally.readthedocs.org/en/latest/index.html diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst index 53e4d3a86..6129ef3a0 100644 --- a/docs/testing/user/userguide/test_results.rst +++ b/docs/testing/user/userguide/test_results.rst @@ -7,7 +7,8 @@ Manual testing In manual mode test results are displayed in the console and result files are put in /home/opnfv/functest/results. -If you want additionnal logs, you may configure the logging.ini under <repo>/functest:functest/ci +If you want additional logs, you may configure the logging.ini under +/usr/lib/python2.7/site-packages/functest/ci. Automated testing -------------- @@ -47,4 +48,4 @@ Based on the results stored in the result database, a `Functest reporting`_ portal is also automatically updated. This portal provides information on the overall status per scenario and per installer -.. _`Functest reporting`: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-fuel.html +.. _`Functest reporting`: http://testresults.opnfv.org/reporting/master/functest/status-apex.html diff --git a/docs/testing/user/userguide/troubleshooting.rst b/docs/testing/user/userguide/troubleshooting.rst index 1c5166081..29facc294 100644 --- a/docs/testing/user/userguide/troubleshooting.rst +++ b/docs/testing/user/userguide/troubleshooting.rst @@ -43,9 +43,9 @@ These test cases can be run inside the container, using new Functest CLI as foll $ functest testcase run vping_userdata The Functest CLI is designed to route a call to the corresponding internal -python scripts, located in paths: -*$REPOS_DIR/functest/functest/opnfv_tests/openstack/vping/vping_ssh.py* and -*$REPOS_DIR/functest/functest/opnfv_tests/openstack/vping/vping_userdata.py* +python scripts, located in paths +/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_ssh.py +and /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_userdata.py Notes: @@ -122,9 +122,10 @@ This test case creates a floating IP on the external network and assigns it to the second instance **opnfv-vping-2**. The purpose of this is to establish a SSH connection to that instance and SCP a script that will ping the first instance. This script is located in the repository under -*$REPOS_DIR/functest/functest/opnfv_tests/openstack/vping/ping.sh* and takes an IP as -a parameter. When the SCP is completed, the test will do an SSH call to that script -inside the second instance. Some problems can happen here:: +/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/ping.sh +and takes an IP as a parameter. When the SCP is completed, the test will do a +SSH call to that script inside the second instance. Some problems can happen +here:: vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting @@ -210,9 +211,6 @@ If this text or similar is shown:: it means that the instance failed to read from the metadata service. Contact the Functest or installer teams for more information. -NOTE: Cloud-init in not supported on scenarios dealing with ONOS and the tests -have been excluded from CI in those scenarios. - Tempest ^^^^^^^ @@ -238,7 +236,7 @@ of the following | execution test cases are | Rally in Functest, does not contain all the needed | | missing | parameters or some parameters are not set properly. | | | The tempest.conf file is located in directory | -| | '/home/opnfv/.rally/verification/verifier-<UUID> | +| | 'root/.rally/verification/verifier-<UUID> | | | /for-deployment-<UUID>' | | | in the Functest Docker container. Use the "rally | | | deployment list" command in order to check the UUID | @@ -266,12 +264,12 @@ Possible scenarios are: * keystone * neutron * nova + * ceilometer * quotas - * requests * vm To know more about what those scenarios are doing, they are defined in directory: -*$REPOS_DIR/functest/functest/opnfv_tests/openstack/rally/scenario* +/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/rally/scenario For more info about Rally scenario definition please refer to the Rally official documentation. `[3]`_ @@ -295,10 +293,6 @@ If any of the other test cases fails, check that Neutron and ODL have been correctly configured to work together. Check Neutron configuration files, accounts, IP addresses etc.). -ONOS -^^^^ -Please refer to the ONOS documentation. `ONOSFW User Guide`_ . - Features -------- @@ -306,7 +300,6 @@ Features Please refer to the dedicated feature user guides for details. - VNF --- @@ -342,6 +335,7 @@ described in the following table: | the VM | the vIMS VNF installation fails | +-----------------------------------+------------------------------------+ - +Please note that this test case requires resources (8 VM (2Go) + 1 VM (4Go)), it +is there fore not recommended to run it on a light configuration. .. _`OPNFV Functest Developer Guide`: http://artifacts.opnfv.org/functest/docs/devguide/# |