diff options
Diffstat (limited to 'docs/testing/user/userguide')
| -rw-r--r-- | docs/testing/user/userguide/index.rst | 74 | ||||
| -rw-r--r-- | docs/testing/user/userguide/intro.rst | 10 | ||||
| -rw-r--r-- | docs/testing/user/userguide/reporting.rst | 28 | ||||
| -rw-r--r-- | docs/testing/user/userguide/runfunctest.rst | 140 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_details.rst | 355 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_overview.rst | 281 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_results.rst | 178 | ||||
| -rw-r--r-- | docs/testing/user/userguide/troubleshooting.rst | 195 |
8 files changed, 544 insertions, 717 deletions
diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst index 66dfd3e7b..1e73cd622 100644 --- a/docs/testing/user/userguide/index.rst +++ b/docs/testing/user/userguide/index.rst @@ -1,8 +1,7 @@ -.. _functest-userguide: - -.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 +.. _functest-userguide: + ******************* Functest User Guide ******************* @@ -10,39 +9,17 @@ Functest User Guide .. toctree:: :maxdepth: 2 - - -Introduction -============ - -The goal of this document is to describe the OPNFV Functest test cases and to -provide a procedure to execute them. - -**IMPORTANT**: It is assumed here that Functest has been properly deployed -following the installation guide procedure `[1]`_. - -.. include:: ./test_overview.rst - -.. include:: ./test_details.rst - -.. include:: ./runfunctest.rst - -.. include:: ./test_results.rst - -.. include:: ./reporting.rst - -.. figure:: ../../../images/functest-reporting-status.png - :align: center - :alt: Functest reporting portal Fuel status page - -.. include:: ./troubleshooting.rst + intro.rst + test_overview.rst + test_details.rst + test_results.rst + reporting.rst + troubleshooting.rst References ========== -`[1]`_: Functest configuration guide - `[2]`_: OpenStack Tempest documentation `[3]`_: Rally documentation @@ -51,7 +28,7 @@ References `[5]`_: Clearwater vIMS blueprint -`[6]`_: NIST web site +`[6]`_: Security Content Automation Protocol `[7]`_: OpenSCAP web site @@ -63,17 +40,13 @@ References `[11]`_: Robot Framework web site -`[12]`_: Functest User guide - -`[13]`_: SNAPS wiki +`[13]`_: SNAPS `[14]`_: vRouter `[15]`_: Testing OpenStack Tempest part 1 -`[16]`_: Running Functest through internal REST API - -`[17]`_: OPNFV Test API +`[16]`_: OPNFV Test API `OPNFV main site`_: OPNFV official web site @@ -81,26 +54,23 @@ References IRC support chan: #opnfv-functest -.. _`[1]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html -.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html -.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html -.. _`[4]`: http://events.linuxfoundation.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf +.. _`[2]`: https://docs.openstack.org/tempest/latest/ +.. _`[3]`: https://rally.readthedocs.io/en/latest/index.html +.. _`[4]`: https://events.static.linuxfound.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf .. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml -.. _`[6]`: https://scap.nist.gov/ +.. _`[6]`: https://en.wikipedia.org/wiki/Security_Content_Automation_Protocol .. _`[7]`: https://github.com/OpenSCAP/openscap .. _`[8]`: https://github.com/openstack/refstack-client -.. _`[9]`: https://github.com/openstack/defcore +.. _`[9]`: https://github.com/openstack/interop .. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst -.. _`[11]`: http://robotframework.org/ -.. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html -.. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO +.. _`[11]`: https://robotframework.org/ +.. _`[13]`: https://git.opnfv.org/snaps/ .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter .. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/ -.. _`[16]`: https://wiki.opnfv.org/display/functest/Running+test+cases+via+new+Functest+REST+API -.. _`[17]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html -.. _`OPNFV main site`: http://www.opnfv.org -.. _`Functest page`: https://wiki.opnfv.org/functest +.. _`[16]`: http://testresults.opnfv.org/test/ +.. _`OPNFV main site`: https://www.opnfv.org/ +.. _`Functest page`: https://github.com/opnfv/functest/ .. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html .. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html -.. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/functest/ci/config_functest.yaml +.. _`config_functest.yaml` : https://github.com/opnfv/functest/blob/master/functest/ci/config_functest.yaml .. _`Functest reporting`: http://testresults.opnfv.org/reporting/master/functest/status-apex.html diff --git a/docs/testing/user/userguide/intro.rst b/docs/testing/user/userguide/intro.rst new file mode 100644 index 000000000..c001afb2f --- /dev/null +++ b/docs/testing/user/userguide/intro.rst @@ -0,0 +1,10 @@ +.. SPDX-License-Identifier: CC-BY-4.0 + +Introduction +============ + +The goal of this document is to describe the OPNFV Functest test cases and to +provide a procedure to execute them. + +**IMPORTANT**: It is assumed here that Functest has been properly deployed +following the installation guide procedure :ref:`functest-install-guide`. diff --git a/docs/testing/user/userguide/reporting.rst b/docs/testing/user/userguide/reporting.rst index 88f5e865a..8fad55d33 100644 --- a/docs/testing/user/userguide/reporting.rst +++ b/docs/testing/user/userguide/reporting.rst @@ -1,4 +1,3 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 Test reporting @@ -26,7 +25,7 @@ and features) corresponding to this scenario. +---------------------+---------+---------+---------+---------+ | vPing_userdata | X | X | X | X | +---------------------+---------+---------+---------+---------+ - | tempest_smoke_serial| X | X | X | X | + | tempest_smoke | X | X | X | X | +---------------------+---------+---------+---------+---------+ | rally_sanity | X | X | X | X | +---------------------+---------+---------+---------+---------+ @@ -42,12 +41,14 @@ and features) corresponding to this scenario. +---------------------+---------+---------+---------+---------+ | copper | X | | | X | +---------------------+---------+---------+---------+---------+ - src: os-odl_l2-nofeature-ha Colorado (see release note for the last matrix version) + + src: os-odl_l2-nofeature-ha Colorado (see release note for the last matrix + version) All the testcases (X) listed in the table are runnable on os-odl_l2-nofeature scenarios. Please note that other test cases (e.g. sfc_odl, bgpvpn) need ODL configuration -addons and, as a consequence, specific scenario. +add-ons and, as a consequence, specific scenario. There are not considered as runnable on the generic odl_l2 scenario. @@ -69,16 +70,20 @@ scoring. In fact the success criteria are not always easy to define and may require specific hardware configuration. Please also note that all the test cases have the same "weight" for the score -calculation whatever the complexity of the test case. Concretely a vping has the -same weith than the 200 tempst tests. +calculation whatever the complexity of the test case. Concretely a vping has +the same weight than the 200 tempest tests. Moreover some installers support more features than others. The more cases your scenario is dealing with, the most difficult to rich a good scoring. Therefore the scoring provides 3 types of indicators: - * the richness of the scenario: if the target scoring is high, it means that the scenario includes lots of features - * the maturity: if the percentage (scoring/target scoring * 100) is high, it means that all the tests are PASS - * the stability: as the number of iteration is included in the calculation, the pecentage can be high only if the scenario is run regularly (at least more than 4 iterations over the last 10 days in CI) + * the richness of the scenario: if the target scoring is high, it means that + the scenario includes lots of features + * the maturity: if the percentage (scoring/target scoring * 100) is high, it + means that all the tests are PASS + * the stability: as the number of iteration is included in the calculation, + the percentage can be high only if the scenario is run regularly (at least + more than 4 iterations over the last 10 days in CI) In any case, the scoring is used to give feedback to the other projects and does not represent an absolute value of the scenario. @@ -86,5 +91,8 @@ does not represent an absolute value of the scenario. See `reporting page`_ for details. For the status, click on the version, Functest then the Status menu. - .. _`reporting page`: http://testresults.opnfv.org/reporting/ + +.. figure:: ../../../images/functest-reporting-status.png + :align: center + :alt: Functest reporting portal Fuel status page diff --git a/docs/testing/user/userguide/runfunctest.rst b/docs/testing/user/userguide/runfunctest.rst deleted file mode 100644 index 89580e564..000000000 --- a/docs/testing/user/userguide/runfunctest.rst +++ /dev/null @@ -1,140 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - - -Executing Functest suites -========================= - -As mentioned in the configuration guide `[1]`_, Alpine docker containers have -been introduced in Euphrates. -Tier containers have been created. -Assuming that you pulled the container and your environement is ready, you can -simply run the tiers by typing (e.g. with functest-healthcheck):: - - sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-healthcheck - -You should get:: - - +----------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:02 | PASS | - | api_check | functest | healthcheck | 03:19 | PASS | - | snaps_health_check | functest | healthcheck | 00:46 | PASS | - +----------------------------+------------------+---------------------+------------------+----------------+ - -You can run functest-healcheck, functest-smoke, functest-features, -functest-components and functest-vnf. - -The result tables show the results by test case, it can be:: - - * PASS - * FAIL - * SKIP: if the scenario/installer does not support the test case - - -Manual run -========== -If you want to run the test step by step, you may add docker option then run the -different commands within the docker. - -Considering the healthcheck example, running functest manaully means:: - - sudo docker run -ti --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-healthcheck /bin/bash - -The docker prompt shall be returned. Then within the docker run the following -commands:: - - $ source /home/opnfv/functest/conf/openstack.creds - -Tier ----- -Each Alpine container provided on the docker hub matches with a tier. -The following commands are available:: - - # functest tier list - - 0. healthcheck: - ['connection_check', 'api_check', 'snaps_health_check'] - # functest tier show healthcheck - +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ - | TIERS | ORDER | CI LOOP | DESCRIPTION | TESTCASES | - +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ - | healthcheck | 0 | (daily)|(weekly) | First tier to be executed to verify the | connection_check api_check | - | | | | basic operations in the VIM. | snaps_health_check | - +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ - -To run all the cases of the tier, type:: - - # functest tier run healthcheck - -Testcase --------- -Testcases can be listed, shown and run though the CLI:: - - # functest testcase list - connection_check - api_check - snaps_health_check - # functest testcase show api_check - +-------------------+--------------------------------------------------+------------------+---------------------------+ - | TEST CASE | DESCRIPTION | CRITERIA | DEPENDENCY | - +-------------------+--------------------------------------------------+------------------+---------------------------+ - | api_check | This test case verifies the retrieval of | 100 | ^((?!netvirt).)*$ | - | | OpenStack clients: Keystone, Glance, | | | - | | Neutron and Nova and may perform some | | | - | | simple queries. When the config value of | | | - | | snaps.use_keystone is True, functest | | | - | | must have access to the cloud's private | | | - | | network. | | | - +-------------------+--------------------------------------------------+------------------+---------------------------+ - # functest testcase run connection_check - ... - # functest run all - -You can also type run_tests -t all to run all the tests. - -Note the list of test cases depend on the installer and the scenario. - - -Reporting results to the test Database -====================================== -In OPNFV CI we collect all the results from CI. A test APi shall be available -as well as a test database `[17]`_. - -Functest internal API -===================== - -An internal API has been introduced in Euphrates. The goal is to trigger -Functest operations through an API in addition of the CLI. -This could be considered as a first step towards a pseudo micro services -approach where the different test projects could expose and consume APIs to the -other test projects. - -In Euphrates the main method of the APIs are: - - * Show credentials - * Update openrc file - * Show environment - * Update hosts info for domain name - * Prepare environment - * List all testcases - * Show a testcase - * Run a testcase - * List all tiers - * Show a tier - * List all testcases within given tier - * Get the result of the specified task - * Get the log of the specified task - -See `[16]`_ to get examples on how to use the API. - - -.. _`[1]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html -.. _`[16]`: https://wiki.opnfv.org/display/functest/Running+test+cases+via+new+Functest+REST+API -.. _`[17]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html diff --git a/docs/testing/user/userguide/test_details.rst b/docs/testing/user/userguide/test_details.rst index 97c4688cc..98247d488 100644 --- a/docs/testing/user/userguide/test_details.rst +++ b/docs/testing/user/userguide/test_details.rst @@ -1,8 +1,7 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 +.. SPDX-License-Identifier: CC-BY-4.0 - -The different test cases are described in the remaining sections of this document. +The different test cases are described in the remaining sections of this +document. VIM (Virtualized Infrastructure Manager) ---------------------------------------- @@ -25,20 +24,10 @@ The tests are: * *connection_check* - * *api_check* - * *snaps_health_check* -Connection_check consists in 9 test cases (test duration < 5s) checking the +Connection_check consists in test cases (test duration < 5s) checking the connectivity with Glance, Keystone, Neutron, Nova and the external network. -Api_check verifies the retrieval of OpenStack clients: Keystone, Glance, -Neutron and Nova and may perform some simple queries. When the config value of -snaps.use_keystone is True, functest must have access to the cloud's private -network. This suite consists in 49 tests (test duration < 2 minutes). - -Snaps_health_check creates a VM with a single port with an IPv4 address that -is assigned by DHCP and then validates the expected IP with the actual. - Self-obviously, successful completion of the 'healthcheck' testcase is a necessary pre-requisite for the execution of all other test Tiers. @@ -59,8 +48,8 @@ Given the script **ping.sh**:: The goal of this test is to establish an SSH connection using a floating IP -on the Public/External network and verify that 2 instances can talk over a Private -Tenant network:: +on the Public/External network and verify that 2 instances can talk over a +Private Tenant network:: vPing_ssh test case +-------------+ +-------------+ @@ -105,7 +94,8 @@ vPing_userdata This test case is similar to vPing_ssh but without the use of Floating IPs and the Public/External network to transfer the ping script. -Instead, it uses Nova metadata service to pass it to the instance at booting time. +Instead, it uses Nova metadata service to pass it to the instance at booting +time. As vPing_ssh, it checks that 2 instances can talk to each other on a Private Tenant network:: @@ -158,17 +148,32 @@ updates the appropriate parameters into the configuration file. When the Tempest suite is executed, each test duration is measured and the full console output is stored to a *log* file for further analysis. -The Tempest testcases are distributed across two +The Tempest testcases are distributed across three Tiers: - * Smoke Tier - Test Case 'tempest_smoke_serial' - * Components Tier - Test case 'tempest_full_parallel' + * Smoke Tier - Test Case 'tempest_smoke' + * Components Tier - Test case 'tempest_full' + * Neutron Trunk Port - Test case 'neutron_trunk' + * OpenStack interop testcases - Test case 'refstack_defcore' + * Testing and verifying RBAC policy enforcement - Test case 'patrole' + +NOTE: Test case 'tempest_smoke' executes a defined set of tempest smoke +tests. Test case 'tempest_full' executes all defined Tempest tests. + +NOTE: The 'neutron_trunk' test set allows to connect a VM to multiple VLAN +separated networks using a single NIC. The feature neutron trunk ports have +been supported by Apex, Fuel and Compass, so the tempest testcases have been +integrated normally. -NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke -tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel' -executes all defined Tempest tests using several concurrent threads -(i.e. parallel mode). The number of threads activated corresponds to the number -of available logical CPUs. +NOTE: Rally is also used to run Openstack Interop testcases `[9]`_, which focus +on testing interoperability between OpenStack clouds. + +NOTE: Patrole is a tempest plugin for testing and verifying RBAC policy +enforcement. It runs Tempest-based API tests using specified RBAC roles, thus +allowing deployments to verify that only intended roles have access to those +APIs. Patrole currently offers testing for the following OpenStack services: +Nova, Neutron, Glance, Cinder and Keystone. Currently in functest, only neutron +and glance are tested. The goal of the Tempest test suite is to check the basic functionalities of the different OpenStack components on an OPNFV fresh installation, using the @@ -182,10 +187,11 @@ Rally `[3]`_ is a benchmarking tool that answers the question: *How does OpenStack work at scale?* -The goal of this test suite is to benchmark all the different OpenStack modules and -get significant figures that could help to define Telco Cloud KPIs. +The goal of this test suite is to benchmark all the different OpenStack modules +and get significant figures that could help to define Telco Cloud KPIs. -The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios: +The OPNFV Rally scenarios are based on the collection of the actual Rally +scenarios: * authenticate * cinder @@ -195,7 +201,6 @@ The OPNFV Rally scenarios are based on the collection of the actual Rally scenar * neutron * nova * quotas - * ceilometer A basic SLA (stop test on errors) has been implemented. @@ -208,104 +213,6 @@ NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test cases. Test case 'rally_full' executes the full defined set of Rally tests. -Refstack-client to run OpenStack interop testcases --------------------------------------------------- - -Refstack-client `[8]`_ is a command line utility that allows you to -execute Tempest test runs based on configurations you specify. -It is the official tool to run Openstack Interop (previously known as Defcore) -testcases `[9]`_, which focus on testing interoperability between OpenStack -clouds. - -Refstack-client is integrated in Functest, consumed by Dovetail, which -intends to define and provide a set of OPNFV related validation criteria -that will provide input for the evaluation of the use of OPNFV trademarks. -This progress is under the guideline of Compliance Verification Program(CVP). - -Running methods -^^^^^^^^^^^^^^^ - -Two running methods are provided after refstack-client integrated into -Functest, Functest command line and manually, respectively. - -By default, for Defcore test cases run by Functest command line, -are run followed with automatically generated -configuration file, i.e., refstack_tempest.conf. In some circumstances, -the automatic configuration file may not quite satisfied with the SUT, -Functest also inherits the refstack-client command line and provides a way -for users to set its configuration file according to its own SUT manually. - -*command line* - -Inside the Functest container, first to prepare Functest environment: - -:: - - functest env prepare - -then to run default defcore testcases by using refstack-client: - -:: - - functest testcase run refstack_defcore - -In OPNFV Continuous Integration(CI) system, the command line method is used. - -*manually* - -Prepare the tempest configuration file and the testcases want to run with the SUT, -run the testcases with: - -:: - - ./refstack-client test -c <Path of the tempest configuration file to use> -v --test-list <Path or URL of test list> - -using help for more information: - -:: - - ./refstack-client --help - ./refstack-client test --help - -Reference tempest configuration -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -*command line method* - -When command line method is used, the default tempest configuration file -is generated by Rally. - -*manually* - -When running manually is used, recommended way to generate tempest configuration -file is: - -:: - - cd /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/refstack_client - python tempest_conf.py - -a file called tempest.conf is stored in the current path by default, users can do -some adjustment according to the SUT: - -:: - - vim refstack_tempest.conf - -a reference article can be used `[15]`_. - - -snaps_smoke ------------- - -This test case contains tests that setup and destroy environments with VMs with -and without Floating IPs with a newly created user and project. Set the config -value snaps.use_floating_ips (True|False) to toggle this functionality. -Please note that When the configuration value of snaps.use_keystone is True, Functest must have access -the cloud's private network. -This suite consists in 38 tests (test duration < 10 minutes) - - SDN Controllers --------------- @@ -320,31 +227,40 @@ OpenDaylight and Neutron. The list of tests can be described as follows: * Basic Restconf test cases + * Connect to Restconf URL * Check the HTTP code status * Neutron Reachability test cases + * Get the complete list of neutron resources (networks, subnets, ports) * Neutron Network test cases + * Check OpenStack networks * Check OpenDaylight networks - * Create a new network via OpenStack and check the HTTP status code returned by Neutron + * Create a new network via OpenStack and check the HTTP status code returned + by Neutron * Check that the network has also been successfully created in OpenDaylight * Neutron Subnet test cases + * Check OpenStack subnets * Check OpenDaylight subnets - * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron + * Create a new subnet via OpenStack and check the HTTP status code returned + by Neutron * Check that the subnet has also been successfully created in OpenDaylight * Neutron Port test cases + * Check OpenStack Neutron for known ports * Check OpenDaylight ports - * Create a new port via OpenStack and check the HTTP status code returned by Neutron + * Create a new port via OpenStack and check the HTTP status code returned by + Neutron * Check that the new port has also been successfully created in OpenDaylight * Delete operations + * Delete the port previously created via OpenStack * Check that the port has been also successfully deleted in OpenDaylight * Delete previously subnet created via OpenStack @@ -356,49 +272,6 @@ Note: the checks in OpenDaylight are based on the returned HTTP status code returned by OpenDaylight. -Features --------- - -Functest has been supporting several feature projects since Brahpamutra: - - -+-----------------+---------+----------+--------+-----------+ -| Test | Brahma | Colorado | Danube | Euphrates | -+=================+=========+==========+========+===========+ -| barometer | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| bgpvpn | | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| copper | | X | | | -+-----------------+---------+----------+--------+-----------+ -| doctor | X | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| domino | | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| fds | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| moon | | X | | | -+-----------------+---------+----------+--------+-----------+ -| multisite | | X | X | | -+-----------------+---------+----------+--------+-----------+ -| netready | | | X | | -+-----------------+---------+----------+--------+-----------+ -| odl_sfc | | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| opera | | | X | | -+-----------------+---------+----------+--------+-----------+ -| orchestra | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| parser | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| promise | X | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| security_scan | | X | X | | -+-----------------+---------+----------+--------+-----------+ - -Please refer to the dedicated feature user guides for details. - - VNF --- @@ -409,9 +282,9 @@ The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an architectural framework for delivering IP multimedia services. vIMS has been integrated in Functest to demonstrate the capability to deploy a -relatively complex NFV scenario on the OPNFV platform. The deployment of a complete -functional VNF allows the test of most of the essential functions needed for a -NFV platform. +relatively complex NFV scenario on the OPNFV platform. The deployment of a +complete functional VNF allows the test of most of the essential functions +needed for a NFV platform. The goal of this test suite consists of: @@ -422,48 +295,31 @@ The goal of this test suite consists of: The Clearwater architecture is described as follows: -.. figure:: ../../../images/clearwater-architecture.png +.. figure:: ../../../images/clearwater-architecture-v2.png :align: center :alt: vIMS architecture +heat_ims +^^^^^^^^ +The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an +architectural framework for delivering IP multimedia services. -cloudify_ims_perf -^^^^^^^^^^^^^^^^^ -This testcase extends the cloudify_ims test case. -The first part is similar but the testing part is different. -The testing part consists in automating a realistic signaling load on the vIMS -using an Ixia loader (proprietary tools) - - You need to have access to an Ixia licence server defined in the configuration - file and have ixia image locally. - -This test case is available but not declared in testcases.yaml. The declaration -of the testcase is simple, connect to your functest-vnf docker, add the following -section in /usr/lib/python2.7/site-packacges/functest/ci/testcases.yaml:: - - - - case_name: cloudify_ims_perf - project_name: functest - criteria: 80 - blocking: false - description: >- - Stress tests based on Cloudify. Ixia loader images and access to Ixia - server license. - dependencies: - installer: '' - scenario: 'os-nosdn-nofeature-ha' - run: - module: 'functest.opnfv_tests.vnf.ims.cloudify_ims_perf' - class: 'CloudifyImsPerf' - -orchestra_openims -^^^^^^^^^^^^^^^^^ -Orchestra test case deals with the deployment of OpenIMS with OpenBaton -orchestrator. +vIMS has been integrated in Functest to demonstrate the capability to deploy a +relatively complex NFV scenario on the OPNFV platform. The deployment of a +complete functional VNF allows the test of most of the essential functions +needed for a NFV platform. -orchestra_clearwaterims -^^^^^^^^^^^^^^^^^^^^^^^ -Orchestra test case deals with the deployment of Clearwater vIMS with OpenBaton -orchestrator. +The goal of this test suite consists of: + +* deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF using + OpenStack Heat orchestrator based on a HOT template defined in `[17]`_ +* run suite of signaling tests on top of this VNF + +The Clearwater architecture is described as follows: + +.. figure:: ../../../images/clearwater-architecture-v2.png + :align: center + :alt: vIMS architecture vyos-vrouter ^^^^^^^^^^^^ @@ -484,14 +340,73 @@ The Workflow is as follows: The vyos-vrouter architecture is described in `[14]`_ +juju_epc +^^^^^^^^ +The Evolved Packet Core (EPC) is the main component of the System Architecture +Evolution (SAE) which forms the core of the 3GPP LTE specification. + +vEPC has been integrated in Functest to demonstrate the capability to deploy a +complex mobility-specific NFV scenario on the OPNFV platform. The OAI EPC +supports most of the essential functions defined by the 3GPP Technical Specs; +hence the successful execution of functional tests on the OAI EPC provides a +good endorsement of the underlying NFV platform. + +This integration also includes ABot, a Test Orchestration system that enables +test scenarios to be defined in high-level DSL. ABot is also deployed as a +VM on the OPNFV platform; and this provides an example of the automation +driver and the Test VNF being both deployed as separate VNFs on the underlying +OPNFV platform. + +The Workflow is as follows: + * Deploy Orchestrator + Deploy Juju controller using Bootstrap command. + * Deploy VNF + Deploy ABot orchestrator and OAI EPC as Juju charms. + Configuration of ABot and OAI EPC components is handled through + built-in Juju relations. + * Test VNF + Execution of ABot feature files triggered by Juju actions. + This executes a suite of LTE signalling tests on the OAI EPC. + * Reporting + ABot test results are parsed accordingly and pushed to Functest Db. + +Details of the ABot test orchestration tool may be found in `[15]`_ + +Kubernetes (K8s) +---------------- + +Kubernetes testing relies on sets of tests, which are part of the Kubernetes +source tree, such as the Kubernetes End-to-End (e2e) tests `[16]`_. + +The kubernetes testcases are distributed across various Tiers: + + * Healthcheck Tier + + * k8s_smoke Test Case: Creates a Guestbook application that contains redis + server, 2 instances of redis slave, frontend application, frontend service + and redis master service and redis slave service. Using frontend service, + the test will write an entry into the guestbook application which will + store the entry into the backend redis database. Application flow MUST + work as expected and the data written MUST be available to read. + + * Smoke Tier + + * k8s_conformance Test Case: Runs a series of k8s e2e tests expected to + pass on any Kubernetes cluster. It is a subset of tests necessary to + demonstrate conformance grows with each release. Conformance is thus + considered versioned, with backwards compatibility guarantees and are + designed to be run with no cloud provider configured. + -.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html -.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html +.. _`[2]`: https://docs.openstack.org/tempest/latest/ +.. _`[3]`: https://rally.readthedocs.io/en/latest/index.html .. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml .. _`[8]`: https://github.com/openstack/refstack-client +.. _`[9]`: https://github.com/openstack/interop .. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst -.. _`[11]`: http://robotframework.org/ -.. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html -.. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO +.. _`[11]`: https://robotframework.org/ +.. _`[13]`: https://git.opnfv.org/snaps/ .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter -.. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/ +.. _`[15]`: https://github.com/RebacaInc/abot_charm +.. _`[16]`: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md +.. _`[17]`: https://github.com/Metaswitch/clearwater-heat/blob/release-129/clearwater.yaml diff --git a/docs/testing/user/userguide/test_overview.rst b/docs/testing/user/userguide/test_overview.rst index a22a5067f..bc3e79dcb 100644 --- a/docs/testing/user/userguide/test_overview.rst +++ b/docs/testing/user/userguide/test_overview.rst @@ -1,5 +1,4 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 +.. SPDX-License-Identifier: CC-BY-4.0 Overview of the Functest suites =============================== @@ -9,158 +8,114 @@ In the Continuous Integration pipeline, it is launched after an OPNFV fresh installation to validate and verify the basic functions of the infrastructure. -The current list of test suites can be distributed over 4 main domains: +The current list of test suites can be distributed over 5 main domains: * VIM (Virtualised Infrastructure Manager) * Controllers (i.e. SDN Controllers) - * Features * VNF (Virtual Network Functions) + * Kubernetes Functest test suites are also distributed in the OPNFV testing categories: -healthcheck, smoke, features, components, performance, VNF, Stress tests. +healthcheck, smoke, benchmarking, VNF, Stress tests. -All the Healthcheck and smoke tests of a given scenario must be succesful to +All the Healthcheck and smoke tests of a given scenario must be successful to validate the scenario for the release. -+-------------+---------------+----------------+----------------------------------+ -| Domain | Tier | Test case | Comments | -+=============+===============+================+==================================+ -| VIM | healthcheck | connection | Check OpenStack connectivity | -| | | _check | through SNAPS framework | -| | +----------------+----------------------------------+ -| | | api_check | Check OpenStack API through | -| | | | SNAPS framework | -| | +----------------+----------------------------------+ -| | | snaps_health | basic instance creation, check | -| | | \_check | DHCP | -| +---------------+----------------+----------------------------------+ -| | smoke | vping_ssh | NFV "Hello World" using an SSH | -| | | | connection to a destination VM | -| | | | over a created floating IP | -| | | | address on the SUT Public / | -| | | | External network. Using the SSH | -| | | | connection a test script is then | -| | | | copied to the destination | -| | | | VM and then executed via SSH. | -| | | | The script will ping another | -| | | | VM on a specified IP address over| -| | | | the SUT Private Tenant network | -| | +----------------+----------------------------------+ -| | | vping_userdata | Uses Ping with given userdata | -| | | | to test intra-VM connectivity | -| | | | over the SUT Private Tenant | -| | | | network. The correct operation | -| | | | of the NOVA Metadata service is | -| | | | also verified in this test | -| | +----------------+----------------------------------+ -| | | tempest_smoke | Generate and run a relevant | -| | | \_serial | Tempest Test Suite in smoke mode.| -| | | | The generated test set is | -| | | | dependent on the OpenStack | -| | | | deployment environment | -| | +----------------+----------------------------------+ -| | | rally_sanity | Run a subset of the OpenStack | -| | | | Rally Test Suite in smoke mode | -| | +----------------+----------------------------------+ -| | | snaps_smoke | Run the SNAPS-OO integration | -| | | | tests | -| | +----------------+----------------------------------+ -| | | refstack | Reference RefStack suite | -| | | \_defcore | tempest selection for NFV | -| +---------------+----------------+----------------------------------+ -| | components | tempest_full | Generate and run a full set of | -| | | \_parallel | the OpenStack Tempest Test Suite.| -| | | | See the OpenStack reference test | -| | | | suite `[2]`_. The generated | -| | | | test set is dependent on the | -| | | | OpenStack deployment environment | -| | +----------------+----------------------------------+ -| | | rally_full | Run the OpenStack testing tool | -| | | | benchmarking OpenStack modules | -| | | | See the Rally documents `[3]`_ | -+-------------+---------------+----------------+----------------------------------+ -| Controllers | smoke | odl | Opendaylight Test suite | -| | | | Limited test suite to check the | -| | | | basic neutron (Layer 2) | -| | | | operations mainly based on | -| | | | upstream testcases. See below | -| | | | for details | -| | +----------------+----------------------------------+ -| | | odl_netvirt | Test Suite for the OpenDaylight | -| | | | SDN Controller when the NetVirt | -| | | | features are installed. It | -| | | | integrates some test suites from | -| | | | upstream using Robot as the test | -| | | | framework | -+-------------+---------------+----------------+----------------------------------+ -| Features | features | bgpvpn | Implementation of the OpenStack | -| | | | bgpvpn API from the SDNVPN | -| | | | feature project. It allows for | -| | | | the creation of BGP VPNs. | -| | | | See `SDNVPN User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | doctor | Doctor platform, as of Colorado | -| | | | release, provides the three | -| | | | features: | -| | | | * Immediate Notification | -| | | | * Consistent resource state | -| | | | awareness for compute host down | -| | | | * Valid compute host status | -| | | | given to VM owner | -| | | | See `Doctor User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | odl-sfc | SFC testing for odl scenarios | -| | | | See `SFC User Guide`_ for details| -| | +----------------+----------------------------------+ -| | | parser | Parser is an integration project | -| | | | which aims to provide | -| | | | placement/deployment templates | -| | | | translation for OPNFV platform, | -| | | | including TOSCA -> HOT, POLICY ->| -| | | | TOSCA and YANG -> TOSCA. it | -| | | | deals with a fake vRNC. | -| | | | See `Parser User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | fds | Test Suite for the OpenDaylight | -| | | | SDN Controller when the GBP | -| | | | features are installed. It | -| | | | integrates some test suites from | -| | | | upstream using Robot as the test | -| | | | framework | -+-------------+---------------+----------------+----------------------------------+ -| VNF | vnf | cloudify_ims | Example of a real VNF deployment | -| | | | to show the NFV capabilities of | -| | | | the platform. The IP Multimedia | -| | | | Subsytem is a typical Telco test | -| | | | case, referenced by ETSI. | -| | | | It provides a fully functional | -| | | | VoIP System | -| | +----------------+----------------------------------+ -| | | orchestra | OpenIMS deployment using | -| | | \_openims | Openbaton orchestrator | -| | +----------------+----------------------------------+ -| | | orchestra | Cleawater IMS deployment using | -| | | \_cleawaterims | Openbaton orchestrator | -| | +----------------+----------------------------------+ -| | | vyos_vrouter | vRouter testing | -| | +----------------+----------------------------------+ -| | | cloudify_ims | Based on cloudify_ims test case | -| | | perf | cloudify_ims_perf substitutes | -| | | | the signaling test suite by an | -| | | | automatic deployment of an Ixia | -| | | | loader and generic SIP stress | -| | | | tests. | -| | | | This work has been initiated | -| | | | during the plugfest and allows | -| | | | realistic load tests on top of | -| | | | cloudify_ims. Please note that | -| | | | this test is available but not | -| | | | declared in testcases.yaml as it | -| | | | requires access to proprietary | -| | | | resources (Ixia loader) | -+-------------+---------------+----------------+----------------------------------+ ++-------------+---------------+------------+----------------------------------+ +| Domain | Tier | Test case | Comments | ++=============+===============+============+==================================+ +| VIM | healthcheck | connection | Check OpenStack connectivity | +| | | \_check | | +| +---------------+------------+----------------------------------+ +| | smoke | vping_ssh | NFV "Hello World" using an SSH | +| | | | connection to a destination VM | +| | | | over a created floating IP | +| | | | address on the SUT Public / | +| | | | External network. Using the SSH | +| | | | connection a test script is then | +| | | | copied to the destination | +| | | | VM and then executed via SSH. | +| | | | The script will ping another | +| | | | VM on a specified IP address over| +| | | | the SUT Private Tenant network | +| | +------------+----------------------------------+ +| | | vping | Uses Ping with given userdata | +| | | \_userdata | to test intra-VM connectivity | +| | | | over the SUT Private Tenant | +| | | | network. The correct operation | +| | | | of the NOVA Metadata service is | +| | | | also verified in this test | +| | +------------+----------------------------------+ +| | | tempest | Generate and run a relevant | +| | | \_smoke | Tempest Test Suite in smoke mode.| +| | | | The generated test set is | +| | | | dependent on the OpenStack | +| | | | deployment environment | +| | +------------+----------------------------------+ +| | | rally | Run a subset of the OpenStack | +| | | \_sanity | Rally Test Suite in smoke mode | +| | +------------+----------------------------------+ +| | | refstack | Reference RefStack suite | +| | | \_defcore | tempest selection for NFV | +| | +------------+----------------------------------+ +| | | patrole | Patrole is a tempest plugin for | +| | | | testing and verifying RBAC policy| +| | | | enforcement, which offers testing| +| | | | for the following OpenStack | +| | | | services: Nova, Neutron, Glance, | +| | | | Cinder and Keystone | +| +---------------+------------+----------------------------------+ +| | | neutron | The neutron trunk port testcases | +| | | \_trunk | have been introduced and they are| +| | | | supported by installers : | +| | | | Apex, Fuel and Compass. | +| +---------------+------------+----------------------------------+ +| | components | tempest | Generate and run a full set of | +| | | \_full | the OpenStack Tempest Test Suite.| +| | | \_parallel | See the OpenStack reference test | +| | | | suite `[2]`_. The generated | +| | | | test set is dependent on the | +| | | | OpenStack deployment environment | +| | +------------+----------------------------------+ +| | | rally_full | Run the OpenStack testing tool | +| | | | benchmarking OpenStack modules | +| | | | See the Rally documents `[3]`_ | ++-------------+---------------+------------+----------------------------------+ +| Controllers | smoke | odl | Opendaylight Test suite | +| | | | Limited test suite to check the | +| | | | basic neutron (Layer 2) | +| | | | operations mainly based on | +| | | | upstream testcases. See below | +| | | | for details | ++-------------+---------------+------------+----------------------------------+ +| VNF | vnf | cloudify | Example of a real VNF deployment | +| | | \_ims | to show the NFV capabilities of | +| | | | the platform. The IP Multimedia | +| | | | Subsystem is a typical Telco test| +| | | | case, referenced by ETSI. | +| | | | It provides a fully functional | +| | | | VoIP System | +| | +------------+----------------------------------+ +| | | vyos | vRouter testing | +| | | \_vrouter | | +| | +------------+----------------------------------+ +| | | juju_epc | Validates deployment of a complex| +| | | | mobility VNF on OPNFV Platform. | +| | | | Uses Juju for deploying the OAI | +| | | | EPC and ABot for defining test | +| | | | scenarios using high-level DSL. | +| | | | VNF tests reference 3GPP | +| | | | Technical Specs and are executed | +| | | | through protocol drivers provided| +| | | | by ABot. | ++-------------+---------------+------------+----------------------------------+ +| Kubernetes | healthcheck | k8s_smoke | Test a running Kubernetes | +| | | | cluster and ensure it satisfies | +| | | | minimal functional requirements | +| +---------------+------------+----------------------------------+ +| | smoke | k8s\_ | Run a subset of Kubernetes | +| | | conformance| End-to-End tests, expected to | +| | | | pass on any Kubernetes cluster | ++-------------+---------------+------------+----------------------------------+ As shown in the above table, Functest is structured into different 'domains', @@ -169,29 +124,30 @@ As shown in the above table, Functest is structured into different 'domains', Test cases also have an implicit execution order. For example, if the early 'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke' -Tier testcases, there is little point to launch a full testcase execution round. +Tier testcases, there is little point to launch a full testcase execution +round. In Danube, we merged smoke and sdn controller tiers in smoke tier. An overview of the Functest Structural Concept is depicted graphically below: -.. figure:: ../../../images/concepts_mapping_final.png +.. figure:: ../../../images/concepts_mapping_fraser.png :align: center :alt: Functest Concepts Structure Some of the test cases are developed by Functest team members, whereas others are integrated from upstream communities or other OPNFV projects. For example, -`Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the +`Tempest <https://docs.openstack.org/tempest/latest/>`_ is the OpenStack integration test suite and Functest is in charge of the selection, integration and automation of those tests that fit suitably to OPNFV. -The Tempest test suite is the default OpenStack smoke test suite but no new test -cases have been created in OPNFV Functest. +The Tempest test suite is the default OpenStack smoke test suite but no new +test cases have been created in OPNFV Functest. The results produced by the tests run from CI are pushed and collected into a -NoSQL database. The goal is to populate the database with results from different -sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot -of a live Functest Dashboard is shown below: +NoSQL database. The goal is to populate the database with results from +different sources and scenarios and to show them on a `Functest Dashboard`_. A +screenshot of a live Functest Dashboard is shown below: .. figure:: ../../../images/FunctestDashboardEuphrates.png :align: center @@ -217,12 +173,11 @@ combinations (which may change from one version to another): Most of the tests are runnable by any combination, but some tests might have restrictions imposed by the utilized installers or due to the available -deployed features. The system uses the environment variables (INSTALLER_TYPE and -DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given -environment. +deployed services. The system uses the environment variables to automatically +determine the valid test cases, for each given environment. A convenience Functest CLI utility is also available to simplify setting up the -Functest evironment, management of the OpenStack environment (e.g. resource +Functest environment, management of the OpenStack environment (e.g. resource clean-up) and for executing tests. The Functest CLI organised the testcase into logical Tiers, which contain in turn one or more testcases. The CLI allows execution of a single specified @@ -230,10 +185,6 @@ testcase, all test cases in a specified Tier, or the special case of execution of **ALL** testcases. The Functest CLI is introduced in more details in next section. -.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html -.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html -.. _`Doctor User Guide`: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html -.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html -.. _`Parser User Guide`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html -.. _`Functest Dashboard`: http://testresults.opnfv.org/kibana_dashboards/ -.. _`SFC User Guide`: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html +.. _`[2]`: https://docs.openstack.org/tempest/latest/ +.. _`[3]`: https://rally.readthedocs.io/en/latest/index.html +.. _`Functest Dashboard`: http://testresults.opnfv.org/ diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst index 3941ba0a1..10f87d8ec 100644 --- a/docs/testing/user/userguide/test_results.rst +++ b/docs/testing/user/userguide/test_results.rst @@ -1,3 +1,5 @@ +.. SPDX-License-Identifier: CC-BY-4.0 + Test results ============ @@ -8,40 +10,154 @@ In manual mode test results are displayed in the console and result files are put in /home/opnfv/functest/results. If you want additional logs, you may configure the logging.ini under -/usr/lib/python2.7/site-packages/functest/ci. +/usr/lib/python3.8/site-packages/xtesting/ci. Automated testing --------------- +----------------- + +In automated mode, tests are run within split Alpine containers, and test +results are displayed in jenkins logs. The result summary is provided at the +end of each suite and can be described as follow. + +Healthcheck suite:: + + +--------------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------------+------------------+---------------------+------------------+----------------+ + | connection_check | functest | healthcheck | 00:03 | PASS | + | tenantnetwork1 | functest | healthcheck | 00:05 | PASS | + | tenantnetwork2 | functest | healthcheck | 00:06 | PASS | + | vmready1 | functest | healthcheck | 00:06 | PASS | + | vmready2 | functest | healthcheck | 00:08 | PASS | + | singlevm1 | functest | healthcheck | 00:32 | PASS | + | singlevm2 | functest | healthcheck | 00:37 | PASS | + | vping_ssh | functest | healthcheck | 00:46 | PASS | + | vping_userdata | functest | healthcheck | 00:39 | PASS | + | cinder_test | functest | healthcheck | 01:05 | PASS | + | tempest_smoke | functest | healthcheck | 05:39 | PASS | + | tempest_horizon | functest | healthcheck | 01:05 | PASS | + | odl | functest | healthcheck | 00:00 | SKIP | + +--------------------------+------------------+---------------------+------------------+----------------+ + +Smoke suite:: + + +---------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+---------------+------------------+----------------+ + | tempest_neutron | functest | smoke | 15:30 | PASS | + | tempest_cinder | functest | smoke | 02:01 | PASS | + | tempest_keystone | functest | smoke | 01:17 | PASS | + | tempest_heat | functest | smoke | 22:14 | PASS | + | tempest_telemetry | functest | smoke | 00:00 | SKIP | + | rally_sanity | functest | smoke | 17:24 | PASS | + | refstack_compute | functest | smoke | 07:03 | PASS | + | refstack_object | functest | smoke | 02:09 | PASS | + | refstack_platform | functest | smoke | 07:31 | PASS | + | tempest_full | functest | smoke | 41:52 | PASS | + | tempest_scenario | functest | smoke | 08:42 | PASS | + | tempest_slow | functest | smoke | 43:42 | PASS | + | patrole_admin | functest | smoke | 21:06 | PASS | + | patrole_member | functest | smoke | 21:23 | PASS | + | patrole_reader | functest | smoke | 21:56 | PASS | + | tempest_barbican | functest | smoke | 02:30 | PASS | + | tempest_octavia | functest | smoke | 00:00 | SKIP | + | tempest_cyborg | functest | smoke | 00:00 | SKIP | + +---------------------------+------------------+---------------+------------------+----------------+ + +Smoke CNTT suite:: + + +-------------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------------+------------------+---------------+------------------+----------------+ + | tempest_neutron_cntt | functest | smoke | 11:35 | PASS | + | tempest_cinder_cntt | functest | smoke | 01:58 | PASS | + | tempest_keystone_cntt | functest | smoke | 01:13 | PASS | + | tempest_heat_cntt | functest | smoke | 22:32 | PASS | + | rally_sanity_cntt | functest | smoke | 17:16 | PASS | + | tempest_full_cntt | functest | smoke | 41:13 | PASS | + | tempest_scenario_cntt | functest | smoke | 08:57 | PASS | + | tempest_slow_cntt | functest | smoke | 35:58 | PASS | + +-------------------------------+------------------+---------------+------------------+----------------+ + +Benchmarking suite:: + + +--------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------+------------------+----------------------+------------------+----------------+ + | rally_full | functest | benchmarking | 93:03 | PASS | + | rally_jobs | functest | benchmarking | 27:05 | PASS | + | vmtp | functest | benchmarking | 17:56 | PASS | + | shaker | functest | benchmarking | 24:02 | PASS | + +--------------------+------------------+----------------------+------------------+----------------+ + +Benchmarking CNTT suite:: + + +-------------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------+------------------+----------------------+------------------+----------------+ + | rally_full_cntt | functest | benchmarking | 89:52 | PASS | + | rally_jobs_cntt | functest | benchmarking | 19:39 | PASS | + | vmtp | functest | benchmarking | 16:59 | PASS | + | shaker | functest | benchmarking | 23:43 | PASS | + +-------------------------+------------------+----------------------+------------------+----------------+ + +Vnf suite:: + + +----------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +----------------------+------------------+--------------+------------------+----------------+ + | cloudify | functest | vnf | 05:08 | PASS | + | cloudify_ims | functest | vnf | 24:46 | PASS | + | heat_ims | functest | vnf | 33:12 | PASS | + | vyos_vrouter | functest | vnf | 15:53 | PASS | + | juju_epc | functest | vnf | 27:52 | PASS | + +----------------------+------------------+--------------+------------------+----------------+ + +Kubernetes healthcheck suite:: + + +-------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------+------------------+---------------------+------------------+----------------+ + | k8s_quick | functest | healthcheck | 00:18 | PASS | + | k8s_smoke | functest | healthcheck | 01:14 | PASS | + +-------------------+------------------+---------------------+------------------+----------------+ + +Kubernetes smoke suite:: + + +---------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+---------------+------------------+----------------+ + | k8s_conformance | functest | smoke | 94:26 | PASS | + | xrally_kubernetes | functest | smoke | 13:05 | PASS | + +---------------------------+------------------+---------------+------------------+----------------+ + +Kubernetes security suite:: + + +---------------------------+------------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+------------------+------------------+----------------+ + | kube_hunter | functest | security | 00:19 | PASS | + | kube_bench_master | functest | security | 00:02 | PASS | + | kube_bench_node | functest | security | 00:01 | PASS | + +---------------------------+------------------+------------------+------------------+----------------+ + +Kubernetes benchmarking suite:: + + +--------------------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------------------+------------------+----------------------+------------------+----------------+ + | xrally_kubernetes_full | functest | benchmarking | 34:16 | PASS | + +--------------------------------+------------------+----------------------+------------------+----------------+ + +Kubernetes cnf suite:: -In automated mode, test results are displayed in jenkins logs, a summary is provided -at the end of the job and can be described as follow:: - - +-------------------------+----------------------------------------------------------+ - | ENV VAR | VALUE | - +-------------------------+----------------------------------------------------------+ - | INSTALLER_TYPE | daisy | - | DEPLOY_SCENARIO | os-nosdn-nofeature-ha | - | BUILD_TAG | jenkins-functest-daisy-baremetal-daily-master-67 | - | CI_LOOP | daily | - +-------------------------+----------------------------------------------------------+ - - +------------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +------------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:08 | PASS | - | api_check | functest | healthcheck | 04:22 | PASS | - | snaps_health_check | functest | healthcheck | 00:35 | PASS | - | vping_ssh | functest | smoke | 00:54 | PASS | - | vping_userdata | functest | smoke | 00:27 | PASS | - | tempest_smoke_serial | functest | smoke | 19:39 | FAIL | - | rally_sanity | functest | smoke | 15:16 | PASS | - | refstack_defcore | functest | smoke | 15:55 | PASS | - | snaps_smoke | functest | smoke | 26:45 | FAIL | - | cloudify_ims | functest | vnf | 23:56 | PASS | - | orchestra_openims | orchestra | vnf | 15:07 | PASS | - | orchestra_clearwaterims | orchestra | vnf | 19:10 | PASS | - | vyos_vrouter | functest | vnf | 00:00 | SKIP | - +------------------------------+------------------+---------------------+------------------+----------------+ + +-------------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------+------------------+--------------+------------------+----------------+ + | k8s_vims | functest | cnf | 09:06 | PASS | + | helm_vims | functest | cnf | 08:54 | PASS | + | cnf_conformance | functest | cnf | 02:00 | PASS | + +-------------------------+------------------+--------------+------------------+----------------+ Results are automatically pushed to the test results database, some additional result files are pushed to OPNFV artifact web sites. diff --git a/docs/testing/user/userguide/troubleshooting.rst b/docs/testing/user/userguide/troubleshooting.rst index 1e342b653..d857ed4c4 100644 --- a/docs/testing/user/userguide/troubleshooting.rst +++ b/docs/testing/user/userguide/troubleshooting.rst @@ -1,5 +1,4 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 +.. SPDX-License-Identifier: CC-BY-4.0 Troubleshooting =============== @@ -15,19 +14,18 @@ credentials:: or:: - source /home/opnfv/functest/conf/openstack.creds + source /home/opnfv/functest/conf/env_file VIM --- This section covers the test cases related to the VIM (healthcheck, vping_ssh, -vping_userdata, tempest_smoke_serial, tempest_full_parallel, rally_sanity, -rally_full). +vping_userdata, tempest_smoke, tempest_full, rally_sanity, rally_full). vPing common ^^^^^^^^^^^^ -For both vPing test cases (**vPing_ssh**, and **vPing_userdata**), the first steps are -similar: +For both vPing test cases (**vPing_ssh**, and **vPing_userdata**), the first +steps are similar: * Create Glance image * Create Network @@ -37,15 +35,17 @@ similar: After these actions, the test cases differ and will be explained in their respective section. -These test cases can be run inside the container, using new Functest CLI as follows:: +These test cases can be run inside the container, using new Functest CLI as +follows:: - $ functest testcase run vping_ssh - $ functest testcase run vping_userdata + $ run_tests -t vping_ssh + $ run_tests -t vping_userdata The Functest CLI is designed to route a call to the corresponding internal -python scripts, located in paths -/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_ssh.py -and /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_userdata.py +python scripts, located in paths:: + + /usr/lib/python3.8/site-packages/functest/opnfv_tests/openstack/vping/vping_ssh.py + /usr/lib/python3.8/site-packages/functest/opnfv_tests/openstack/vping/vping_userdata.py Notes: @@ -73,10 +73,11 @@ Some of the common errors that can appear in this test case are:: vPing_ssh- ERROR - There has been a problem when creating the neutron network.... -This means that there has been some problems with Neutron, even before creating the -instances. Try to create manually a Neutron network and a Subnet to see if that works. -The debug messages will also help to see when it failed (subnet and router creation). -Example of Neutron commands (using 10.6.0.0/24 range for example):: +This means that there has been some problems with Neutron, even before creating +the instances. Try to create manually a Neutron network and a Subnet to see if +that works. The debug messages will also help to see when it failed (subnet and +router creation). Example of Neutron commands (using 10.6.0.0/24 range for +example):: neutron net-create net-test neutron subnet-create --name subnet-test --allocation-pool start=10.6.0.2,end=10.6.0.100 \ @@ -85,7 +86,8 @@ Example of Neutron commands (using 10.6.0.0/24 range for example):: neutron router-interface-add <ROUTER_ID> test_subnet neutron router-gateway-set <ROUTER_ID> <EXT_NET_NAME> -Another related error can occur while creating the Security Groups for the instances:: +Another related error can occur while creating the Security Groups for the +instances:: vPing_ssh- ERROR - Failed to create the security group... @@ -100,9 +102,9 @@ In this case, proceed to create it manually. These are some hints:: --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0 The next step is to create the instances. The image used is located in -*/home/opnfv/functest/data/cirros-0.4.0-x86_64-disk.img* and a Glance image is created -with the name **functest-vping**. If booting the instances fails (i.e. the status -is not **ACTIVE**), you can check why it failed by doing:: +*/home/opnfv/functest/data/cirros-0.5.1-x86_64-disk.img* and a Glance image is +created with the name **functest-vping**. If booting the instances fails (i.e. +the status is not **ACTIVE**), you can check why it failed by doing:: nova list nova show <INSTANCE_ID> @@ -113,7 +115,8 @@ It might show some messages about the booting failure. To try that manually:: This will spawn a VM using the network created previously manually. In all the OPNFV tested scenarios from CI, it never has been a problem with the -previous actions. Further possible problems are explained in the following sections. +previous actions. Further possible problems are explained in the following +sections. vPing_SSH @@ -122,7 +125,7 @@ This test case creates a floating IP on the external network and assigns it to the second instance **opnfv-vping-2**. The purpose of this is to establish a SSH connection to that instance and SCP a script that will ping the first instance. This script is located in the repository under -/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/ping.sh +/usr/lib/python3.8/site-packages/functest/opnfv_tests/openstack/vping/ping.sh and takes an IP as a parameter. When the SCP is completed, the test will do a SSH call to that script inside the second instance. Some problems can happen here:: @@ -130,8 +133,8 @@ here:: vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting If this is displayed, stop the test or wait for it to finish, if you have used -the special method of test invocation with specific supression of OpenStack -resource clean-up, as explained earler. It means that the Container can not +the special method of test invocation with specific suppression of OpenStack +resource clean-up, as explained earlier. It means that the Container can not reach the Public/External IP assigned to the instance **opnfv-vping-2**. There are many possible reasons, and they really depend on the chosen scenario. For most of the ODL-L3 and ONOS scenarios this has been noticed and it is a known @@ -143,19 +146,21 @@ from the DHCP agent. It can be checked by doing:: nova console-log opnfv-vping-2 If the message *Sending discover* and *No lease, failing* is shown, it probably -means that the Neutron dhcp-agent failed to assign an IP or even that it was not -responding. At this point it does not make sense to try to ping the floating IP. +means that the Neutron dhcp-agent failed to assign an IP or even that it was +not responding. At this point it does not make sense to try to ping the +floating IP. -If the instance got an IP properly, try to ping manually the VM from the container:: +If the instance got an IP properly, try to ping manually the VM from the +container:: nova list <grab the public IP> ping <public IP> -If the ping does not return anything, try to ping from the Host where the Docker -container is running. If that solves the problem, check the iptable rules because -there might be some rules rejecting ICMP or TCP traffic coming/going from/to the -container. +If the ping does not return anything, try to ping from the Host where the +Docker container is running. If that solves the problem, check the iptables +rules because there might be some rules rejecting ICMP or TCP traffic +coming/going from/to the container. At this point, if the ping does not work either, try to reproduce the test manually with the steps described above in the vPing common section with the @@ -176,7 +181,8 @@ vPing_userdata This test case does not create any floating IP neither establishes an SSH connection. Instead, it uses nova-metadata service when creating an instance to pass the same script as before (ping.sh) but as 1-line text. This script -will be executed automatically when the second instance **opnfv-vping-2** is booted. +will be executed automatically when the second instance **opnfv-vping-2** is +booted. The only known problem here for this test to fail is mainly the lack of support of cloud-init (nova-metadata service). Check the console of the instance:: @@ -219,79 +225,69 @@ In the upstream OpenStack CI all the Tempest test cases are supposed to pass. If some test cases fail in an OPNFV deployment, the reason is very probably one of the following -+-----------------------------+-----------------------------------------------------+ -| Error | Details | -+=============================+=====================================================+ -| Resources required for test | Such resources could be e.g. an external network | -| case execution are missing | and access to the management subnet (adminURL) from | -| | the Functest docker container. | -+-----------------------------+-----------------------------------------------------+ -| OpenStack components or | Check running services in the controller and compute| -| services are missing or not | nodes (e.g. with "systemctl" or "service" commands).| -| configured properly | Configuration parameters can be verified from the | -| | related .conf files located under '/etc/<component>'| -| | directories. | -+-----------------------------+-----------------------------------------------------+ -| Some resources required for | The tempest.conf file, automatically generated by | -| execution test cases are | Rally in Functest, does not contain all the needed | -| missing | parameters or some parameters are not set properly. | -| | The tempest.conf file is located in directory | -| | 'root/.rally/verification/verifier-<UUID> | -| | /for-deployment-<UUID>' | -| | in the Functest Docker container. Use the "rally | -| | deployment list" command in order to check the UUID | -| | the UUID of the current deployment. | -+-----------------------------+-----------------------------------------------------+ ++----------------------------+------------------------------------------------+ +| Error | Details | ++============================+================================================+ +| Resources required for | Such resources could be e.g. an external | +| testcase execution are | network and access to the management subnet | +| missing | (adminURL) from the Functest docker container. | ++----------------------------+------------------------------------------------+ +| OpenStack components or | Check running services in the controller and | +| services are missing or | compute nodes (e.g. with "systemctl" or | +| not configured properly | "service" commands). | +| | Configuration parameters can be verified from | +| | the related .conf files located under | +| | '/etc/<component>' directories. | ++----------------------------+------------------------------------------------+ +| Some resources required | The tempest.conf file, automatically generated | +| for execution test cases | by Rally in Functest, does not contain all the | +| are missing | needed parameters or some parameters are not | +| | set properly. | +| | The tempest.conf file is located in directory | +| | 'root/.rally/verification/verifier-<UUID> | +| | /for-deployment-<UUID>' | +| | in the Functest Docker container. Use the | +| | "rally deployment list" command in order to | +| | check the UUID of the current deployment. | ++----------------------------+------------------------------------------------+ When some Tempest test case fails, captured traceback and possibly also the -related REST API requests/responses are output to the console. More detailed debug -information can be found from tempest.log file stored into related Rally deployment -folder. +related REST API requests/responses are output to the console. More detailed +debug information can be found from tempest.log file stored into related Rally +deployment folder. Functest offers a possibility to test a customized list of Tempest test cases. -To enable that, add a new entry in docker/components/testcases.yaml on the "components" container -with the following content:: - - - - case_name: tempest_custom - project_name: functest - criteria: 100 - blocking: false - description: >- - The test case allows running a customized list of tempest - test cases - dependencies: - installer: '' - scenario: '' - run: - module: 'functest.opnfv_tests.openstack.tempest.tempest' - class: 'TempestCustom' - -Also, a list of the Tempest test cases must be provided to the container or modify -the existing one in -/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt - -Example of custom list of tests 'my-custom-tempest-tests.txt':: - - tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops[compute,id-7fff3fb3-91d8-4fd0-bd7d-0204f1f180ba,network,smoke] - tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops[compute,id-f323b3ba-82f8-4db7-8ea6-6a895869ec49,network,smoke] +To enable that, add a new entry in docker/smoke/testcases.yaml on the +"smoke" container with the following content:: + + - + case_name: tempest_custom + project_name: functest + criteria: 100 + blocking: false + description: >- + The test case allows running a customized list of tempest + test cases + run: + name: tempest_common + args: + mode: "tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops|\ + tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops" This is an example of running a customized list of Tempest tests in Functest:: sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ -v $(pwd)/images:/home/opnfv/functest/images \ - -v $(pwd)/my-custom-testcases.yaml:/usr/lib/python2.7/site-packages/functest/ci/testcases.yaml \ - -v $(pwd)/my-custom-tempest-tests.txt:/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt \ - opnfv/functest-components run_tests -t tempest_custom - + -v $(pwd)/my-custom-testcases.yaml:/usr/lib/python3.8/site-packages/xtesting/ci/testcases.yaml \ + opnfv/functest-smoke run_tests -t tempest_custom Rally ^^^^^ -The same error causes which were mentioned above for Tempest test cases, may also -lead to errors in Rally as well. +The same error causes which were mentioned above for Tempest test cases, may +also lead to errors in Rally as well. Possible scenarios are: * authenticate @@ -301,18 +297,19 @@ Possible scenarios are: * keystone * neutron * nova - * ceilometer * quotas * vm -To know more about what those scenarios are doing, they are defined in directory: -/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/rally/scenario -For more info about Rally scenario definition please refer to the Rally official -documentation. `[3]`_ +To know more about what those scenarios are doing, they are defined in +directory: +/usr/lib/python3.8/site-packages/functest/opnfv_tests/openstack/rally/scenario +For more info about Rally scenario definition please refer to the Rally +official documentation. `[3]`_ To check any possible problems with Rally, the logs are stored under */home/opnfv/functest/results/rally/* in the Functest Docker container. +.. _`[3]`: https://rally.readthedocs.io/en/latest/index.html Controllers ----------- @@ -372,7 +369,7 @@ described in the following table: | the VM | the vIMS VNF installation fails | +-----------------------------------+------------------------------------+ -Please note that this test case requires resources (8 VM (2Go) + 1 VM (4Go)), it -is there fore not recommended to run it on a light configuration. +Please note that this test case requires resources (8 VM (2Go) + 1 VM (4Go)), +it is there fore not recommended to run it on a light configuration. .. _`OPNFV Functest Developer Guide`: http://artifacts.opnfv.org/functest/docs/testing_developer_devguide/index.html# |