diff options
Diffstat (limited to 'docs/testing/user/userguide/test_details.rst')
| -rw-r--r-- | docs/testing/user/userguide/test_details.rst | 191 |
1 files changed, 60 insertions, 131 deletions
diff --git a/docs/testing/user/userguide/test_details.rst b/docs/testing/user/userguide/test_details.rst index 1ce4fce76..fe7e6a9b6 100644 --- a/docs/testing/user/userguide/test_details.rst +++ b/docs/testing/user/userguide/test_details.rst @@ -1,8 +1,7 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 +.. SPDX-License-Identifier: CC-BY-4.0 - -The different test cases are described in the remaining sections of this document. +The different test cases are described in the remaining sections of this +document. VIM (Virtualized Infrastructure Manager) ---------------------------------------- @@ -59,8 +58,8 @@ Given the script **ping.sh**:: The goal of this test is to establish an SSH connection using a floating IP -on the Public/External network and verify that 2 instances can talk over a Private -Tenant network:: +on the Public/External network and verify that 2 instances can talk over a +Private Tenant network:: vPing_ssh test case +-------------+ +-------------+ @@ -105,7 +104,8 @@ vPing_userdata This test case is similar to vPing_ssh but without the use of Floating IPs and the Public/External network to transfer the ping script. -Instead, it uses Nova metadata service to pass it to the instance at booting time. +Instead, it uses Nova metadata service to pass it to the instance at booting +time. As vPing_ssh, it checks that 2 instances can talk to each other on a Private Tenant network:: @@ -164,16 +164,29 @@ Tiers: * Smoke Tier - Test Case 'tempest_smoke_serial' * Components Tier - Test case 'tempest_full_parallel' * Neutron Trunk Port - Test case 'neutron_trunk' + * OpenStack interop testcases - Test case 'refstack_defcore' + * Testing and verifying RBAC policy enforcement - Test case 'patrole' NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke -tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel' -executes all defined Tempest tests using several concurrent threads -(i.e. parallel mode). The number of threads activated corresponds to the number -of available logical CPUs. - -NOTE: The 'neutron_trunk' test set allows to connect a VM to multiple VLAN separated -networks using a single NIC. The feature neutron trunk ports have been supported -by Apex, Fuel and Compass, so the tempest testcases have been integrated normally. +tests with a single thread (i.e. serial mode). Test case +'tempest_full_parallel' executes all defined Tempest tests using several +concurrent threads (i.e. parallel mode). The number of threads activated +corresponds to the number of available logical CPUs. + +NOTE: The 'neutron_trunk' test set allows to connect a VM to multiple VLAN +separated networks using a single NIC. The feature neutron trunk ports have +been supported by Apex, Fuel and Compass, so the tempest testcases have been +integrated normally. + +NOTE: Rally is also used to run Openstack Interop testcases `[9]`_, which focus +on testing interoperability between OpenStack clouds. + +NOTE: Patrole is a tempest plugin for testing and verifying RBAC policy +enforcement. It runs Tempest-based API tests using specified RBAC roles, thus +allowing deployments to verify that only intended roles have access to those +APIs. Patrole currently offers testing for the following OpenStack services: +Nova, Neutron, Glance, Cinder and Keystone. Currently in functest, only neutron +and glance are tested. The goal of the Tempest test suite is to check the basic functionalities of the different OpenStack components on an OPNFV fresh installation, using the @@ -187,10 +200,11 @@ Rally `[3]`_ is a benchmarking tool that answers the question: *How does OpenStack work at scale?* -The goal of this test suite is to benchmark all the different OpenStack modules and -get significant figures that could help to define Telco Cloud KPIs. +The goal of this test suite is to benchmark all the different OpenStack modules +and get significant figures that could help to define Telco Cloud KPIs. -The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios: +The OPNFV Rally scenarios are based on the collection of the actual Rally +scenarios: * authenticate * cinder @@ -200,7 +214,6 @@ The OPNFV Rally scenarios are based on the collection of the actual Rally scenar * neutron * nova * quotas - * ceilometer A basic SLA (stop test on errors) has been implemented. @@ -213,102 +226,15 @@ NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test cases. Test case 'rally_full' executes the full defined set of Rally tests. -Refstack-client to run OpenStack interop testcases --------------------------------------------------- - -Refstack-client `[8]`_ is a command line utility that allows you to -execute Tempest test runs based on configurations you specify. -It is the official tool to run Openstack Interop (previously known as Defcore) -testcases `[9]`_, which focus on testing interoperability between OpenStack -clouds. - -Refstack-client is integrated in Functest, consumed by Dovetail, which -intends to define and provide a set of OPNFV related validation criteria -that will provide input for the evaluation of the use of OPNFV trademarks. -This progress is under the guideline of Compliance Verification Program(CVP). - -Running methods -^^^^^^^^^^^^^^^ - -Two running methods are provided after refstack-client integrated into -Functest, Functest command line and manually, respectively. - -By default, for Defcore test cases run by Functest command line, -are run followed with automatically generated -configuration file, i.e., refstack_tempest.conf. In some circumstances, -the automatic configuration file may not quite satisfied with the SUT, -Functest also inherits the refstack-client command line and provides a way -for users to set its configuration file according to its own SUT manually. - -*command line* - -Inside the Functest container, first to prepare Functest environment: - -:: - - functest env prepare - -then to run default defcore testcases by using refstack-client: - -:: - - functest testcase run refstack_defcore - -In OPNFV Continuous Integration(CI) system, the command line method is used. - -*manually* - -Prepare the tempest configuration file and the testcases want to run with the SUT, -run the testcases with: - -:: - - ./refstack-client test -c <Path of the tempest configuration file to use> -v --test-list <Path or URL of test list> - -using help for more information: - -:: - - ./refstack-client --help - ./refstack-client test --help - -Reference tempest configuration -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -*command line method* - -When command line method is used, the default tempest configuration file -is generated by Rally. - -*manually* - -When running manually is used, recommended way to generate tempest configuration -file is: - -:: - - cd /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/refstack_client - python tempest_conf.py - -a file called tempest.conf is stored in the current path by default, users can do -some adjustment according to the SUT: - -:: - - vim refstack_tempest.conf - -a reference article can be used `[15]`_. - - snaps_smoke ------------ This test case contains tests that setup and destroy environments with VMs with and without Floating IPs with a newly created user and project. Set the config value snaps.use_floating_ips (True|False) to toggle this functionality. -Please note that When the configuration value of snaps.use_keystone is True, Functest must have access -the cloud's private network. -This suite consists in 38 tests (test duration < 10 minutes) +Please note that When the configuration value of snaps.use_keystone is True, +Functest must have access the cloud's private network. +This suite consists in 120 tests (test duration ~= 50 minutes) SDN Controllers @@ -325,31 +251,40 @@ OpenDaylight and Neutron. The list of tests can be described as follows: * Basic Restconf test cases + * Connect to Restconf URL * Check the HTTP code status * Neutron Reachability test cases + * Get the complete list of neutron resources (networks, subnets, ports) * Neutron Network test cases + * Check OpenStack networks * Check OpenDaylight networks - * Create a new network via OpenStack and check the HTTP status code returned by Neutron + * Create a new network via OpenStack and check the HTTP status code returned + by Neutron * Check that the network has also been successfully created in OpenDaylight * Neutron Subnet test cases + * Check OpenStack subnets * Check OpenDaylight subnets - * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron + * Create a new subnet via OpenStack and check the HTTP status code returned + by Neutron * Check that the subnet has also been successfully created in OpenDaylight * Neutron Port test cases + * Check OpenStack Neutron for known ports * Check OpenDaylight ports - * Create a new port via OpenStack and check the HTTP status code returned by Neutron + * Create a new port via OpenStack and check the HTTP status code returned by + Neutron * Check that the new port has also been successfully created in OpenDaylight * Delete operations + * Delete the port previously created via OpenStack * Check that the port has been also successfully deleted in OpenDaylight * Delete previously subnet created via OpenStack @@ -364,7 +299,7 @@ code returned by OpenDaylight. Features -------- -Functest has been supporting several feature projects since Brahpamutra: +Functest has been supporting several feature projects since Brahmaputra: +-----------------+---------+----------+--------+-----------+ @@ -414,9 +349,9 @@ The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an architectural framework for delivering IP multimedia services. vIMS has been integrated in Functest to demonstrate the capability to deploy a -relatively complex NFV scenario on the OPNFV platform. The deployment of a complete -functional VNF allows the test of most of the essential functions needed for a -NFV platform. +relatively complex NFV scenario on the OPNFV platform. The deployment of a +complete functional VNF allows the test of most of the essential functions +needed for a NFV platform. The goal of this test suite consists of: @@ -438,12 +373,14 @@ This testcase extends the cloudify_ims test case. The first part is similar but the testing part is different. The testing part consists in automating a realistic signaling load on the vIMS using an Ixia loader (proprietary tools) - - You need to have access to an Ixia licence server defined in the configuration - file and have ixia image locally. + + - You need to have access to an Ixia licence server defined in the + configuration file and have ixia image locally. This test case is available but not declared in testcases.yaml. The declaration -of the testcase is simple, connect to your functest-vnf docker, add the following -section in /usr/lib/python2.7/site-packacges/functest/ci/testcases.yaml:: +of the testcase is simple, connect to your functest-vnf docker, add the +following section in +/usr/lib/python2.7/site-packacges/functest/ci/testcases.yaml:: - case_name: cloudify_ims_perf @@ -460,16 +397,6 @@ section in /usr/lib/python2.7/site-packacges/functest/ci/testcases.yaml:: module: 'functest.opnfv_tests.vnf.ims.cloudify_ims_perf' class: 'CloudifyImsPerf' -orchestra_openims -^^^^^^^^^^^^^^^^^ -Orchestra test case deals with the deployment of OpenIMS with OpenBaton -orchestrator. - -orchestra_clearwaterims -^^^^^^^^^^^^^^^^^^^^^^^ -Orchestra test case deals with the deployment of Clearwater vIMS with OpenBaton -orchestrator. - vyos-vrouter ^^^^^^^^^^^^ This test case deals with the deployment and the test of vyos vrouter with @@ -489,14 +416,16 @@ The Workflow is as follows: The vyos-vrouter architecture is described in `[14]`_ +juju_epc +^^^^^^^^ .. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html .. _`[3]`: https://rally.readthedocs.org/en/latest/index.html .. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml .. _`[8]`: https://github.com/openstack/refstack-client +.. _`[9]`: https://github.com/openstack/defcore .. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst .. _`[11]`: http://robotframework.org/ .. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html .. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter -.. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/ |