diff options
| -rw-r--r-- | docs/images/concepts_mapping_final.png | bin | 203419 -> 145191 bytes | |||
| -rw-r--r-- | docs/images/functest-reporting-status.png | bin | 0 -> 22859 bytes | |||
| -rw-r--r-- | docs/userguide/index.rst | 198 | ||||
| -rw-r--r-- | docs/userguide/introduction.rst | 46 | ||||
| -rw-r--r-- | docs/userguide/runfunctest.rst | 149 | ||||
| -rw-r--r-- | docs/userguide/troubleshooting.rst | 40 |
6 files changed, 248 insertions, 185 deletions
diff --git a/docs/images/concepts_mapping_final.png b/docs/images/concepts_mapping_final.png Binary files differindex 7a51f1117..5ad0fc569 100644 --- a/docs/images/concepts_mapping_final.png +++ b/docs/images/concepts_mapping_final.png diff --git a/docs/images/functest-reporting-status.png b/docs/images/functest-reporting-status.png Binary files differnew file mode 100644 index 000000000..9e230df57 --- /dev/null +++ b/docs/images/functest-reporting-status.png diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst index 327a1756b..5a584f5fa 100644 --- a/docs/userguide/index.rst +++ b/docs/userguide/index.rst @@ -90,7 +90,7 @@ Tenant network:: | | to VM2 | | | +------------------->| | | | | | - | | Stablish SSH | | + | | Establish SSH | | | | connection to VM2 | | | | through floating IP| | | +------------------->| | @@ -238,42 +238,41 @@ OpenDaylight and Neutron. The list of tests can be described as follows: - * Restconf.basic: Get the controller modules via Restconf - * Neutron.Networks + * Basic Restconf test cases + * Connect to Restconf URL + * Check the HTTP code status - * Check OpenStack Networks :: Checking OpenStack Neutron for known networks - * Check OpenDaylight Networks :: Checking OpenDaylight Neutron API - * Create Network :: Create new network in OpenStack - * Check Network :: Check Network created in OpenDaylight - * Neutron.Networks :: Checking Network created in OpenStack are pushed + * Neutron Reachability test cases + * Get the complete list of neutron resources (networks, subnets, ports) - * Neutron.Subnets + * Neutron Network test cases + * Check OpenStack networks + * Check OpenDaylight networks + * Create a new network via OpenStack and check the HTTP status code returned by Neutron + * Check that the network has also been successfully created in OpenDaylight - * Check OpenStack Subnets :: Checking OpenStack Neutron for known Subnets - * Check OpenDaylight subnets :: Checking OpenDaylight Neutron API - * Create New subnet :: Create new subnet in OpenStack - * Check New subnet :: Check new subnet created in OpenDaylight - * Neutron.Subnets :: Checking Subnets created in OpenStack are pushed + * Neutron Subnet test cases + * Check OpenStack subnets + * Check OpenDaylight subnets + * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron + * Check that the subnet has also been successfully created in OpenDaylight - * Neutron.Ports + * Neutron Port test cases + * Check OpenStack Neutron for known ports + * Check OpenDaylight ports + * Create a new port via OpenStack and check the HTTP status code returned by Neutron + * Check that the new port has also been successfully created in OpenDaylight - * Check OpenStack ports :: Checking OpenStack Neutron for known ports - * Check OpenDaylight ports :: Checking OpenDaylight Neutron API - * Create New Port :: Create new port in OpenStack - * Check New Port :: Check new subnet created in OpenDaylight - * Neutron.Ports :: Checking Port created in OpenStack are pushed + * Delete operations + * Delete the port previously created via OpenStack + * Check that the port has been also succesfully deleted in OpenDaylight + * Delete previously subnet created via OpenStack + * Check that the subnet has also been successfully deleted in OpenDaylight + * Delete the network created via OpenStack + * Check that the network has also been succesfully deleted in OpenDaylight - * Delete Ports - - * Delete previously created subnet in OpenStack - * Check subnet deleted in OpenDaylight - * Check subnet deleted in OpenStack - - * Delete network - - * Delete previously created network in OpenStack - * Check network deleted in OpenDaylight - * Check network deleted in OpenStack +Note: the checks in OpenDaylight are based on the returned HTTP status +code returned by OpenDaylight. ONOS @@ -320,70 +319,25 @@ The test cases are described as follows: * Delete External Gateway: Delete the External Gateway and check that it is 'NULL' in ONOS -Open Contrail -^^^^^^^^^^^^^ -**TODO:** - - Features -------- -Doctor -^^^^^^ - -**TODO:** - - - -Promise -^^^^^^^ - -Promise provides a basic set of test cases as part of the deliverable. - -The available 33 test cases can be grouped into 7 test suites: - - #. Add a new OpenStack provider into resource pool: Registers - OpenStack into a new resource pool and adds more capacity associated - with this pool. - - #. Allocation without reservation: Creates a new server in OpenStack - and adds a new allocation record in Promise shim-layer. +Most of the features have been developped by feature projects. +Security_scan has been initiated in Functest repository but should soon +be declared in its own repository as well. - #. Allocation using reservation for immediate use: Creates a resource - reservation record with no start/end time and immediately creates a new - server in OpenStack and add a new allocation record in Promise - shim-layer. +Please refer to the dedicated feature user guides for details: - #. Reservation for future use: Creates a resource reservation record - for a future start time, queries, modifies and cancels the newly created - reservation. - - #. Capacity planning: Decreases and increases the available capacity - from a provider in the future and queries the available collections and - utilizations. - - #. Reservation with conflict: Tries to create reservations for - immediate and future use with conflict. - - #. Cleanup test allocations: Destroys all allocations in OpenStack. - - -bgpvpn -^^^^^^ -Many telecom network functions are relying on layer-3 infrastructure services, -within a VNF between components, or towards existing external networks. -In many cases, these external networks are implemented in MPLS/BGP technology in -existing service provider wide-area-networks (WAN). This proven technology -provides a good mechanism for inter-operation of a NFV Infrastructure (NFVI) -and WAN. - -The SDNVPN project defined a 'bgpvpn' test suite. -This 'bgpvpn' test suite deals with 2 Tempest cases dedicated to the test of -the OpenStack 'bgpvpn' API: - - * test_create_bgpvpn - * test_create_bgpvpn_as_non_admin_fail + * bgpvpn: ** TODO link ** + * copper: ** TODO link ** + * doctor: ** TODO link ** + * domino: ** TODO link ** + * moon: ** TODO link ** + * multisites: ** TODO link ** + * onos-sfc: ** TODO link ** + * odl-sfc: ** TODO link ** + * promise: ** TODO link ** security_scan ^^^^^^^^^^^^^ @@ -418,6 +372,8 @@ The current work flow is as follows: At present, only the Apex installer is supported, with support for other installers due within D-release. + + VNF --- @@ -446,16 +402,71 @@ The Clearwater architecture is described as follows: :alt: vIMS architecture +parser +^^^^^^ + +See parser user guide for details: ** TODO link ** + + .. include:: ./runfunctest.rst Test results ============ -Note that the results are documented per scenario basis. Although most of the test -cases might show the same output, some of them are not supported by -certain scenarios. Please select the appropriate scenario and compare the results -to that referenced in the documentation. +Manual testing +-------------- + +In manual mode test results are displayed in the console and result files +are put in /home/opnfv/functest/results. + +Automated testing +-------------- + +In automated mode, test results are displayed in jenkins logs, a summary is provided +at the end of the job and can be described as follow:: + + +==================================================================================================================================================+ + | FUNCTEST REPORT | + +==================================================================================================================================================+ + | | + | Deployment description: | + | INSTALLER: fuel | + | SCENARIO: os-odl_l2-nofeature-ha | + | BUILD TAG: jenkins-functest-fuel-baremetal-daily-master-324 | + | CI LOOP: daily | + | | + +=========================+===============+============+===============+===========================================================================+ + | TEST CASE | TIER | DURATION | RESULT | URL | + +=========================+===============+============+===============+===========================================================================+ + | healthcheck | healthcheck | 03:07 | PASS | | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | vping_ssh | smoke | 00:56 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac13d79377c54b278bd4c1 | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | vping_userdata | smoke | 00:41 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac14019377c54b278bd4c2 | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | tempest_smoke_serial | smoke | 16:05 | FAIL | http://testresults.opnfv.org/test/api/v1/results/57ac17ca9377c54b278bd4c3 | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | rally_sanity | smoke | 12:19 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1aad9377c54b278bd4cd | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | odl | sdn_suites | 00:24 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1ad09377c54b278bd4ce | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | promise | features | 00:41 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1ae59377c54b278bd4cf | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + +Results are automatically pushed to the test results database, some additional +result files are pushed to OPNFV artifact web sites. + +Based on the results stored in the result database, a `Functest reporting`_ +portal is also automatically updated. This portal provides information on: + + * The overall status per scenario and per installer + * Tempest: Tempest test case including reported errors per scenario and installer + * vIMS: vIMS details per scenario and installer + +.. figure:: ../images/functest-reporting-status.png + :align: center + :alt: Functest reporting portal Fuel status page Test Dashboard @@ -492,3 +503,4 @@ IRC support chan: #opnfv-testperf .. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html .. _`config_test.py` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.py .. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml +.. _`Functest reporting`: http://testresults.opnfv.org/reporting/functest/release/master/index-status-fuel.html diff --git a/docs/userguide/introduction.rst b/docs/userguide/introduction.rst index 8624bee83..19a7ead1b 100644 --- a/docs/userguide/introduction.rst +++ b/docs/userguide/introduction.rst @@ -58,7 +58,11 @@ Features and VNF (Virtual Network Functions). | | | | See the Rally documents `[3]`_. | +-------------+---------------+----------------+----------------------------------+ | Controllers | sdn_suites | odl | Opendaylight Test suite | -| | | | TODO: Find a document reference! | +| | | | Limited test suite to check the | +| | | | basic neutron (Layer 2) | +| | | | operations mainly based on | +| | | | upstream testcases. See below | +| | | | for details | | | +----------------+----------------------------------+ | | | onos | Test suite of ONOS L2 and L3 | | | | | functions. | @@ -81,7 +85,8 @@ Features and VNF (Virtual Network Functions). | | | | features: | | | | | * Immediate Notification | | | | | * Consistent resource state | -| | | | awareness (compute). See the | +| | | | awareness (compute) | +| | | | * Get Valid Server state | | | | | See `Doctor User Guide`_ for | | | | | details | | | +----------------+----------------------------------+ @@ -96,8 +101,28 @@ Features and VNF (Virtual Network Functions). | | | | security scan. (Currently | | | | | available only for the Apex | | | | | installer environment) | -| | | | TODO: Add document link from | -| | | | Luke Hinds; when received. | +| | +----------------+----------------------------------+ +| | | onos-sfc | SFC testing for onos scenarios | +| | | | TODO See for details | +| | +----------------+----------------------------------+ +| | | odl-sfc | SFC testing for odl scenarios | +| | | | TODO See for details | +| | +----------------+----------------------------------+ +| | | domino | Service template publication | +| | | | service based on OASIS | +| | | | specification "TOSCA Simple | +| | | | Profile for Network Function | +| | | | Virtualization (NFV) Version 1.0"| +| | | | TODO See for details | +| | +----------------+----------------------------------+ +| | | copper | Deployment policy | +| | | | TODO See for details | +| | +----------------+----------------------------------+ +| | | multisites | Multisites | +| | | | TODO See for details | +| | +----------------+----------------------------------+ +| | | moon | Security management system | +| | | | TODO See for details | +-------------+---------------+----------------+----------------------------------+ | VNF | vnf | vims | Example of a real VNF deployment | | | | | to show the NFV capabilities of | @@ -105,7 +130,10 @@ Features and VNF (Virtual Network Functions). | | | | Subsytem is a typical Telco test | | | | | case, referenced by ETSI. | | | | | It provides a fully functional | -| | | | VoIP System, | +| | | | VoIP System | ++ +---------------+----------------+----------------------------------+ +| | | parser | Service template translation | +| | | | TODO See for details | +-------------+---------------+----------------+----------------------------------+ @@ -129,8 +157,8 @@ are integrated from upstream communities or other OPNFV projects. For example, OpenStack integration test suite and Functest is in charge of the selection, integration and automation of those tests that fit suitably to OPNFV. -The Tempest test suite has been customized but no new test cases have been -created in OPNFV Functest. +The Tempest test suite is the default OpenStack smoke test suite but no new test +cases have been created in OPNFV Functest. The results produced by the tests run from CI are pushed and collected into a NoSQL database. The goal is to populate the database with results from different @@ -176,6 +204,4 @@ section `Executing the functest suites`_ of this document. .. _`Promise User Guide`: http://artifacts.opnfv.org/promise/brahmaputra/docs/userguide/index.html .. _`ONOSFW User Guide`: http://artifacts.opnfv.org/onosfw/brahmaputra/docs/userguide/index.html .. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/brahmaputra/docs/userguide/featureusage.html -.. _`Functest Dashboard`: http://testresults.opnfv.org/dashboard/ - - +.. _`Functest Dashboard`: http://testresults.opnfv.org/kibana_dashboards/ diff --git a/docs/userguide/runfunctest.rst b/docs/userguide/runfunctest.rst index 4c2e3f54b..f3da540a4 100644 --- a/docs/userguide/runfunctest.rst +++ b/docs/userguide/runfunctest.rst @@ -38,8 +38,6 @@ for the execution of Test Tiers or Test Cases:: show Shows information about a tier. root@22e436918db0:~/repos/functest/ci# functest testcase --help - and - Usage: functest testcase [OPTIONS] COMMAND [ARGS]... Options: @@ -217,7 +215,7 @@ command is used with 'functest tier':: Please note that for some scenarios some test cases might not be launched. For example, the last example displayed only the 'odl' testcase for the given -environment. In this system the deployment does not support the 'onos' SDN +environment. In this particular system the deployment does not support the 'onos' SDN Controller Test Case; for example. **Important** If you use the command 'functest tier run <tier_name>', then the @@ -264,12 +262,12 @@ Example:: : etc. -Functest includes cleaning mechanism in order to remove all the OpenStack -resources except what was present before running any test. The script +Functest includes a cleaning mechanism in order to remove all the OpenStack +resources except those present before running any test. The script *$repos_dir/functest/utils/generate_defaults.py* is called once when setting up the Functest environment (i.e. CLI command 'functest env prepare') to snapshot all the OpenStack resources (images, networks, volumes, security groups, tenants, -users) so that an eventual cleanup does not remove any of this defaults. +users) so that an eventual cleanup does not remove any of these defaults. The script **clean_openstack.py** which is located in *$repos_dir/functest/utils/* is normally called after a test execution. It is @@ -297,21 +295,23 @@ do a direct call to the desired test script. For example: Automated testing ----------------- -**TODO** Jose, the next section has not yet been revised! - As mentioned previously, the Functest Docker container preparation as well as invocation of Test Cases can be called within the container from the Jenkins CI -system. There are 2 jobs that automate the whole process. The first job runs all -the tests and the second job allows testing test suite by test suite specifying +system. There are 3 jobs that automate the whole process. The first job runs all +the tests referenced in the daily loop (i.e. that must been run daily), the second +job runs the tests referenced in the weekly loop (usually long duration tests run +once a week maximum) and the third job allows testing test suite by test suite specifying the test suite name. The user may also use either of these Jenkins jobs to execute the desired test suites. -One of the most challenging task in the Brahmaputra release consists +One of the most challenging task in the Colorado release consists in dealing with lots of scenarios and installers. Thus, when the tests are automatically started from CI, a basic algorithm has been created in order to detect whether a given test is runnable or not on the given scenario. Some Functest test suites cannot be systematically run (e.g. ODL suite can not -be run on an ONOS scenario). +be run on an ONOS scenario). Moreover since Colorado, we also introduce the +notion of daily/weekly in order to save CI time and avoid running systematically +long duration tests. CI provides some useful information passed to the container as environment variables: @@ -320,65 +320,86 @@ variables: * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with - * controller = (odl|onos|ocl|nosdn) - * feature = (ovs(dpdk)|kvm|sfc|bgpvpn) - * mode = (ha|noha) + * controller = (odl|onos|ocl|nosdn) + * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|moon|multisites) + * mode = (ha|noha) The constraints per test case are defined in the Functest configuration file -*/home/opnfv/functest/config/config_functest.yaml*:: - - test-dependencies: - functest: - vims: - scenario: '(ocl)|(odl)|(nosdn)' - vping: - vping_userdata: - scenario: '(ocl)|(odl)|(nosdn)' - tempest: - rally: - odl: - scenario: 'odl' - onos: - scenario: 'onos' +*/home/opnfv/repos/functest/ci/testcases.yaml*:: + + tiers: + - + name: healthcheck + order: 0 + ci_loop: '(daily)|(weekly)' + description : >- + First tier to be executed to verify the basic + operations in the VIM. + testcases: + - + name: healthcheck + criteria: 'status == "PASS"' + blocking: true + description: >- + This test case verifies the basic OpenStack services like + Keystone, Glance, Cinder, Neutron and Nova. + + dependencies: + installer: '' + scenario: '' + + - + name: smoke + order: 1 + ci_loop: '(daily)|(weekly)' + description : >- + Set of basic Functional tests to validate the OpenStack deployment. + testcases: + - + name: vping_ssh + criteria: 'status == "PASS"' + blocking: true + description: >- + This test case verifies: 1) SSH to an instance using floating + IPs over the public network. 2) Connectivity between 2 instances + over a private network. + dependencies: + installer: '' + scenario: '^((?!bgpvpn|odl_l3).)*$' .... -At the end of the Functest environment creation, a file -*/home/opnfv/functest/conf/testcase-list.txt* is created with the list of -all the runnable tests. -Functest considers the static constraints as regular expressions and compare them -with the given scenario name. -For instance, ODL suite can be run only on an scenario including 'odl' in its name. - -The order of execution is also described in the Functest configuration file:: - - test_exec_priority: - - 1: vping_ssh - 2: vping_userdata - 3: tempest - 4: odl - 5: onos - 6: ovno - 7: doctor - 8: promise - 9: odl-vpnservice - 10: bgpvpn - #11: openstack-neutron-bgpvpn-api-extension-tests - 12: vims - 13: rally - -The tests are executed in the following order: - - 1) vPing test cases - 2) Tempest suite - 3) SDN controller suites - 4) Feature project tests cases (Promise, Doctor, BGPVPN...) +We may distinguish 2 levels in the test case description: + * Tier level + * Test case level + +At the tier level, we define the following parameters: + + * ci_loop: indicate if in automated mode, the test case must be run in daily and/or weekly jobs + * description: a high level view of the test case + +For a given test case we defined: + * the name of the test case + * the criteria (experimental): a criteria used to declare the test case as PASS or FAIL + * blocking: if set to true, if the test is failed, the execution of the following tests is canceled + * the description of the test case + * the dependencies: a combination of 2 regex on the scenario and the installer name + +The order of execution is the one defined in the file if all test cases are selected. + +In CI daily job the tests are executed in the following order: + + 1) healthcheck (blocking) + 2) smoke: both vPings are blocking + 3) SDN controller suites (blocking) + 4) Feature project tests cases + +In CI weekly job we add 2 tiers: + 5) vIMS suite 6) Rally suite As explained before, at the end of an automated execution, the OpenStack resources might be eventually removed. +Please note that a system snapshot is taken before any test case execution. -**END of TODO** - - +This testcase.yaml file is used for CI, for the CLI and for the automatic reporting. diff --git a/docs/userguide/troubleshooting.rst b/docs/userguide/troubleshooting.rst index cd79bafcf..c23b00602 100644 --- a/docs/userguide/troubleshooting.rst +++ b/docs/userguide/troubleshooting.rst @@ -312,15 +312,19 @@ To check any possible problems with Rally, the logs are stored under Controllers ----------- -ODL -^^^ -2 versions are supported in Colorado, depending on the scenario: - * Lithium - * Berylium +Opendaylight +^^^^^^^^^^^^ + +If the Basic Restconf test suite fails, check that the ODL controller is +reachable and its Restconf module has been installed. + +If the Neutron Reachability test fails, verify that the modules +implementing Neutron requirements have been properly installed. + +If any of the other test cases fails, check that Neutron and ODL have +been correctly configured to work together. Check Neutron configuration +files, accounts, IP addresses etc.). -The upstream test suites have not been adapted, so you may get 18 or 15 tests -passed on 18 depending on your configuration. The 3 testcases are partly failed -due to wrong return code. ONOS ^^^^ @@ -331,20 +335,14 @@ Please refer to the ONOS documentation. `ONOSFW User Guide`_ . Features -------- - -Doctor -^^^^^^ -Please refer to the Doctor documentation. `Doctor User Guide`_ +Please refer to the dedicated feature user guides for details. -Promise -^^^^^^^ -Please refer to the Promise documentation. `Promise User Guide`_ +security_scan +^^^^^^^^^^^^^ +** TODO ** -bgpvpn -^^^^^^ -Please refer to the SNVPN documentation. `SDNVPN User Guide`_ NFV @@ -383,4 +381,10 @@ described in the following table: +-----------------------------------+------------------------------------+ +parser +^^^^^^ + +See parser user guide for details: ** TODO link ** + + .. _`OPNFV Functest Developer Guide`: http://artifacts.opnfv.org/functest/docs/devguide/# |
