aboutsummaryrefslogtreecommitdiffstats
path: root/docs/userguide
diff options
context:
space:
mode:
Diffstat (limited to 'docs/userguide')
-rw-r--r--docs/userguide/description.rst75
-rw-r--r--docs/userguide/index.rst247
-rw-r--r--docs/userguide/runfunctest.rst140
3 files changed, 232 insertions, 230 deletions
diff --git a/docs/userguide/description.rst b/docs/userguide/description.rst
new file mode 100644
index 00000000..a8fdcd0f
--- /dev/null
+++ b/docs/userguide/description.rst
@@ -0,0 +1,75 @@
+Description of the test cases
+=============================
+
+Functest is an OPNFV project dedicated to functional testing.
+In the continuous integration, it is launched after an OPNFV fresh installation.
+The Functest target is to verify the basic functions of the infrastructure.
+
+Functest includes different test suites which several test cases within.
+Test cases are developed in Functest and in feature projects.
+
+The current list of test suites can be distributed in 3 main domains::
+
+ +----------------+----------------+--------------------------------------------+
+ | Domain | Test suite | Comments |
+ +================+================+============================================+
+ | | vPing | NFV "Hello World" |
+ | +----------------+--------------------------------------------+
+ | VIM | vPing_userdata | Ping using userdata and cloud-init |
+ | | | mechanism |
+ | +----------------+--------------------------------------------+
+ |(Virtualised | Tempest | OpenStack reference test suite `[2]`_ |
+ | Infrastructure +----------------+--------------------------------------------+
+ | Manager) | Rally scenario | OpenStack testing tool testing OpenStack |
+ | | | modules `[3]`_ |
+ +----------------+----------------+--------------------------------------------+
+ | | OpenDaylight | Opendaylight Test suite |
+ | +----------------+--------------------------------------------+
+ | Controllers | ONOS | Test suite of ONOS L2 and L3 functions |
+ | +----------------+--------------------------------------------+
+ | | OpenContrail | |
+ +----------------+----------------+--------------------------------------------+
+ | Features | vIMS | Show the capability to deploy a real NFV |
+ | | | test cases. |
+ | | | The IP Multimedia Subsytem is a typical |
+ | | | Telco test case, referenced by ETSI. |
+ | | | It provides a fully functional VoIP System.|
+ | +----------------+--------------------------------------------+
+ | | Promise | Resource reservation and management project|
+ | | | to identify NFV related requirements and |
+ | | | realize resource reservation for future |
+ | | | usage by capacity management of resource |
+ | | | pools regarding compute, network and |
+ | | | storage. |
+ | +----------------+--------------------------------------------+
+ | | SDNVPN | |
+ +----------------+----------------+--------------------------------------------+
+
+
+Most of the test suites are developed upstream.
+For example, `Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the
+OpenStack integration test suite.
+Functest is in charge of the integration of different functional test suites.
+
+The Tempest suite has been customized but no new test cases have been created.
+Some OPNFV feature projects (e.g. SDNVPN) have created Tempest tests cases and
+pushed to upstream.
+
+The tests run from CI are pushed into a database.
+The goal is to populate the database with results and to show them on a Test
+Dashboard.
+
+There is no real notion of Test domain or Test coverage yet.
+Basic components (VIM, controllers) are tested through their own suites.
+Feature projects also provide their own test suites.
+
+vIMS test case was integrated to demonstrate the capability to deploy a
+relatively complex NFV scenario on top of the OPNFV infrastructure.
+
+Functest considers OPNFV as a black box.
+OPNFV, since Brahmaputra, offers lots of possible combinations:
+
+ * 3 controllers (OpenDayligh, ONOS, OpenContrail)
+ * 4 installers (Apex, Compass, Fuel, Joid)
+
+However most of the tests shall be runnable on any configuration.
diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst
index ba20507b..cf36d9e8 100644
--- a/docs/userguide/index.rst
+++ b/docs/userguide/index.rst
@@ -17,81 +17,7 @@ A presentation has been created for the first OPNFV Summit `[4]`_.
It is assumed that Functest container has been properly installed `[1]`_.
-
-Description of the test cases
-=============================
-
-Functest is an OPNFV project dedicated to functional testing.
-In the continuous integration, it is launched after an OPNFV fresh installation.
-The Functest target is to verify the basic functions of the infrastructure.
-
-Functest includes different test suites which several test cases within.
-Test cases are developed in Functest and in feature projects.
-
-The current list of test suites can be distributed in 3 main domains::
-
- +----------------+----------------+--------------------------------------------+
- | Method | Test suite | Comments |
- +================+================+============================================+
- | | vPing | NFV "Hello World" |
- | +----------------+--------------------------------------------+
- | VIM | vPing_userdata | Ping using userdata and cloud-init |
- | | | mechanism |
- | +----------------+--------------------------------------------+
- |(Virtualised | Tempest | OpenStack reference test suite `[2]`_ |
- | Infrastructure +----------------+--------------------------------------------+
- | Manager) | Rally scenario | OpenStack testing tool testing OpenStack |
- | | | modules `[3]`_ |
- +----------------+----------------+--------------------------------------------+
- | | OpenDaylight | Opendaylight Test suite |
- | +----------------+--------------------------------------------+
- | Controllers | ONOS | Test suite of ONOS L2 and L3 functions |
- | +----------------+--------------------------------------------+
- | | OpenContrail | |
- +----------------+----------------+--------------------------------------------+
- | Features | vIMS | Show the capability to deploy a real NFV |
- | | | test cases. |
- | | | The IP Multimedia Subsytem is a typical |
- | | | Telco test case, referenced by ETSI. |
- | | | It provides a fully functional VoIP System.|
- | +----------------+--------------------------------------------+
- | | Promise | Resource reservation and management project|
- | | | to identify NFV related requirements and |
- | | | realize resource reservation for future |
- | | | usage by capacity management of resource |
- | | | pools regarding compute, network and |
- | | | storage. |
- | +----------------+--------------------------------------------+
- | | SDNVPN | |
- +----------------+----------------+--------------------------------------------+
-
-
-Most of the test suites are developed upstream.
-For example, Tempest `[2]`_ is the OpenStack integration test suite.
-Functest is in charge of the integration of different functional test suites.
-
-The Tempest suite has been customized but no new test cases have been created.
-Some OPNFV feature projects (.e.g. SDNVPN) have created Tempest tests cases and
-pushed to upstream.
-
-The tests run from CI are pushed into a database.
-The goal is to populate the database with results and to show them on a Test
-Dashboard.
-
-There is no real notion of Test domain or Test coverage yet.
-Basic components (VIM, controllers) are tested through their own suites.
-Feature projects also provide their own test suites.
-
-vIMS test case was integrated to demonstrate the capability to deploy a
-relatively complex NFV scenario on top of the OPNFV infrastructure.
-
-Functest considers OPNFV as a black box.
-OPNFV, since Brahmaputra, offers lots of possible combinations:
-
- * 3 controllers (OpenDayligh, ONOS, OpenContrail)
- * 4 installers (Apex, Compass, Fuel, Joid)
-
-However most of the tests shall be runnable on any configuration.
+.. include:: ./description.rst
The different scenarios are described in the section hereafter.
@@ -177,7 +103,7 @@ Tempest has batteries of tests for:
* OpenStack API validation
* Scenarios
- * other specific tests useful in validating an OpenStack deployment
+ * Other specific tests useful in validating an OpenStack deployment
We use Rally `[3]`_ to run Tempest suite.
Rally generates automatically tempest.conf configuration file.
@@ -187,10 +113,10 @@ When the Tempest suite is run, each test duration is measured.
The full console output is stored in the tempest.log file.
As an addition of Arno, Brahmaputra runs a customized set of Tempest test cases.
-The list is specificed through --tests-file when running Rally.
+The list is specificed through *--tests-file* when running Rally.
This option has been introduced in Rally in version 0.1.2.
-The customized test list is available in the Functest repo `[4]`_
+The customized test list is available in the Functest repo `[4]`_.
This list contains more than 200 Tempest test cases.
The list can be divied into two main parts:
@@ -416,145 +342,7 @@ include::
flavor_ram: 512
flavor_disk: 0
-
-Manual testing
-==============
-
-Once the Functest docker container is running and Functest environment ready
-(through /home/opnfv/repos/functest/docker/prepare_env.sh script), the system is
-ready to run the tests.
-
-The script run_tests.sh is located in $repos_dir/functest/docker and it has
-several options::
-
- ./run_tests.sh -h
- Script to trigger the tests automatically.
-
- usage:
- bash run_tests.sh [--offline] [-h|--help] [-t <test_name>]
-
- where:
- -h|--help show this help text
- -r|--report push results to database (false by default)
- -n|--no-clean do not clean up OpenStack resources after test run
- -t|--test run specific set of tests
- <test_name> one or more of the following: vping,vping_userdata,odl,rally,tempest,vims,onos,promise. Separated by comma.
-
- examples:
- run_tests.sh
- run_tests.sh --test vping,odl
- run_tests.sh -t tempest,rally --no-clean
-
-The -r option is used by the Continuous Integration in order to push the test
-results into a test collection database, see in next section for details.
-In manual mode, you must not use it, your try will be anyway probably rejected
-as your POD must be declared in the database to collect the data.
-
-The -n option is used for preserving all the existing OpenStack resources after
-execution test cases.
-
-The -t option can be used to specify the list of test you want to launch, by
-default Functest will try to launch all its test suites in the following order
-vPing, odl, Tempest, vIMS, Rally.
-You may launch only one single test by using -t <the test you want to launch>
-
-Within Tempest test suite you can define which test cases you want to execute in
-your environment by editing test_list.txt file before executing run_tests.sh
-script.
-
-Please note that Functest includes cleaning mechanism in order to remove
-everything except what was present after a fresh install.
-If you create your own VMs, tenants, networks etc. and then launch Functest,
-they all will be deleted after executing the tests. Use --no-clean option with
-run_test.sh in order to preserve all the existing resources.
-However, be aware that Tempest and Rally create of lot of resources (users,
-tenants, networks, volumes etc.) that are not always properly cleaned, so this
-cleaning function has been set to keep the system as clean as possible after a
-full Functest run.
-
-You may also add you own test by adding a section into the function run_test()
-
-
-Automated testing
-=================
-
-As mentioned in `[1]`, the prepare-env.sh and run_test.sh can be executed within
-the container from jenkins.
-2 jobs have been created, one to run all the test and one that allows testing
-test suite by test suite.
-You thus just have to launch the acurate jenkins job on the target lab, all the
-tests shall be automatically run.
-
-When the tests are automatically started from CI, a basic algorithm has been
-created in order to detect whether the test is runnable or not on the given
-scenario.
-In fact, one of the most challenging task in Brahmaputra consists in dealing
-with lots of scenario and installers.
-Functest test suites cannot be systematically run (e.g. run the ODL suite on an
-ONOS scenario).
-
-CI provides several information:
-
- * The installer (apex|compass|fuel|joid)
- * The scenario [controller]-[feature]-[mode] with
-
- * controller = (odl|onos|ocl|nosdn)
- * feature = (ovs(dpdk)|kvm)
- * mode = (ha|noha)
-
-Constraints per test case are defined in the Functest configuration file
-/home/opnfv/functest/config/config_functest.yaml::
-
- test-dependencies:
- functest:
- vims:
- scenario: '(ocl)|(odl)|(nosdn)'
- vping:
- vping_userdata:
- scenario: '(ocl)|(odl)|(nosdn)'
- tempest:
- rally:
- odl:
- scenario: 'odl'
- onos:
- scenario: 'onos'
- ....
-
-At the end of the Functest environment creation (prepare_env.sh see `[1]`_), a
-file (/home/opnfv/functest/conf/testcase-list.txt) is created with the list of
-all the runnable tests.
-We consider the static constraints as regex and compare them with the scenario.
-For instance, odl can be run only on scenario including odl in its name.
-
-The order of execution is also described in the Functest configuration file::
-
- test_exec_priority:
-
- 1: vping
- 2: vping_userdata
- 3: tempest
- 4: odl
- 5: onos
- 6: ovno
- #7: doctor
- 8: promise
- 9: odl-vpnservice
- 10: bgpvpn
- #11: openstack-neutron-bgpvpn-api-extension-tests
- 12: vims
- 13: rally
-
-The tests are executed as follow:
-
- * Basic scenario (vPing, vPing_userdata, Tempest)
- * Controller suites: ODL or ONOS or OpenContrail
- * Feature projects
- * vIMS
- * Rally (benchmark scenario)
-
-At the end of an automated execution, everything is cleaned.
-We keep only the users/networks that have been statically declared in '[9]'_
-
+.. include:: ./runfunctest.rst
Test results
============
@@ -733,7 +521,7 @@ The results of ODL tests can be seen in the console::
* log.html
* report.html
- ODL result page
+**ODL result page**
.. figure:: ../images/functestODL.png
:width: 170mm
@@ -744,8 +532,8 @@ The results of ODL tests can be seen in the console::
ONOS
^^^^
-The ONOS test logs can be found in OnosSystemTest/TestON/logs
-(ONOSCI_PATH to be added),and also can be seen in the console::
+The ONOS test logs can be found in OnosSystemTest/, and TestON/, and logs/
+(ONOSCI_PATH to be added), and can also be seen in the console::
******************************
Result summary for Testcase4
@@ -902,7 +690,7 @@ steps:
* INFO: environment prepared successfully => environment OK
* INFO - Cloudify-manager server is UP ! => orchestrator deployed
- * INFO The deployment of clearwater-opnfv is ended => VNF deployed
+ * INFO - The deployment of clearwater-opnfv is ended => VNF deployed
* Multiple Identities (UDP) - (6505550771, 6505550675) Passed => tests run
* DEBUG - Pushing results to DB.... => tests saved
@@ -1035,7 +823,7 @@ But all were not ready for Brahmaputra.
The API can described as follow:
-**Version:**::
+Version:
+--------+--------------------------+------------------------------------------+
| Method | Path | Description |
@@ -1044,7 +832,7 @@ The API can described as follow:
+--------+--------------------------+------------------------------------------+
-**Pods:**::
+Pods:
+--------+--------------------------+------------------------------------------+
| Method | Path | Description |
@@ -1059,7 +847,7 @@ The API can described as follow:
| | | } |
+--------+--------------------------+------------------------------------------+
-**Projects:**::
+Projects:
+--------+--------------------------+------------------------------------------+
| Method | Path | Description |
@@ -1087,7 +875,7 @@ The API can described as follow:
+--------+--------------------------+------------------------------------------+
-**Test cases:**::
+Test cases:
+--------+--------------------------+------------------------------------------+
| Method | Path | Description |
@@ -1105,7 +893,7 @@ The API can described as follow:
| | | } |
+--------+--------------------------+------------------------------------------+
| PUT | /test_projects/{project}?| Modify a test case of {project} |
- | | case_name={case} | |
+ | | case_name={case} | |
| | | Content-Type: application/json |
| | | { |
| | | <the field(s) you want to modify> |
@@ -1113,9 +901,9 @@ The API can described as follow:
+--------+--------------------------+------------------------------------------+
| DELETE | /test_projects/{project}/| Delete a test case |
| | case_name={case} | |
- +----------------+----------------+--------------------------------------------+
+ +--------+--------------------------+------------------------------------------+
-**Test Results:**::
+Test Results:
+--------+--------------------------+------------------------------------------+
| Method | Path | Description |
@@ -1155,7 +943,7 @@ The API can described as follow:
+--------+--------------------------+------------------------------------------+
-**Dashboard:**::
+Dashboard:
+--------+--------------------------+------------------------------------------+
| Method | Path | Description |
@@ -1292,4 +1080,3 @@ IRC support chan: #opnfv-testperf
.. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
.. _`config_test.py` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.py
.. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml
-
diff --git a/docs/userguide/runfunctest.rst b/docs/userguide/runfunctest.rst
new file mode 100644
index 00000000..8125e814
--- /dev/null
+++ b/docs/userguide/runfunctest.rst
@@ -0,0 +1,140 @@
+Executing the functest suites
+=============================
+
+Manual testing
+--------------
+
+Once the Functest docker container is running and Functest environment ready
+(through /home/opnfv/repos/functest/docker/prepare_env.sh script), the system is
+ready to run the tests.
+
+The script *run_tests.sh* is located in $repos_dir/functest/docker and it has
+several options::
+
+ ./run_tests.sh -h
+ Script to trigger the tests automatically.
+
+ usage:
+ bash run_tests.sh [--offline] [-h|--help] [-t <test_name>]
+
+ where:
+ -h|--help show this help text
+ -r|--report push results to database (false by default)
+ -n|--no-clean do not clean up OpenStack resources after test run
+ -t|--test run specific set of tests
+ <test_name> one or more of the following: vping,vping_userdata,odl,rally,tempest,vims,onos,promise. Separated by comma.
+
+ examples:
+ run_tests.sh
+ run_tests.sh --test vping,odl
+ run_tests.sh -t tempest,rally --no-clean
+
+The *-r* option is used by the Continuous Integration in order to push the test
+results into a test collection database, see in next section for details.
+In manual mode, you must not use it, your try will be anyway probably rejected
+as your POD must be declared in the database to collect the data.
+
+The *-n* option is used for preserving all the existing OpenStack resources after
+execution test cases.
+
+The *-t* option can be used to specify the list of test you want to launch, by
+default Functest will try to launch all its test suites in the following order
+vPing, odl, Tempest, vIMS, Rally.
+You may launch only one single test by using *-t <the test you want to launch>*.
+
+Within Tempest test suite you can define which test cases you want to execute in
+your environment by editing test_list.txt file before executing *run_tests.sh*
+script.
+
+Please note that Functest includes cleaning mechanism in order to remove
+everything except what was present after a fresh install.
+If you create your own VMs, tenants, networks etc. and then launch Functest,
+they all will be deleted after executing the tests. Use the *--no-clean* option with
+run_test.sh in order to preserve all the existing resources.
+However, be aware that Tempest and Rally create of lot of resources (users,
+tenants, networks, volumes etc.) that are not always properly cleaned, so this
+cleaning function has been set to keep the system as clean as possible after a
+full Functest run.
+
+You may also add you own test by adding a section into the function run_test().
+
+
+Automated testing
+-----------------
+
+As mentioned in `[1]`, the *prepare-env.sh* and *run_test.sh* can be executed within
+the container from jenkins.
+2 jobs have been created, one to run all the test and one that allows testing
+test suite by test suite.
+You thus just have to launch the acurate jenkins job on the target lab, all the
+tests shall be automatically run.
+
+When the tests are automatically started from CI, a basic algorithm has been
+created in order to detect whether the test is runnable or not on the given
+scenario.
+In fact, one of the most challenging task in Brahmaputra consists in dealing
+with lots of scenario and installers.
+Functest test suites cannot be systematically run (e.g. run the ODL suite on an
+ONOS scenario).
+
+CI provides several information:
+
+ * The installer (apex|compass|fuel|joid)
+ * The scenario [controller]-[feature]-[mode] with
+
+ * controller = (odl|onos|ocl|nosdn)
+ * feature = (ovs(dpdk)|kvm)
+ * mode = (ha|noha)
+
+Constraints per test case are defined in the Functest configuration file
+/home/opnfv/functest/config/config_functest.yaml::
+
+ test-dependencies:
+ functest:
+ vims:
+ scenario: '(ocl)|(odl)|(nosdn)'
+ vping:
+ vping_userdata:
+ scenario: '(ocl)|(odl)|(nosdn)'
+ tempest:
+ rally:
+ odl:
+ scenario: 'odl'
+ onos:
+ scenario: 'onos'
+ ....
+
+At the end of the Functest environment creation (prepare_env.sh see `[1]`_), a
+file (/home/opnfv/functest/conf/testcase-list.txt) is created with the list of
+all the runnable tests.
+We consider the static constraints as regex and compare them with the scenario.
+For instance, odl can be run only on scenario including odl in its name.
+
+The order of execution is also described in the Functest configuration file::
+
+ test_exec_priority:
+
+ 1: vping
+ 2: vping_userdata
+ 3: tempest
+ 4: odl
+ 5: onos
+ 6: ovno
+ 7: doctor
+ 8: promise
+ 9: odl-vpnservice
+ 10: bgpvpn
+ 11: openstack-neutron-bgpvpn-api-extension-tests
+ 12: vims
+ 13: rally
+
+The tests are executed as follow:
+
+ * Basic scenario (vPing, vPing_userdata, Tempest)
+ * Controller suites: ODL or ONOS or OpenContrail
+ * Feature projects
+ * vIMS
+ * Rally (benchmark scenario)
+
+At the end of an automated execution, everything is cleaned.
+We keep only the users/networks that have been statically declared in 'https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/libraries/os_defaults.yaml'_