diff options
Diffstat (limited to 'docs/results')
| -rw-r--r-- | docs/results/compass-os-nosdn-nofeature-ha.rst | 20 | ||||
| -rw-r--r-- | docs/results/compass-os-odl_l2-nofeature-ha.rst | 19 | ||||
| -rw-r--r-- | docs/results/compass-os-onos-nofeature-ha.rst | 10 | ||||
| -rw-r--r-- | docs/results/joid-os-odl_l2-nofeature-ha.rst | 18 | ||||
| -rw-r--r-- | docs/results/overview.rst | 83 |
5 files changed, 77 insertions, 73 deletions
diff --git a/docs/results/compass-os-nosdn-nofeature-ha.rst b/docs/results/compass-os-nosdn-nofeature-ha.rst index e09597f17..294cb09f7 100644 --- a/docs/results/compass-os-nosdn-nofeature-ha.rst +++ b/docs/results/compass-os-nosdn-nofeature-ha.rst @@ -4,15 +4,13 @@ Detailed test results for compass-os-nosdn-nofeature-ha ------------------------------------------------------- -The following section outlines the detailed functest results for the Brahmaputra scenario -deploying OpenStack in a Pharos environment by the Compass installer. - VIM --- vping_ssh ^^^^^^^^^ :: + FUNCTEST.info: Running vPing-SSH test... vPing_ssh- INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.4-x86_64-disk.img'... vPing_ssh- INFO - Creating neutron network vping-net... @@ -44,12 +42,12 @@ vping_ssh vPing_ssh- INFO - vPing duration:'43.2' s. vPing_ssh- INFO - Cleaning up... vPing_ssh- INFO - vPing OK -:: + vping_userdata ^^^^^^^^^^^^^^ - :: + FUNCTEST.info: Running vPing-userdata test... vPing_userdata- INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.4-x86_64-disk.img'... vPing_userdata- INFO - Creating neutron network vping-net... @@ -90,11 +88,12 @@ vping_userdata vPing_userdata- INFO - vPing OK vPing_userdata- INFO - Cleaning up... vPing_userdata- INFO - Deleting network 'vping-net'... -:: + Tempest ^^^^^^^ :: + FUNCTEST.info: Running Tempest tests... run_tempest - INFO - Creating tenant and user for Tempest suite 2016-02-23 04:37:19.865 23855 INFO rally.verification.tempest.tempest [-] Starting: Creating configuration file for Tempest. @@ -328,11 +327,12 @@ Tempest run_tempest - INFO - Results: {'timestart': '2016-02-2304:37:25.793306', 'duration': 261, 'tests': 210, 'failures': 0} run_tempest - INFO - Pushing results to DB: 'http://testresults.opnfv.org/testapi/results'. run_tempest - INFO - Deleting tenant and user for Tempest suite) -:: + Rally ^^^^^ :: + FUNCTEST.info: Running Rally benchmark suite... run_rally - INFO - Starting test scenario "authenticate" ... run_rally - INFO - @@ -1622,15 +1622,16 @@ Rally +===================+============+===============+===========+ | TOTAL: | 01:03:52 | 229 | 98.59% | +===================+============+===============+===========+ -:: + + Feature tests ------------- vIMS ^^^^ - :: + FUNCTEST.info: Running vIMS test... vIMS - INFO - Prepare OpenStack plateform (create tenant and user) vIMS - INFO - Update OpenStack creds informations @@ -1664,4 +1665,3 @@ vIMS vIMS - INFO - Cloudify-manager server has been successfully removed! vIMS - INFO - Removing vIMS tenant .. vIMS - INFO - Removing vIMS user .. -:: diff --git a/docs/results/compass-os-odl_l2-nofeature-ha.rst b/docs/results/compass-os-odl_l2-nofeature-ha.rst index 088097b2e..bed7f8646 100644 --- a/docs/results/compass-os-odl_l2-nofeature-ha.rst +++ b/docs/results/compass-os-odl_l2-nofeature-ha.rst @@ -10,6 +10,7 @@ VIM vping_ssh ^^^^^^^^^ :: + FUNCTEST.info: Running vPing-SSH test... vPing_ssh- INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.4-x86_64-disk.img'... vPing_ssh- INFO - Creating neutron network vping-net... @@ -41,12 +42,12 @@ vping_ssh vPing_ssh- INFO - vPing duration:'26.0' s. vPing_ssh- INFO - Cleaning up... vPing_ssh- INFO - vPing OK -:: + vping_userdata ^^^^^^^^^^^^^^ - :: + FUNCTEST.info: Running vPing-userdata test... vPing_userdata- INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.4-x86_64-disk.img'... vPing_userdata- INFO - Creating neutron network vping-net... @@ -87,11 +88,12 @@ vping_userdata vPing_userdata- INFO - vPing OK vPing_userdata- INFO - Cleaning up... vPing_userdata- INFO - Deleting network 'vping-net'... -:: + Tempest ^^^^^^^ :: + FUNCTEST.info: Running Tempest tests... run_tempest - INFO - Creating tenant and user for Tempest suite INFO rally.verification.tempest.tempest [-] Starting: Creating configuration file for Tempest. @@ -324,11 +326,12 @@ Tempest run_tempest - INFO - Results: {'timestart': '2016-02-2222:27:42.722872', 'duration': 171, 'tests': 210, 'failures': 1} run_tempest - INFO - Pushing results to DB: 'http://testresults.opnfv.org/testapi/results'. run_tempest - INFO - Deleting tenant and user for Tempest suite) -:: + Rally ^^^^^ :: + FUNCTEST.info: Running Rally benchmark suite... run_rally - INFO - Starting test scenario "authenticate" ... run_rally - INFO - @@ -1617,7 +1620,7 @@ Rally +===================+============+===============+===========+ | TOTAL: | 00:57:39 | 229 | 98.51% | +===================+============+===============+===========+ -:: + SDN Controller -------------- @@ -1625,6 +1628,7 @@ SDN Controller ODL ^^^ :: + ============================================================================== Basic ============================================================================== @@ -1721,15 +1725,15 @@ ODL Neutron :: Test suite for Neutron Plugin | PASS | 18 critical tests, 18 passed, 0 failed 18 tests total, 18 passed, 0 failed -:: + Feature tests ------------- vIMS ^^^^ - :: + FUNCTEST.info: Running vIMS test... vIMS - INFO - Prepare OpenStack plateform (create tenant and user) vIMS - INFO - Update OpenStack creds informations @@ -1763,4 +1767,3 @@ vIMS vIMS - INFO - Cloudify-manager server has been successfully removed! vIMS - INFO - Removing vIMS tenant .. vIMS - INFO - Removing vIMS user .. -:: diff --git a/docs/results/compass-os-onos-nofeature-ha.rst b/docs/results/compass-os-onos-nofeature-ha.rst index b9961fa0f..87103c307 100644 --- a/docs/results/compass-os-onos-nofeature-ha.rst +++ b/docs/results/compass-os-onos-nofeature-ha.rst @@ -10,6 +10,7 @@ VIM vping_ssh ^^^^^^^^^ :: + FUNCTEST.info: Running vPing-SSH test... vPing_ssh- INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.4-x86_64-disk.img'... vPing_ssh- INFO - Creating neutron network vping-net... @@ -41,11 +42,11 @@ vping_ssh vPing_ssh- INFO - vPing duration:'262.5' s. vPing_ssh- INFO - Cleaning up... vPing_ssh- INFO - vPing OK -:: Tempest ^^^^^^^ :: + FUNCTEST.info: Running Tempest tests... run_tempest - INFO - Creating tenant and user for Tempest suite 2016-02-22 16:28:55.743 23855 INFO rally.verification.tempest.tempest [-] Starting: Creating configuration file for Tempest. @@ -279,11 +280,12 @@ Tempest run_tempest - INFO - Results: {'timestart': '2016-02-2216:29:01.778060', 'duration': 1082, 'tests': 210, 'failures': 11} run_tempest - INFO - Pushing results to DB: 'http://testresults.opnfv.org/testapi/results'. run_tempest - INFO - Deleting tenant and user for Tempest suite) -:: + Rally ^^^^^ :: + FUNCTEST.info: Running Rally benchmark suite... run_rally - INFO - Starting test scenario "authenticate" ... run_rally - INFO - @@ -1581,7 +1583,7 @@ Rally +===================+============+===============+===========+ | TOTAL: | 00:55:38 | 229 | 99.15% | +===================+============+===============+===========+ -:: + SDN Controller -------------- @@ -1589,6 +1591,7 @@ SDN Controller ONOS ^^^^ :: + FUNCTEST.info: Running ONOS test case... Cloning into '/home/opnfv/repos/functest/testcases/Controllers/ONOS/Teston/CI/OnosSystemTest'... FUNCvirNetNB - INFO - Creating component Handle: ONOSrest @@ -2113,4 +2116,3 @@ ONOS Success Percentage : 100% Execution Result : 100% Disconnecting from ONOSrest: <drivers.common.api.controller.onosrestdriver.OnosRestDriver object at 0x7fcc82fc7ad0> -:: diff --git a/docs/results/joid-os-odl_l2-nofeature-ha.rst b/docs/results/joid-os-odl_l2-nofeature-ha.rst index dc338ab42..29f4b37c1 100644 --- a/docs/results/joid-os-odl_l2-nofeature-ha.rst +++ b/docs/results/joid-os-odl_l2-nofeature-ha.rst @@ -10,6 +10,7 @@ VIM vping_ssh ^^^^^^^^^ :: + FUNCTEST.info: Running vPing-SSH test... vPing_ssh- INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.4-x86_64-disk.img'... vPing_ssh- INFO - Creating neutron network vping-net... @@ -41,12 +42,12 @@ vping_ssh vPing_ssh- INFO - vPing duration:'33.5' s. vPing_ssh- INFO - Cleaning up... vPing_ssh- INFO - vPing OK -:: + vping_userdata ^^^^^^^^^^^^^^ - :: + FUNCTEST.info: Running vPing-userdata test... vPing_userdata- INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.4-x86_64-disk.img'... vPing_userdata- INFO - Creating neutron network vping-net... @@ -87,11 +88,12 @@ vping_userdata vPing_userdata- INFO - vPing OK vPing_userdata- INFO - Cleaning up... vPing_userdata- INFO - Deleting network 'vping-net'... -:: + Tempest ^^^^^^^ :: + FUNCTEST.info: Running Tempest tests... run_tempest - INFO - Creating tenant and user for Tempest suite 2016-02-23 10:28:17.615 23826 INFO rally.verification.tempest.tempest [-] Starting: Creating configuration file for Tempest. @@ -325,11 +327,12 @@ Tempest run_tempest - INFO - Results: {'timestart': '2016-02-2310:28:27.307831', 'duration': 931, 'tests': 210, 'failures': 5} run_tempest - INFO - Pushing results to DB: 'http://testresults.opnfv.org/testapi/results'. run_tempest - INFO - Deleting tenant and user for Tempest suite) -:: + Rally ^^^^^ :: + FUNCTEST.info: Running Rally benchmark suite... run_rally - INFO - Starting test scenario "authenticate" ... run_rally - INFO - @@ -1630,7 +1633,7 @@ Rally +===================+============+===============+===========+ | TOTAL: | 01:25:53 | 232 | 100.00% | +===================+============+===============+===========+ -:: + SDN Controller -------------- @@ -1638,6 +1641,7 @@ SDN Controller ODL ^^^ :: + ============================================================================== Basic ============================================================================== @@ -1734,7 +1738,7 @@ ODL Neutron :: Test suite for Neutron Plugin | PASS | 18 critical tests, 18 passed, 0 failed 18 tests total, 18 passed, 0 failed -:: + Feature tests ------------- @@ -1742,6 +1746,7 @@ Feature tests Promise ^^^^^^^ :: + FUNCTEST.info: Running PROMISE test case... Promise- INFO - Creating tenant 'promise'... Promise- INFO - Adding role 'eed9b04b536646c994679496e7b653cc' to tenant 'promise'... @@ -2182,4 +2187,3 @@ Promise **************************************** Promise- INFO - Pushing results to DB... -:: diff --git a/docs/results/overview.rst b/docs/results/overview.rst index 9a9b68d32..ea71426b0 100644 --- a/docs/results/overview.rst +++ b/docs/results/overview.rst @@ -14,54 +14,49 @@ considered to evaluate whether the test case can be run or not. That is why all the number of test cases may vary from 1 installer to another and from 1 scenario to another. -The list of scenario supported by installer can be described as follows:: +The list of scenario supported by installer can be described as follows: - +================+=========+=========+=========+=========+ - | Scenario | Apex | Compass | Fuel | Joid | - +================+=========+=========+=========+=========+ - | odl_l2 | X | X | X | X | - +----------------+---------+---------+---------+---------+ - | odl_l3 | X | | X | | - +----------------+---------+---------+---------+---------+ - | onos | X | X | X | X | - +----------------+---------+---------+---------+---------+ - | nosdn | | X | X | X | - +----------------+---------+---------+---------+---------+ - | odl_l2-sfc | X | | | | - +----------------+---------+---------+---------+---------+ - | odl_l2-bgpvpn | | | X | | - +----------------+---------+---------+---------+---------+ - | ovs (dpdk) | | | X | | - +----------------+---------+---------+---------+---------+ ++----------------+---------+---------+---------+---------+ +| Scenario | Apex | Compass | Fuel | Joid | ++================+=========+=========+=========+=========+ +| odl_l2 | X | X | X | X | ++----------------+---------+---------+---------+---------+ +| onos | | X | | | ++----------------+---------+---------+---------+---------+ +| nosdn | | X | X | | ++----------------+---------+---------+---------+---------+ +| ovs (dpdk) | | | X | | ++----------------+---------+---------+---------+---------+ The matrix below details the Functest runnable tests versus the installer and -the scenario:: +the scenario: - +================+=============+=============+=============+=============+ - | Test cases | Apex | Compass | Fuel | Joid | - +================+=============+=============+=============+=============+ - | vPing |all scenario | all | all | all | - +----------------+-------------+-------------+-------------+-------------+ - | vPing userdata | all except | all except | all except | all except | - | | ONOS | ONOS | ONOS | ONOS | - +----------------+-------------+-------------+-------------+-------------+ - | Tempest | all | all | all | all | - +----------------+-------------+-------------+-------------+-------------+ - | Rally | all | all | all | all | - +----------------+-------------+-------------+-------------+-------------+ - | ODL | all ODL | all ODL | all ODL | all ODL | - +----------------+-------------+-------------+-------------+-------------+ - | ONOS | ONOS | ONOS | ONOS | ONOS | - +----------------+-------------+-------------+-------------+-------------+ - | Promise | no | no | all | all | - +----------------+-------------+-------------+-------------+-------------+ - | vIMS | all except | all except | all except | all except | - | | ONOS | ONOS | ONOS | ONOS | - +----------------+-------------+-------------+-------------+-------------+ - | Doctor | all | no | no | no | - +----------------+-------------+-------------+-------------+-------------+ - | SDNVPN | no | no | bgpvpn | no | - +----------------+-------------+-------------+-------------+-------------+ ++----------------+-------------+-------------+-------------+-------------+ +| Test cases | Apex | Compass | Fuel | Joid | ++================+=============+=============+=============+=============+ +| vPing SSH | all | all | all | all | ++----------------+-------------+-------------+-------------+-------------+ +| vPing userdata | all except | all except | all except | all except | +| | ONOS | ONOS | ONOS | ONOS | ++----------------+-------------+-------------+-------------+-------------+ +| Tempest | all | all | all | all | ++----------------+-------------+-------------+-------------+-------------+ +| Rally | all | all | all | all | ++----------------+-------------+-------------+-------------+-------------+ +| ODL | all ODL | all ODL | all ODL | all ODL | ++----------------+-------------+-------------+-------------+-------------+ +| ONOS | ONOS | ONOS | ONOS | ONOS | ++----------------+-------------+-------------+-------------+-------------+ +| Promise | no | no | all | all | ++----------------+-------------+-------------+-------------+-------------+ +| vIMS | all except | all except | all except | all except | +| | ONOS | ONOS | ONOS | ONOS | ++----------------+-------------+-------------+-------------+-------------+ +| Doctor | all | no | no | no | ++----------------+-------------+-------------+-------------+-------------+ + +all means that the test case is run on all the scenarios related to the +installer. Functest results from continuous integration can be found in: * jenkins logs: https://build.opnfv.org/ci/view/functest/ |