diff options
author | Romanos Skiadas <rski@intracom-telecom.com> | 2016-12-14 10:13:23 +0200 |
---|---|---|
committer | Romanos Skiadas <rski@intracom-telecom.com> | 2016-12-16 14:42:45 +0200 |
commit | ca3cb806082b7f39ffacde3b7c7776597fa80ec9 (patch) | |
tree | d2765155d53b122b2b290b9919dba0fcb9f9db64 /sdnvpn/test/functest/config.yaml | |
parent | 89ce471643c0cbaf41cac0e6995e754b2169889b (diff) |
Modify tests to require 100% subtest success, fix tempest
This is required to better align with functest, since that will handle overall
success/failure status. Also if a subestest failed, there were expected failures
that could seem like they pass making the overall success % look the same, which
would make the test look fine, even thought it failed.
Some tests check things that are known not to work. Disable these until they are
fixed upstream.
Also remove the success_criteria field from the config. Fail the test if any
subtest fails. This has the added benefit that it fixes the tempest suite, which
was still trying to read the value of `succes_criteria` that had been renamed to
`success_criteria`.
JIRA: SDNVPN-89
Change-Id: Iba5e5cc0db6c371f29a4cbba66ca16c4a1838e12
Signed-off-by: Romanos Skiadas <rski@intracom-telecom.com>
Diffstat (limited to 'sdnvpn/test/functest/config.yaml')
-rw-r--r-- | sdnvpn/test/functest/config.yaml | 6 |
1 files changed, 0 insertions, 6 deletions
diff --git a/sdnvpn/test/functest/config.yaml b/sdnvpn/test/functest/config.yaml index ccd5db7..ab05ef9 100644 --- a/sdnvpn/test/functest/config.yaml +++ b/sdnvpn/test/functest/config.yaml @@ -5,14 +5,12 @@ testcases: tempest: enabled: true description: Neutron BGPVPN tests in tempest - success_criteria: 100 # all the subtests must pass testname_db: functest_tempest # declared name in the test api # http://testresults.opnfv.org/test/api/v1/projects/sdnvpn/cases testcase_1: enabled: true description: VPN provides connectivity between subnets - success_criteria: 75 # we let fail 25% of the subtests testname_db: functest_testcase_1 instance_1_name: sdnvpn-1-1 instance_2_name: sdnvpn-1-2 @@ -37,7 +35,6 @@ testcases: testcase_2: enabled: true description: Tenant separation - success_criteria: 100 # all the subtests must pass testname_db: functest_testcase_2 instance_1_name: sdnvpn-2-1 instance_2_name: sdnvpn-2-2 @@ -72,13 +69,11 @@ testcases: testcase_3: enabled: false description: Data center gateway integration - success_criteria: 100 # all the subtests must pass testname_db: functest_testcase_3 testcase_4: enabled: true description: VPN provides connectivity between subnets using router association - success_criteria: 75 # we let fail 25% of the subtests testname_db: functest_testcase_4 instance_1_name: sdnvpn-4-1 instance_2_name: sdnvpn-4-2 @@ -103,7 +98,6 @@ testcases: testcase_7: enabled: false description: Network associate VPNs with routers attached (ODL Bug 6962) - success_criteria: 100 testname_db: functest_testcase_7 image_name: sdnvpn-image instance_1_name: sdnvpn-7-1 |