summaryrefslogtreecommitdiffstats
path: root/etc/testcase
AgeCommit message (Collapse)AuthorFilesLines
2018-12-03Move 2 sub test cases in patrole to optional listxudan2-2/+26
JIRA: DOVETAIL-756 Change-Id: I8931f4b0c83acca24b5220d16e935194d4bc8ae7 Signed-off-by: xudan <xudan16@huawei.com>
2018-11-23Integrate k8s test cassesxudan2-0/+30
In order to integrate k8s test cases, Dovetail framework should do some refactor and make it more general for k8s test cases as well as ONAP ones. Integrate 2 k8s test cases. JIRA: DOVETAIL-748 Change-Id: Ibd87754ffb5fb29f6b4ce79232af860c2ed2da9c Signed-off-by: xudan <xudan16@huawei.com>
2018-11-07Update BGPVPN and VNF test casesxudan6-0/+12
Due to change to Functest Gambia, bgpvpn and vnf test cases failed. Update the testcases.yaml according to Functest's new format. Change-Id: Ia5d9ba291161f7da84a0aadf241e9ab81d83cfa9 Signed-off-by: xudan <xudan16@huawei.com>
2018-11-07Update tests based on tempest_customxudan11-11/+11
All tempest_custom test cases should be updated and defined with the new test case config yaml file. Change-Id: Iaff99fcfcef61060a8ca02819cac2763c4bdd2a0 Signed-off-by: xudan <xudan16@huawei.com>
2018-10-25Stop redefine testcases.yaml for some test casesxudan8-43/+45
1. For Functest healthcheck, patrole, refstack_defcore, neutron_trunk and vping test cases, stop to redefine testcases.yaml. 2. For VNF and bgpvpn, they still have scenario dependencies for Functest. How to handle them are still under consideration. 3. For other tempest test cases, they rely on tempest_custom. Leave them under discussion. JIRA: DOVETAIL-673 Change-Id: Ibd10c9883d8df3749f4a93e280deaca77fe6bc00 Signed-off-by: xudan <xudan16@huawei.com>
2018-09-10Remove neutron trunk test cases with href linksxudan1-2/+0
There is a bug with neutron when using href links to get url. This is fixed with Pike. This is the patch, https://review.openstack.org/#/c/356707/ These are the related files of Ocata and Pike, https://github.com/openstack/neutron/blob/stable/ocata/neutron/api/api_common.py#L68 https://github.com/openstack/neutron/blob/stable/pike/neutron/api/api_common.py#L95 There are 2 test cases in neutron trunk related to this. Remove these 2 test cases since Dovetail should support 4 versions from M to P. Change-Id: I8aac88e4688157def1d6642eaa64bba56dde49ba Signed-off-by: xudan <xudan16@huawei.com>
2018-09-04Update test list for OS interop testsGeorg Kunz1-4/+0
The OpenStack Interop WG has recently deprecated [1] 4 cinder volume reservation tests. As a result, they are not part of Functest's set of interop tests anymore. This patch cleans up the test case validation list for the web portal. [1] https://review.openstack.org/#/c/583970/ Change-Id: I183602b099cb4d44c6cb3d21af6acaa4d3ee01ca Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-09-03Remove identity v2 TCs,not supported by FunctestPanagiotis Karalis1-29/+0
As Functest does not support identity v2 anymore, it does not make sense to try to run tempest identity v2 tests. They need to be removed correspondingly. JIRA: DOVETAIL-735 Change-Id: I09c06faf52893cc535bbed1fd406b81a7665e787 Signed-off-by: Panagiotis Karalis <pkaralis@intracom-telecom.com>
2018-08-13Remove one sub test cases from neutron trunkxudan1-1/+0
There is one sub test case that can't work correctly on Mitaka, neutron.tests.tempest.api.test_trunk.TrunkTestJSON.test_create_update_trunk_with_description It will get a 'Bad request' error. Details: {u'message': u"Unrecognized attribute(s) 'description'", u'type': u'HTTPBadRequest', u'detail': u''}. That's mainly because of that, For Mitaka, there's only several extension resources that support attribute 'description', which is 'security_group_rules', 'security_groups', 'ports', 'subnets', 'networks', 'routers', 'floatingips' and 'subnetpools'. All other extension resources including trunk port can't create with description. See https://github.com/openstack/neutron/blob/mitaka-eol/neutron/extensions/standardattrdescription.py#L22 For Newton, all subclasses of class 'HasStandardAttributes' support the attribute 'description' including class Trunk. See https://github.com/openstack/neutron/blob/newton-eol/neutron/extensions/standardattrdescription.py#L49, https://github.com/openstack/neutron/blob/newton-eol/neutron/db/standard_attr.py#L158 and https://github.com/openstack/neutron/blob/newton-eol/neutron/services/trunk/models.py#L26 JIRA: DOVETAIL-723 Change-Id: Ifdfea7c54a1888d933c8ddd78a6feea1db93368c Signed-off-by: xudan <xudan16@huawei.com>
2018-08-14Remove 2 sub test cases from neutron_trunk check listxudan1-2/+0
1. neutron.tests.tempest.api.test_trunk.TrunkTestJSON.test_create_update_trunk This test case will get the value of key 'revision_number' [1], While this is a new extension for Newton and not support by Mitaka. Please refer to the history of neutron github and the differences between mitaka tag [3] (which has no file revisions.py) and newton [4]. 2. neutron.tests.tempest.api.test_trunk.TrunkTestJSON.test_show_trunk_has_project_id This test case requires extension 'project-id' [5], while this is also a new extension for Newton. Please refer to the history of neutron [6] and the difference between mitaka tag [3] (which has no file project_id.py) and newton [7]. Also it is a little unreasonable to check both tenant_id and project_id. For some products, they may only have either tenant_id or project_id (not both). Has sent an email to the neutron_trunk team but get no feedback until now. [1] https://github.com/openstack/neutron/blob/stable%2Fpike/neutron/tests/tempest/api/test_trunk.py#L123 [2] https://github.com/openstack/neutron/commits/newton-eol/neutron/extensions/revisions.py [3] https://github.com/openstack/neutron/tree/mitaka-eol/neutron/extensions [4] https://github.com/openstack/neutron/blob/newton-eol/neutron/extensions/revisions.py [5] https://github.com/openstack/neutron/blob/stable%2Fpike/neutron/tests/tempest/api/test_trunk.py#L110 [6] https://github.com/openstack/neutron/commits/newton-eol/neutron/extensions/project_id.py [7] https://github.com/openstack/neutron/blob/master/neutron/extensions/project_id.py JIRA: DOVETAIL-723 Change-Id: Idb4a56d47148b14ae4674549f98b1be65e9b8f15 Signed-off-by: xudan <xudan16@huawei.com>
2018-07-31Renaming test cases to include test framework namesGeorg Kunz44-220/+220
Replacing the generic "dovetail" prefix of the testcase definitions with the name of the OPNFV test framework which hosts the test. JIRA: DOVETAIL-719 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com> Change-Id: I8f90423b8908197bdc7adddf82c61ff394893f52
2018-07-30Dovetail failed to archive tempest.log and tempest-report.html filesxudan18-36/+28
It failed to archive tempest.log and tempest-report.html files. See CI results https://build.opnfv.org/ci/view/dovetail/job/dovetail-compass-baremetal-default-mandatory-fraser/7/artifact/results/ That's because the path of them have been changed 1. from tempest to tempest_custom 2. from refstack to refstack_defcore There seems to be no file bgpvpn.log, remove the copy operation. JIRA: DOVETAIL-714 Change-Id: I77d88124a44d3e5d869ada209760f69d1f6553bf Signed-off-by: xudan <xudan16@huawei.com>
2018-07-27Delete 2 sub test cases from patrolexudan1-2/+0
These 2 test cases failed on OSP10 with reasons 1. Provided object does not match schema 'image': u'shared' is not one of ['public', 'private'] 2. Provided object does not match schema 'image': u'community' is not one of ['public', 'private'] 'shared' and 'community' was added in the Image API v2.5 https://developer.openstack.org/api-ref/image/v2/index.html#images JIRA: DOVETAIL-704 Change-Id: Ie7ab06d5e2abe7bde25fdb7318abe341f0119bc7 Signed-off-by: xudan <xudan16@huawei.com>
2018-07-23Delete skipped sub test cases from patrole and neutron_trunk check listxudan2-23/+0
According to Apex, Compass, Fuel CI results, delete the skipped sub test cases from patrole and trunk_port check list. JIRA: DOVETAIL-703 Change-Id: I86e3036a287d9cf56d78e43f92ce0d9ffed5941c Signed-off-by: xudan <xudan16@huawei.com>
2018-07-22Remove functest_testsuite and functest_testcase in dovetail_config.ymlxudan2-0/+187
The 2 sections 'functest_testsuite' and 'functest_testcase' are used by all Functest test cases. They aim to parse the results of Tempest sub test cases. Remove them, and check the sub test case list, 1. if it's empty, that means no need to check sub test cases 2. if it is not empty, check the sub test cases one by one. Additional, remove some useless keys in dovetail_config.yml JIRA: DOVETAIL-658 Change-Id: Ic829264b0c8eaedd8e2b3fa55b38a34644482c20 Signed-off-by: xudan <xudan16@huawei.com>
2018-07-02Splitting the networking tests in API and scenario testsGeorg Kunz2-11/+35
Split the networking tempest test case in Tempest API tests and scenario tests. The API tests will be promoted to mandatory while the scenario tests will remain optional. JIRA: DOVETAIL-678 Change-Id: I03022316076809536782e82fd531d98f45105caa Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-29Splitting IPv6 tests in API and scenario testsGeorg Kunz2-14/+38
Spliting the IPv6 test case in Tempest API tests and scenario tests. The API tests will be promoted to mandatory while the scenario tests will remain optional. JIRA: DOVETAIL-677 Change-Id: If3dca5ecf2ce5c9e02de5c6d45e674da029b609f Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-20Adding IDs to the list of OpenStack interop testsGeorg Kunz1-204/+204
In Fraser, the RefStack / OpenStack interop tests are executed through Rally in Functest. Rally requires the fully qualified name of a tempest test, including its tier and ID, in order to find and run it. This patch add the missing information. JIRA: DOVETAIL-662 Change-Id: I3a38b7b8d0fe79dd6ad0c0023c62c6647611d231 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-19Removing test_mtu_sized_frames Tempest testGeorg Kunz1-1/+0
This test fails in CI on all installers (Apex, Compass, Fuel) and on virtual deployments. Change-Id: I3a451e880c2a5d2151cb88aaf8e5a6895368e63b Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-19Create specific flavor for neutron-l3-agent HA testGeorg Kunz1-5/+9
The neutron-l3-agent test assumes that a default image and flavor are available. On current OPNFV deployments, these resources do not exist anymore by default and hence the test case fails. This patch enables Dovetail to create a specific flavor as part of the test case config. This is temporary workaround until the test case itself takes care of checking and creating all resources needed. A ticket for the HA project has been created accordingly: https://jira.opnfv.org/browse/HA-36 Change-Id: I6523d8411fe853a5645566b2fabe1409b5715491 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-15Running BGPVPN Tempest tests as tempest_custom testGeorg Kunz1-5/+22
We need to disable one BGPVPN tempest test due to a bug in ODL. Since the SDNVPN test framework does not allow for disabling a single test case, we are switching to tempest_custom. Change-Id: Ifbd7c1dc5e799d6a6c774e08e9fc1a3b2294b47c Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-15Copy tempest_conf.yaml file for neutron_port and patrolexudan2-0/+6
Change-Id: I58e37c14cb4a287436e86b345919deff3d3674d0 Signed-off-by: xudan <xudan16@huawei.com>
2018-06-14Cosmetic cleanup of tempest test case listsGeorg Kunz4-28/+0
There are some superflouos lines separating the list of tempest test cases. These are restructuring artifacts which can be deleted. Change-Id: I7b3474090fa3d7b1506d5a4f3dcd1b32dd68900a Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-08Upgrading image version of HA test of neutron L3 agentGeorg Kunz1-1/+1
In line with the updated documentation, this patch upgrades the cirros image from version 0.3.5 to 0.4.0 of the HA neutron-l3-agent test Change-Id: I352e861f3d3099491ff6f367a3129dc8f3115394 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-05Correct reporting of vEPC test caseGeorg Kunz1-0/+5
The configuration of the vEPC test case was missing information for generating a correct report and hence it incorrectlt reported FAILed despite passing. JIRA:DOVETAIL-666 Change-Id: Id296fa0fd750e47dc841ef37acb2018b23de0643 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-29Adding vEPC test case to Dovetail and proposed_tests suiteGeorg Kunz1-0/+10
Adding the vEPC test case to Dovetail. The test deploys a vEPC using Juju as VNF manager and performs application level tests. JIRA: DOVETAIL-650 Change-Id: I15c2cdddee510e12a5faf18efc8950af5cf7b664 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-27Archive all test case result filesxudan39-4/+286
1. All detailed tempest test cases results are recorded with file tempest-report.html rather than tempest.log now. 2. Archive all results files include functest.log, yardstick.log ... for each test case. Otherwise, they will all be in one log file. 3. Support to set the source files and dest files need to be archived with the test case config files rather than hardcode with the source code. 4. Give the file which used to be parse the results (PASS/FAIL) with the test case config files rather than hardcode with the source code. Change-Id: I30f435d91ab90cf20def47007f177fe98187484d Signed-off-by: xudan <xudan16@huawei.com>
2018-05-18Bugfix the name of test case dovetail.neutron_l3_agentxudan1-1/+1
CI broken because of the wrong test case name. https://build.opnfv.org/ci/view/dovetail/job/dovetail-fuel-baremetal-proposed_tests-master/294/console Change-Id: I6674acf92c8f67ae2bd8dba082186a16211dcd8a Signed-off-by: xudan <xudan16@huawei.com>
2018-05-18Correct the names of sub test cases due to the new Tempest versionxudan9-55/+30
1. Change from Functest Danube to Fraser, the Tempest version it integrates is from 14.0.0 to 17.1.0. 2. Some sub test cases names have been changed. 3. Some are removed from tempest_smoke. 4. Some are new added into tempest_smoke. JIRA: DOVETAIL-657 Change-Id: Ie5611f5b7a5888bea3942667907341bd071d399d Signed-off-by: xudan <xudan16@huawei.com>
2018-05-17Adding OpenStack controller database HA test caseGeorg Kunz1-0/+11
Adding the OpenStack controller database HA test case (Yardstick tc090) to Dovetail and the proposed_tests test suite. JIRA: DOVETAIL-654 Change-Id: I4d20b907de8df2e9567891354b49e832cce13c7c Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-15Updating the list of OS interop testsGeorg Kunz1-190/+184
Updating the list of test cases used in the OpenStack Interop test case to match i) the tests used in the 2017.09 guideline, and ii) the version of Tempest used by Functest We keep the explicit list of test cases in the test case definition file for now to enable Dovetail to verify the list of executed tests corresponds to the tests we want to run. JIRA: DOVETAIL-644 Change-Id: Ic627ef2bbad1dcfa5dd8773301c68dffb545a3b2 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-15Renaming test cases for better readabilityGeorg Kunz80-698/+249
This patchset proposes a new naming scheme for test cases. The ides is to replace the current numbered scheme <testarea>.tc<number> with: <testarea>.<capability/test case description> The motivation for this is that a the numbering does not convey any information about what the test case is doing and hides to which test case implementation in Functest or Yardstick a test case spec refers. Note: this patchset has not yet been tested in Dovetail itself. Its goal is start a discussion about the naming scheme. JIRA: DOVETAIL-647 Change-Id: If33b4bf88e451b372c2af10895ff92a87d9d961d Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-15Split tempest_smoke test casexudan9-134/+176
1. Total number of sub test cases is 119 2. Remove all 14 object_storage test cases which need swift 3. Remove 26 test cases already included in osinterop test area 4. Remove 21 test cases already included in ipv6 test area 5. Remove 2 scenario test cases already included in tempest test area 6. Split other 56 test cases into 8 groups Change-Id: I9eef20593ebf5f69347a2ec3a612c9ebc5a753db Signed-off-by: xudan <xudan16@huawei.com>
2018-05-11Adding Neutron trunk port tests to proposed_tests suiteGeorg Kunz1-0/+9
Adding the Neutron trunk port tempest tests to Dovetail and the proposed_tests test suite. JIRA: DOVETAIL-648 Change-Id: Ia77980a245fd1bca6e5f0e567467432e12bf974e Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-11Fixing SDNVPN tempest API testsGeorg Kunz1-1/+1
The code to run Tempest tests has changed in SDNVPN between Euphrates and Fraser, causing these tests to fail. Moreover, due to a bug in SDNVPN, not all bgpvpn API tests are currently being run. This is being addressed by https://gerrit.opnfv.org/gerrit/#/c/57299/ and needs to be cherrypicked to Fraser 6.1 for inclusion in OVP 2.0. Change-Id: I0a7575e8a9ebc23af5edaa26c1a9ce1e8149919c Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-01Integrating Functest Patrole test caseGeorg Kunz1-0/+9
Adding the new Patrole test case of Functest to Dovetail and to the proposed_tests test suite. JIRA: DOVETAIL-649 Change-Id: I2c3f583cdbd3af75cec020b0fe97f287d23498a5 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-04-17Switch to OPNFV Fraser releasexudan33-38/+33
1. Use Functest Fraser docker images whose tag is fraser 2. Use Yardstick stable docker images 3. Functest fraser has moved testcases.yaml file from functest/ci to xtesting/ci 4. Functest fraser has removed 'functest env prepare' command 5. Functest uses rally to execute osinterop test cases instead of refstack_client Change-Id: I81ebefe6ea959ca77c2c87f1d75ef56ac18c165f Signed-off-by: xudan <xudan16@huawei.com>
2018-04-10Add functest healthcheck test suiteEddie Arrage3-0/+27
- Add Functest healthcheck tier to Dovetail as new test suite - Intention is that these test cases are not part of mandatory test cases in compliance program - Use for debug and SUT pre-flight check - May consider running tc001 (connection_check) at other test execution stages to determine if SUT is still functioning, as this test takes < 10 sec (more important as test complexity increases such as in vnf test tier) - Replacement for patch: https://gerrit.opnfv.org/gerrit/#/c/52461/ - Removed CI filters, as suggested Change-Id: I2e72931b27e43359d6250f6959a17af8526c5094 Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-04-10Decoupling Yardstick Docker config items from source codexudan1-2/+1
There are some special Docker setting items when creating Yardstick Docker Containers. 1. Remove the extra settings which are required by Yardstick Danube. 2. Remove the settings for pushing results to DB because Dovetail plan to disable this function. 3. Move the other special Functest settings to config files. JIRA: DOVETAIL-640 Change-Id: Ic2d90393e227d7886b14d805eaa307b7aef56e4a Signed-off-by: xudan <xudan16@huawei.com>
2018-04-10Support to specify docker image for each test casexudan7-0/+7
1. Currently each type of test cases will use one docker image. 2. For example, Functest test cases use opnfv/functest-restapi:euphrates Yardstick uses opnfv/yardstick:opnfv-5.1.0 3. It needs to support to specify the docker image for each test case. 4. For example, vping test cases use opnfv/functest-smoke:euphrates sdnvpn uses opnfv/functest-features:euphrates 5. The benefit of this is that these docker images are well verified by Functest CI and from Functest plan, they may not support functest-restapi docker image in its future release. JIRA: DOVETAIL-638 Change-Id: I8a30eea2233aeff809af2e241b6c939194397832 Signed-off-by: xudan <xudan16@huawei.com>
2018-04-04Decoupling Functest Docker config items from source codexudan7-12/+1
There are some special Docker setting item when creating Functest Containers. 1. Remove the extra settings which are required by Functest Danube. 2. Remove the settings for pushing results to DB because Dovetail plans to disable this function. 3. Move the other special Functest settings to config files. JIRA: DOVETAIL-639 Change-Id: Ibc4090b5c0aafdf9ed64e65a9a92e34a12a16df5 Signed-off-by: xudan <xudan16@huawei.com>
2018-03-29Integrate dovetail with ONAPMoshe1-0/+9
Change-Id: I6a1fa5f27df2f5127eb00156b3135dc79caf5e77 Issue-ID: DOVETAIL-629 Signed-off-by: Moshe <moshehoa@amdocs.com>
2018-03-15Disable restricts of sdnvpn test casesxudan6-1/+12
1. The original testcases.yaml file of Functest restricts sdnvpn test cases to run on Fuel, Apex or Netvirt on bgpvpn scenario. https://github.com/opnfv/functest/blob/master/functest/ci/testcases.yaml#L278 2. Dovetail has a testcases.yaml file to allow sdnvpn test cases run on any commercial SUTs. https://github.com/opnfv/dovetail/blob/master/etc/userconfig/testcases.yaml#L113 Change-Id: I0d5ed5b22e55dc3dc0fe584dc57b28239334b7de Signed-off-by: xudan <xudan16@huawei.com>
2018-03-15substitute image copy with docker volumeSerenaFeng1-1/+1
Change-Id: Ie470d03bd168c909fd02c316523049c67a6bb095 Signed-off-by: SerenaFeng <feng.xiaowei@zte.com.cn>
2018-03-06leverage data_files to eliminate so many hardcoded abs_path_seekingsSerenaFeng58-0/+1193
1. mv conf/ userconfig/ compliance/ patch to etc/ 2. after installation etc/ will be put under /etc/dovetail 3. leveraging /etc/dovetail in coding Change-Id: Icf9cb00ff7fc7cf35755f60d3e258b23e2c70839 Signed-off-by: SerenaFeng <feng.xiaowei@zte.com.cn>