Age | Commit message (Collapse) | Author | Files | Lines |
|
There is a bug with neutron when using href links to get url.
This is fixed with Pike.
This is the patch,
https://review.openstack.org/#/c/356707/
These are the related files of Ocata and Pike,
https://github.com/openstack/neutron/blob/stable/ocata/neutron/api/api_common.py#L68
https://github.com/openstack/neutron/blob/stable/pike/neutron/api/api_common.py#L95
There are 2 test cases in neutron trunk related to this.
Remove these 2 test cases since Dovetail should support 4 versions from M to P.
Change-Id: I8aac88e4688157def1d6642eaa64bba56dde49ba
Signed-off-by: xudan <xudan16@huawei.com>
|
|
The OpenStack Interop WG has recently deprecated [1] 4 cinder volume
reservation tests. As a result, they are not part of Functest's set of
interop tests anymore. This patch cleans up the test case validation
list for the web portal.
[1] https://review.openstack.org/#/c/583970/
Change-Id: I183602b099cb4d44c6cb3d21af6acaa4d3ee01ca
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
As Functest does not support identity v2 anymore,
it does not make sense to try to run tempest identity v2 tests.
They need to be removed correspondingly.
JIRA: DOVETAIL-735
Change-Id: I09c06faf52893cc535bbed1fd406b81a7665e787
Signed-off-by: Panagiotis Karalis <pkaralis@intracom-telecom.com>
|
|
If mapping to host's /tmp, it won't clean all .log and .out files before
run a new test. It causes the results file including all previous results.
See https://build.opnfv.org/ci/view/dovetail/job/dovetail-apex-baremetal-default-mandatory-master/26/artifact/results/stress_logs/bottlenecks.stress.ping.out/*view*/
It will affect the results check.
Mapping to DOVETAIL_HOME/results/bottlenecks, which will always be cleaned
before running another test.
Change-Id: I22bfc871d6f041ae38308bf8e8f685993a376973
Signed-off-by: xudan <xudan16@huawei.com>
|
|
All results files including the .out files are stored in Yardstick /tmp
Need to map that with host's /tmp to parse the results.
JIRA: DOVETAIL-727
Change-Id: I5a81e6e6bdf79b2e0074df505deb425c37b93300
Signed-off-by: xudan <xudan16@huawei.com>
|
|
For ping, it set the ssh login username to be ubuntu.
So need to use ubuntu image rather than cirros.
JIRA: DOVETAIL-727
Change-Id: Iaac8c8280a464efe3a431d4eb260c4a0a626e12d
Signed-off-by: xudan <xudan16@huawei.com>
|
|
Change-Id: I32aaee370512c3d7c8d6e1fbb6ef25aebc5d483f
Signed-off-by: xudan <xudan16@huawei.com>
|
|
There is one sub test case that can't work correctly on Mitaka,
neutron.tests.tempest.api.test_trunk.TrunkTestJSON.test_create_update_trunk_with_description
It will get a 'Bad request' error.
Details: {u'message': u"Unrecognized attribute(s) 'description'",
u'type': u'HTTPBadRequest', u'detail': u''}.
That's mainly because of that,
For Mitaka, there's only several extension resources that support
attribute 'description', which is 'security_group_rules', 'security_groups',
'ports', 'subnets', 'networks', 'routers', 'floatingips' and 'subnetpools'.
All other extension resources including trunk port can't create with description.
See https://github.com/openstack/neutron/blob/mitaka-eol/neutron/extensions/standardattrdescription.py#L22
For Newton, all subclasses of class 'HasStandardAttributes' support
the attribute 'description' including class Trunk.
See https://github.com/openstack/neutron/blob/newton-eol/neutron/extensions/standardattrdescription.py#L49,
https://github.com/openstack/neutron/blob/newton-eol/neutron/db/standard_attr.py#L158
and
https://github.com/openstack/neutron/blob/newton-eol/neutron/services/trunk/models.py#L26
JIRA: DOVETAIL-723
Change-Id: Ifdfea7c54a1888d933c8ddd78a6feea1db93368c
Signed-off-by: xudan <xudan16@huawei.com>
|
|
1. neutron.tests.tempest.api.test_trunk.TrunkTestJSON.test_create_update_trunk
This test case will get the value of key 'revision_number' [1],
While this is a new extension for Newton and not support by Mitaka.
Please refer to the history of neutron github and the differences between
mitaka tag [3] (which has no file revisions.py) and newton [4].
2. neutron.tests.tempest.api.test_trunk.TrunkTestJSON.test_show_trunk_has_project_id
This test case requires extension 'project-id' [5], while this is also a new
extension for Newton. Please refer to the history of neutron [6] and the difference
between mitaka tag [3] (which has no file project_id.py) and newton [7].
Also it is a little unreasonable to check both tenant_id and project_id.
For some products, they may only have either tenant_id or project_id (not both).
Has sent an email to the neutron_trunk team but get no feedback until now.
[1] https://github.com/openstack/neutron/blob/stable%2Fpike/neutron/tests/tempest/api/test_trunk.py#L123
[2] https://github.com/openstack/neutron/commits/newton-eol/neutron/extensions/revisions.py
[3] https://github.com/openstack/neutron/tree/mitaka-eol/neutron/extensions
[4] https://github.com/openstack/neutron/blob/newton-eol/neutron/extensions/revisions.py
[5] https://github.com/openstack/neutron/blob/stable%2Fpike/neutron/tests/tempest/api/test_trunk.py#L110
[6] https://github.com/openstack/neutron/commits/newton-eol/neutron/extensions/project_id.py
[7] https://github.com/openstack/neutron/blob/master/neutron/extensions/project_id.py
JIRA: DOVETAIL-723
Change-Id: Idb4a56d47148b14ae4674549f98b1be65e9b8f15
Signed-off-by: xudan <xudan16@huawei.com>
|
|
JIRA: DOVETAIL-726
Change-Id: I57cb14cd5fe03a51e1691d7b48359bdd9b472ae4
Signed-off-by: xudan <xudan16@huawei.com>
|
|
It failed to run trunk_port test case because of the wrong class name.
See CI results
https://build.opnfv.org/ci/view/dovetail/job/dovetail-compass-baremetal-default-mandatory-fraser/7/artifact/results/tempest_logs/dovetail.tempest.neutron_trunk_ports.functest.log/*view*/
Update the testcases.yaml file based on the latest version of Functest Fraser.
JIRA: DOVETAIL-715
Change-Id: Ib102d1489d1e5610a04d76aab0553d5171ec3d3e
Signed-off-by: xudan <xudan16@huawei.com>
|
|
Replacing the generic "dovetail" prefix of the testcase definitions with
the name of the OPNFV test framework which hosts the test.
JIRA: DOVETAIL-719
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
Change-Id: I8f90423b8908197bdc7adddf82c61ff394893f52
|
|
It failed to archive tempest.log and tempest-report.html files.
See CI results
https://build.opnfv.org/ci/view/dovetail/job/dovetail-compass-baremetal-default-mandatory-fraser/7/artifact/results/
That's because the path of them have been changed
1. from tempest to tempest_custom
2. from refstack to refstack_defcore
There seems to be no file bgpvpn.log, remove the copy operation.
JIRA: DOVETAIL-714
Change-Id: I77d88124a44d3e5d869ada209760f69d1f6553bf
Signed-off-by: xudan <xudan16@huawei.com>
|
|
These 2 test cases failed on OSP10 with reasons
1. Provided object does not match schema 'image': u'shared'
is not one of ['public', 'private']
2. Provided object does not match schema 'image': u'community'
is not one of ['public', 'private']
'shared' and 'community' was added in the Image API v2.5
https://developer.openstack.org/api-ref/image/v2/index.html#images
JIRA: DOVETAIL-704
Change-Id: Ie7ab06d5e2abe7bde25fdb7318abe341f0119bc7
Signed-off-by: xudan <xudan16@huawei.com>
|
|
1. Copy a ubuntu image to Bottlenecks to support offline
2. Add an env DEPLOY_SCENARIO to bottlenecks to support DPDK
3. Set push results to DB False
JIRA: DOVETAIL-706
JIRA: DOVETAIL-707
Change-Id: I81f293bb9fddc63fd8be455546ba4642ca670fc7
Signed-off-by: xudan <xudan16@huawei.com>
|
|
We have moved all newly proposed tests to the next OVP test suite, so
these tests do not need to be in here anymore - and do not need to be
run in CI.
Change-Id: I5f51b76a37b706c5b4abc75272ad799a72841f67
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
The results.json file is currently not a properly formated json file,
but contains one json structure per line/testcase. Whit this change, all
test results are being collected at runtime and then written to a
properly formated json file at the end of a test run.
Moreover, this patch introduces 'mandatory' and 'optional' sections to
the test suites definition files in order to clearly indicate if a test
is part of the corresponding test scope.
The web portal needs to be adapted accordingly to be able to read the
new results file.
JIRA: DOVETAIL-672
Change-Id: Id0706c7e82622fc9c2c2fece26b2d6c07c1580be
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
According to Apex, Compass, Fuel CI results, delete the skipped sub test cases
from patrole and trunk_port check list.
JIRA: DOVETAIL-703
Change-Id: I86e3036a287d9cf56d78e43f92ce0d9ffed5941c
Signed-off-by: xudan <xudan16@huawei.com>
|
|
The 2 sections 'functest_testsuite' and 'functest_testcase' are used by
all Functest test cases.
They aim to parse the results of Tempest sub test cases.
Remove them, and check the sub test case list,
1. if it's empty, that means no need to check sub test cases
2. if it is not empty, check the sub test cases one by one.
Additional, remove some useless keys in dovetail_config.yml
JIRA: DOVETAIL-658
Change-Id: Ic829264b0c8eaedd8e2b3fa55b38a34644482c20
Signed-off-by: xudan <xudan16@huawei.com>
|
|
Update test suite ovp.next according to the latest OVP scope.
Modify Dovetail daily jobs to run all the scope.
There should be 2 jobs.
One for all mandatory test cases and the other for all optional ones.
That's mainly because of that the total executed time will be too large
(more than 300 minutes). Then the job will always failed because of time out.
It's hard to enlarge the time because it's already larger than common 3 hours.
Split it into 2 jobs can avoid the time out issue and make the results
clearer for reviewing.
JIRA: DOVETAIL-694
Change-Id: Ie0ea6221868941781af1477f7c7719f7cb4351a4
Signed-off-by: xudan <xudan16@huawei.com>
|
|
The fraser-based OPNFV tools use the newer cirros-0.4.0 image and we
hence only ask users to download this version in the user guide.
This updates the bottlenecks test to comply with this.
Change-Id: I5a94d6a7196ae04e00150dc17a4a16add2d3fef6
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
These 2 commands can't work now.
Also there is no need to support them, so remove them.
JIRA: DOVETAIL-695
Change-Id: I6707c7fc1d19fe06690cd14ef39f9d421d00a2bc
Signed-off-by: xudan <xudan16@huawei.com>
|
|
DPDK-enabled NFVIs require support for hugepages which
in turn is configured by means of extra_specs in flavors.
Currently, Functest enables hugepage support if the name
of the scenario contains the substring "ovs".
Dovetail allows end users to run all tests on DPDK-enabled systems
and the documentation has been updates accordingly.
JIRA: DOVETAIL-665
Change-Id: If4898da168f8b4c69a045ef3b00b8a8862f29b0c
Signed-off-by: Panagiotis Karalis <pkaralis@intracom-telecom.com>
|
|
Yardstick fraser has changed from openstack python client to shade.
OS_INSECURE is not a key for shade.
It needs to be OS_VERIFY.
Change-Id: I0973853d0527b04ec371ad4f6ce2f6c67ef50ccf
Signed-off-by: xudan <xudan16@huawei.com>
|
|
Split the networking tempest test case in Tempest API tests and scenario
tests. The API tests will be promoted to mandatory while the scenario
tests will remain optional.
JIRA: DOVETAIL-678
Change-Id: I03022316076809536782e82fd531d98f45105caa
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
The compliance_set test suite is not used. Updating the description of
the debug test suite.
Change-Id: Id2c3ca922ffb083f3443e00e5ca1a0648aec3e4a
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
Spliting the IPv6 test case in Tempest API tests and scenario tests. The
API tests will be promoted to mandatory while the scenario tests will
remain optional.
JIRA: DOVETAIL-677
Change-Id: If3dca5ecf2ce5c9e02de5c6d45e674da029b609f
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
In Fraser, the RefStack / OpenStack interop tests are executed through
Rally in Functest. Rally requires the fully qualified name of a tempest
test, including its tier and ID, in order to find and run it. This patch
add the missing information.
JIRA: DOVETAIL-662
Change-Id: I3a38b7b8d0fe79dd6ad0c0023c62c6647611d231
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
This test fails in CI on all installers (Apex, Compass, Fuel) and on
virtual deployments.
Change-Id: I3a451e880c2a5d2151cb88aaf8e5a6895368e63b
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
The neutron-l3-agent test assumes that a default image and flavor are
available. On current OPNFV deployments, these resources do not exist
anymore by default and hence the test case fails.
This patch enables Dovetail to create a specific flavor as part of the
test case config. This is temporary workaround until the test case
itself takes care of checking and creating all resources needed. A
ticket for the HA project has been created accordingly:
https://jira.opnfv.org/browse/HA-36
Change-Id: I6523d8411fe853a5645566b2fabe1409b5715491
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
We need to disable one BGPVPN tempest test due to a bug in ODL. Since
the SDNVPN test framework does not allow for disabling a single test
case, we are switching to tempest_custom.
Change-Id: Ifbd7c1dc5e799d6a6c774e08e9fc1a3b2294b47c
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
Change-Id: I58e37c14cb4a287436e86b345919deff3d3674d0
Signed-off-by: xudan <xudan16@huawei.com>
|
|
There are some superflouos lines separating the list of tempest test
cases. These are restructuring artifacts which can be deleted.
Change-Id: I7b3474090fa3d7b1506d5a4f3dcd1b32dd68900a
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
Use openstack shade to get endpoint infomation rather than executing
openstack commands directly.
The next step can replace openstack commands by shade for some test cases that
need to create openstack resources (images, flavors) if we agree to use shade.
JIRA: DOVETAIL-659
Change-Id: I72b302fb2d4a61e9249a7e53f81ea9464944dcd3
Signed-off-by: xudan <xudan16@huawei.com>
|
|
In line with the updated documentation, this patch upgrades the cirros
image from version 0.3.5 to 0.4.0 of the HA neutron-l3-agent test
Change-Id: I352e861f3d3099491ff6f367a3129dc8f3115394
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
Functest has switched from Tempest 14 in Danube to 17.1.0 in Fraser.
This updates the patch used to disable strict API response validation.
Change-Id: Ief47eba3bc91c222d5ad30e65ab1fdaf39fc169e
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
The configuration of the vEPC test case was missing information for
generating a correct report and hence it incorrectlt reported FAILed
despite passing.
JIRA:DOVETAIL-666
Change-Id: Id296fa0fd750e47dc841ef37acb2018b23de0643
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
Adding the vEPC test case to Dovetail. The test deploys a vEPC using
Juju as VNF manager and performs application level tests.
JIRA: DOVETAIL-650
Change-Id: I15c2cdddee510e12a5faf18efc8950af5cf7b664
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
1. All detailed tempest test cases results are recorded with
file tempest-report.html rather than tempest.log now.
2. Archive all results files include functest.log, yardstick.log ...
for each test case. Otherwise, they will all be in one log file.
3. Support to set the source files and dest files need to be archived with
the test case config files rather than hardcode with the source code.
4. Give the file which used to be parse the results (PASS/FAIL) with the
test case config files rather than hardcode with the source code.
Change-Id: I30f435d91ab90cf20def47007f177fe98187484d
Signed-off-by: xudan <xudan16@huawei.com>
|
|
CI broken because of the wrong test case name.
https://build.opnfv.org/ci/view/dovetail/job/dovetail-fuel-baremetal-proposed_tests-master/294/console
Change-Id: I6674acf92c8f67ae2bd8dba082186a16211dcd8a
Signed-off-by: xudan <xudan16@huawei.com>
|
|
1. Return empty test case list when test suite is empty.
2. Don't quite understand why comment out all proposed_tests.
3. Move stress ping test case at the end of the list,
because it may broken the message queue.
Change-Id: Ib717fc91e3763be03f243f82e53679462ecdae04
Signed-off-by: xudan <xudan16@huawei.com>
|
|
1. Change from Functest Danube to Fraser, the Tempest version it integrates
is from 14.0.0 to 17.1.0.
2. Some sub test cases names have been changed.
3. Some are removed from tempest_smoke.
4. Some are new added into tempest_smoke.
JIRA: DOVETAIL-657
Change-Id: Ie5611f5b7a5888bea3942667907341bd071d399d
Signed-off-by: xudan <xudan16@huawei.com>
|
|
Adding the OpenStack controller database HA test case (Yardstick tc090)
to Dovetail and the proposed_tests test suite.
JIRA: DOVETAIL-654
Change-Id: I4d20b907de8df2e9567891354b49e832cce13c7c
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
Updating the list of test cases used in the OpenStack Interop test case
to match
i) the tests used in the 2017.09 guideline, and
ii) the version of Tempest used by Functest
We keep the explicit list of test cases in the test case definition file
for now to enable Dovetail to verify the list of executed tests
corresponds to the tests we want to run.
JIRA: DOVETAIL-644
Change-Id: Ic627ef2bbad1dcfa5dd8773301c68dffb545a3b2
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
This patchset proposes a new naming scheme for test cases. The ides is
to replace the current numbered scheme
<testarea>.tc<number>
with:
<testarea>.<capability/test case description>
The motivation for this is that a the numbering does not convey any
information about what the test case is doing and hides to which test
case implementation in Functest or Yardstick a test case spec refers.
Note: this patchset has not yet been tested in Dovetail itself. Its goal
is start a discussion about the naming scheme.
JIRA: DOVETAIL-647
Change-Id: If33b4bf88e451b372c2af10895ff92a87d9d961d
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
1. Total number of sub test cases is 119
2. Remove all 14 object_storage test cases which need swift
3. Remove 26 test cases already included in osinterop test area
4. Remove 21 test cases already included in ipv6 test area
5. Remove 2 scenario test cases already included in tempest test area
6. Split other 56 test cases into 8 groups
Change-Id: I9eef20593ebf5f69347a2ec3a612c9ebc5a753db
Signed-off-by: xudan <xudan16@huawei.com>
|
|
switching to functest:fraser from functest:opnfv-6.0.0 to incorporate
bug fixes which have been backported to stable/fraser.
Change-Id: Ieab9064ac0187b3460b09fe74861fc844b0b0195
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
Adding the Neutron trunk port tempest tests to Dovetail and the
proposed_tests test suite.
JIRA: DOVETAIL-648
Change-Id: Ia77980a245fd1bca6e5f0e567467432e12bf974e
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
The code to run Tempest tests has changed in SDNVPN between Euphrates
and Fraser, causing these tests to fail. Moreover, due to a bug in
SDNVPN, not all bgpvpn API tests are currently being run. This is being
addressed by https://gerrit.opnfv.org/gerrit/#/c/57299/ and needs to be
cherrypicked to Fraser 6.1 for inclusion in OVP 2.0.
Change-Id: I0a7575e8a9ebc23af5edaa26c1a9ce1e8149919c
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
|
|
The CLI of dovetail tool needs some improvement.
1. Add an option '--testcase' to support specify test case.
2. Add an option '--no-clean' to keep functest/yardstick/bottlenecks containers
for debugging after running the test cases.
3. Improve the local report section.
a. Remove some info that is not necessary
b. Make the info more clearly
JIRA: DOVETAIL-588
Change-Id: Ic769a6e2788b407c3df08ccd593a0ecdaa35e5ea
Signed-off-by: xudan <xudan16@huawei.com>
|