summaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)AuthorFilesLines
2018-06-29Splitting IPv6 tests in API and scenario testsGeorg Kunz3-15/+40
Spliting the IPv6 test case in Tempest API tests and scenario tests. The API tests will be promoted to mandatory while the scenario tests will remain optional. JIRA: DOVETAIL-677 Change-Id: If3dca5ecf2ce5c9e02de5c6d45e674da029b609f Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-22Fixing a typo in the API validation warning of the web portal.Georg Kunz1-2/+2
Change-Id: I0489d1e2fbd53bc90c24781a9d62140e2299c5a4 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-20Update OVP portal docker images for missing packageEddie Arrage1-2/+2
- The missing futures packages was added in patch: https://gerrit.opnfv.org/gerrit/#/c/58833/ - The patch above does resolve the issue and has been valdiated on a development instance. - The issue was likely introduced in patch https://gerrit.opnfv.org/gerrit/#/c/56623/ Change-Id: I850947e386024e6b65514bdb518799c359234811 Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-06-20Add Missing Futures PackageTrevor Bramwell1-0/+1
This is needed by the motor package. Change-Id: Ie19958040bc026b834fd3891da3a205cb248df92 Signed-off-by: Trevor Bramwell <tbramwell@linuxfoundation.org>
2018-06-20Update CVP Container TagsTrevor Bramwell1-2/+2
These include updates for the new web portal. Change-Id: I2951c8a1b00947a4546b4f90440ff8e0af8640e4 Signed-off-by: Trevor Bramwell <tbramwell@linuxfoundation.org>
2018-06-20Adding IDs to the list of OpenStack interop testsGeorg Kunz1-204/+204
In Fraser, the RefStack / OpenStack interop tests are executed through Rally in Functest. Rally requires the fully qualified name of a tempest test, including its tier and ID, in order to find and run it. This patch add the missing information. JIRA: DOVETAIL-662 Change-Id: I3a38b7b8d0fe79dd6ad0c0023c62c6647611d231 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-19Removing test_mtu_sized_frames Tempest testGeorg Kunz1-1/+0
This test fails in CI on all installers (Apex, Compass, Fuel) and on virtual deployments. Change-Id: I3a451e880c2a5d2151cb88aaf8e5a6895368e63b Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-19Create specific flavor for neutron-l3-agent HA testGeorg Kunz1-5/+9
The neutron-l3-agent test assumes that a default image and flavor are available. On current OPNFV deployments, these resources do not exist anymore by default and hence the test case fails. This patch enables Dovetail to create a specific flavor as part of the test case config. This is temporary workaround until the test case itself takes care of checking and creating all resources needed. A ticket for the HA project has been created accordingly: https://jira.opnfv.org/browse/HA-36 Change-Id: I6523d8411fe853a5645566b2fabe1409b5715491 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-15Running BGPVPN Tempest tests as tempest_custom testGeorg Kunz2-6/+33
We need to disable one BGPVPN tempest test due to a bug in ODL. Since the SDNVPN test framework does not allow for disabling a single test case, we are switching to tempest_custom. Change-Id: Ifbd7c1dc5e799d6a6c774e08e9fc1a3b2294b47c Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-15Copy tempest_conf.yaml file for neutron_port and patrolexudan2-0/+6
Change-Id: I58e37c14cb4a287436e86b345919deff3d3674d0 Signed-off-by: xudan <xudan16@huawei.com>
2018-06-14Cosmetic cleanup of tempest test case listsGeorg Kunz4-28/+0
There are some superflouos lines separating the list of tempest test cases. These are restructuring artifacts which can be deleted. Change-Id: I7b3474090fa3d7b1506d5a4f3dcd1b32dd68900a Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-14Replace openstack commands with shadexudan7-59/+48
Use openstack shade to get endpoint infomation rather than executing openstack commands directly. The next step can replace openstack commands by shade for some test cases that need to create openstack resources (images, flavors) if we agree to use shade. JIRA: DOVETAIL-659 Change-Id: I72b302fb2d4a61e9249a7e53f81ea9464944dcd3 Signed-off-by: xudan <xudan16@huawei.com>
2018-06-08Upgrading image version of HA test of neutron L3 agentGeorg Kunz1-1/+1
In line with the updated documentation, this patch upgrades the cirros image from version 0.3.5 to 0.4.0 of the HA neutron-l3-agent test Change-Id: I352e861f3d3099491ff6f367a3129dc8f3115394 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-07Updating the API response validation patchGeorg Kunz1-144/+279
Functest has switched from Tempest 14 in Danube to 17.1.0 in Fraser. This updates the patch used to disable strict API response validation. Change-Id: Ief47eba3bc91c222d5ad30e65ab1fdaf39fc169e Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-07Add improvements to OVP directory and logo uploadEddie Arrage7-6/+110
- Formatting changes to directory on home page based on OPNFV marketing - Updated branding guide link - Added test_id field to mongo applications collection to associate to approved results for directory - Set Test ID from user accounts with administrator role in Applications view - Provide file upload function for administrator to post company logos for OVP directory - Company logos are stored and served through tornado from cvp-cvpapi container rather than cvp-web JIRA: DOVETAIL-663 JIRA: DOVETAIL-664 Change-Id: I1226b42883afa2ea2eb5551e3836211abbb94b20 Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-06-06Make exit status a constant in run.pyMiikka Koistinen1-4/+4
JIRA: DOVETAIL-625 Change-Id: Ib96e6e39bad708f495af97ed43884909e394da10 Signed-off-by: Miikka Koistinen <miikka.koistinen@nokia.com>
2018-06-06Fix error status when executing CLI run commandMiikka Koistinen2-1/+7
This commit fixes CLI run command to exit with non-zero exit status when bad arguments are given for '--testsuite' or '--testarea'. JIRA: DOVETAIL-660 Change-Id: I4539b19978d9e93d8a90be9000f36d0aa1895f71 Signed-off-by: Miikka Koistinen <miikka.koistinen@nokia.com>
2018-06-05Correct reporting of vEPC test caseGeorg Kunz1-0/+5
The configuration of the vEPC test case was missing information for generating a correct report and hence it incorrectlt reported FAILed despite passing. JIRA:DOVETAIL-666 Change-Id: Id296fa0fd750e47dc841ef37acb2018b23de0643 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-29Adding reference to EXTERNAL_NETWORK to user guideGeorg Kunz1-2/+5
Adding a reference to the EXTERNAL_NETWORK environment variable to the user guide so that users add this to their env_config.sh. This variable is required, among others, to overwrite the default external network in Yarstick. Change-Id: I50e609a8c6e20b1dd01c61d5d950ffba792e98fa Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-29Adding vEPC test case to Dovetail and proposed_tests suiteGeorg Kunz4-0/+26
Adding the vEPC test case to Dovetail. The test deploys a vEPC using Juju as VNF manager and performs application level tests. JIRA: DOVETAIL-650 Change-Id: I15c2cdddee510e12a5faf18efc8950af5cf7b664 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-28Update user guidexudan1-139/+199
1. Remove python related content because python is installed in Dovetail image. 2. Remove DB related content because there is no need to launch local DB now. 3. Update the infomation of Docker images and VM images. Change-Id: I010d3341aef9f2c74713fc0fc087b361331109d1 Signed-off-by: xudan <xudan16@huawei.com>
2018-05-27Archive all test case result filesxudan48-92/+354
1. All detailed tempest test cases results are recorded with file tempest-report.html rather than tempest.log now. 2. Archive all results files include functest.log, yardstick.log ... for each test case. Otherwise, they will all be in one log file. 3. Support to set the source files and dest files need to be archived with the test case config files rather than hardcode with the source code. 4. Give the file which used to be parse the results (PASS/FAIL) with the test case config files rather than hardcode with the source code. Change-Id: I30f435d91ab90cf20def47007f177fe98187484d Signed-off-by: xudan <xudan16@huawei.com>
2018-05-27Changing the link to the updated participation formGeorg Kunz2-2/+2
The participation form on docusign has changed due to incorporating additional fields for the API validation exemption. This patch updates the links in the documentation accordingly. Note: this patch should only me merged after the new form is online. Change-Id: I697fa13d9afe2797faba2bf0478bbf27820bca20 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-22Implemented directory within OVP portalEddie Arrage9-6/+251
- Added OVP directory to portal home page as summary view to showcase vendors offerings that have received marks - Each row of table in directory summary represents a product offering that has received compliance verification marks and will be ordered by approve date - Currently, a vendor with multiple products will be displayed as separate rows, as agreed by OPNFV marketing - Added new angular component/controller for OVP directory secondary drill-down view, which provides more detailed info including Description, Product Info, SUT HW Version (optional from vendor) and SUT Version. Product info links to product_documentation field in Applications collection but can be improved as requirements are flushed out - Added new db fields to existing applications collection in mongodb - Applications collection is fetched on home page and detailed directory views - Updated Applications management view (administrator role) with new db fields including description, ovp_category, ovp_version, company_logo, approve_date, approved, sut_version, sut_hw_version, allowing admin to delete and add back with updates. - Updated participation form link for exemption process addition - Consider for another patch - upload form for company logos in Applications view for administrator to add. The initial wave of logos will need to be included in the patch for now when available from OPNFV marketing - Consider for another patch - improve 'Product Info' field to accept HTML as input allowing administrator to add raw HTML in mongod - Consider for another patch - update an existing application for admin role in Applications view rather than forcing a delete/add operation for an update Change-Id: I41d08a7164ee4513106bd09ea30d20be5abd911e Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-05-19Updating user guide with new ssh key config in pod.yamlGeorg Kunz1-4/+5
Dovetail does not copy the ssh key from pre_config to /root/.ssh anymore inside Yardstick but instead, we point the pod.yaml file directly to the key inside the pre_config directory. JIRA: DOVETAIL-655 Change-Id: I23156d8940194547363bcf9a5cc581aab0d5c640 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-18Bugfix the name of test case dovetail.neutron_l3_agentxudan1-1/+1
CI broken because of the wrong test case name. https://build.opnfv.org/ci/view/dovetail/job/dovetail-fuel-baremetal-proposed_tests-master/294/console Change-Id: I6674acf92c8f67ae2bd8dba082186a16211dcd8a Signed-off-by: xudan <xudan16@huawei.com>
2018-05-18Got an exception when test suite is emptyxudan2-9/+13
1. Return empty test case list when test suite is empty. 2. Don't quite understand why comment out all proposed_tests. 3. Move stress ping test case at the end of the list, because it may broken the message queue. Change-Id: Ib717fc91e3763be03f243f82e53679462ecdae04 Signed-off-by: xudan <xudan16@huawei.com>
2018-05-18Correct the names of sub test cases due to the new Tempest versionxudan9-55/+30
1. Change from Functest Danube to Fraser, the Tempest version it integrates is from 14.0.0 to 17.1.0. 2. Some sub test cases names have been changed. 3. Some are removed from tempest_smoke. 4. Some are new added into tempest_smoke. JIRA: DOVETAIL-657 Change-Id: Ie5611f5b7a5888bea3942667907341bd071d399d Signed-off-by: xudan <xudan16@huawei.com>
2018-05-17Adding OpenStack controller database HA test caseGeorg Kunz3-19/+27
Adding the OpenStack controller database HA test case (Yardstick tc090) to Dovetail and the proposed_tests test suite. JIRA: DOVETAIL-654 Change-Id: I4d20b907de8df2e9567891354b49e832cce13c7c Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-16Committer resignation of zenghuiGeorg Kunz1-4/+3
Removing zengshui from the list of committers. Change-Id: I5821b324445f21fc43b45164024a508de00ef5b5 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-15Updating the list of OS interop testsGeorg Kunz1-190/+184
Updating the list of test cases used in the OpenStack Interop test case to match i) the tests used in the 2017.09 guideline, and ii) the version of Tempest used by Functest We keep the explicit list of test cases in the test case definition file for now to enable Dovetail to verify the list of executed tests corresponds to the tests we want to run. JIRA: DOVETAIL-644 Change-Id: Ic627ef2bbad1dcfa5dd8773301c68dffb545a3b2 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-15Renaming test cases for better readabilityGeorg Kunz85-785/+304
This patchset proposes a new naming scheme for test cases. The ides is to replace the current numbered scheme <testarea>.tc<number> with: <testarea>.<capability/test case description> The motivation for this is that a the numbering does not convey any information about what the test case is doing and hides to which test case implementation in Functest or Yardstick a test case spec refers. Note: this patchset has not yet been tested in Dovetail itself. Its goal is start a discussion about the naming scheme. JIRA: DOVETAIL-647 Change-Id: If33b4bf88e451b372c2af10895ff92a87d9d961d Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-15Split tempest_smoke test casexudan10-135/+184
1. Total number of sub test cases is 119 2. Remove all 14 object_storage test cases which need swift 3. Remove 26 test cases already included in osinterop test area 4. Remove 21 test cases already included in ipv6 test area 5. Remove 2 scenario test cases already included in tempest test area 6. Split other 56 test cases into 8 groups Change-Id: I9eef20593ebf5f69347a2ec3a612c9ebc5a753db Signed-off-by: xudan <xudan16@huawei.com>
2018-05-14Switching to functest:fraser container tagGeorg Kunz1-1/+1
switching to functest:fraser from functest:opnfv-6.0.0 to incorporate bug fixes which have been backported to stable/fraser. Change-Id: Ieab9064ac0187b3460b09fe74861fc844b0b0195 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-11Adding documentation for API validation exemptionGeorg Kunz3-4/+29
Updating the user guide and OVP addendum to include references to the detailed document describing the API response validation exemption. JIRA: DOVETAIL-651 Change-Id: I4026f84f021704f0986b45680471e8fb711fc314 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-11Adding Neutron trunk port tests to proposed_tests suiteGeorg Kunz5-0/+32
Adding the Neutron trunk port tempest tests to Dovetail and the proposed_tests test suite. JIRA: DOVETAIL-648 Change-Id: Ia77980a245fd1bca6e5f0e567467432e12bf974e Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-11Fix the versions of openstack python client in Dovetail containerxudan1-2/+10
1. Now the openstack python clients installed are always the latest. 2. Fix them to certain versions according to the openstack requirements. 3. The first version openstack requirements provided is newton. https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=stable/newton JIRA: DOVETAIL-615 Change-Id: I107f8d3a4dbd95af3441c0ba410de37ba8128be3 Signed-off-by: xudan <xudan16@huawei.com>
2018-05-11Fixing SDNVPN tempest API testsGeorg Kunz2-2/+2
The code to run Tempest tests has changed in SDNVPN between Euphrates and Fraser, causing these tests to fail. Moreover, due to a bug in SDNVPN, not all bgpvpn API tests are currently being run. This is being addressed by https://gerrit.opnfv.org/gerrit/#/c/57299/ and needs to be cherrypicked to Fraser 6.1 for inclusion in OVP 2.0. Change-Id: I0a7575e8a9ebc23af5edaa26c1a9ce1e8149919c Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-08Improve the CLI of Dovetail toolxudan4-73/+86
The CLI of dovetail tool needs some improvement. 1. Add an option '--testcase' to support specify test case. 2. Add an option '--no-clean' to keep functest/yardstick/bottlenecks containers for debugging after running the test cases. 3. Improve the local report section. a. Remove some info that is not necessary b. Make the info more clearly JIRA: DOVETAIL-588 Change-Id: Ic769a6e2788b407c3df08ccd593a0ecdaa35e5ea Signed-off-by: xudan <xudan16@huawei.com>
2018-05-07Adding a tox environment for locally building docsGeorg Kunz4-1/+299
Run tox -edocs to locally build the Dovetail docs. This is quite useful because the default OPNFV docs preview build toolchain is broken for Dovetail. Change-Id: I0a55a701b96437a20aa09edc47bc3e49e32c0b87 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-07Decoupling Bottlenecks Docker config items from source codexudan5-106/+88
There are some special Docker setting items when creating Bottlenecks Docker Cntainers. Move the special settings to config files. Disable the function to push results to DB. JIRA: DOVETAIL-645 Change-Id: Icc033222edf74c0fe4853dede2794f4226fd4252 Signed-off-by: xudan <xudan16@huawei.com>
2018-05-04Remove local DBxudan15-521/+50
Remove local DB and can just report without launch local DB. Need to do a little modification with the web portal code. JIRA: DOVETAIL-593 Change-Id: I595db39cce4156596ee0522d4b7e642a23d78ab9 Signed-off-by: xudan <xudan16@huawei.com>
2018-05-01Integrating Functest Patrole test caseGeorg Kunz6-1/+36
Adding the new Patrole test case of Functest to Dovetail and to the proposed_tests test suite. JIRA: DOVETAIL-649 Change-Id: I2c3f583cdbd3af75cec020b0fe97f287d23498a5 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-04-26Extend web portal to show if API exemption was used during a test runchenjiankun3-1/+35
JIRA: DOVETAIL-634 The OVP web port needs to read from the test results if the strict API response validation exemption was used or not and show it correspondingly. Detailed description: As discussed on the Dovetail call on 2018-04-11, the web portal should show if API response validation was enabled or disabled. The information if API response validation was enabled or disabled is to be determined based on the dovetail.log file. If API response validation was disabled,** the log file contains the line "Strict API response validation DISABLED.". Upon uploading a result tar-package, the web portal should extract this information from the log file, store it in the database, and display it as follows: This indication shall be displayed along with the test summary on the Test Run Results page as either: "API response validation enabled" or "API response validation disabled" For clarification: API response validation is enabled by default. API response validation is disabled only if the --no-api-validation command line option is given. Change-Id: Ifacb1eb2127cc8dc6364c98dead1217aeedfd359 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2018-04-24The refstack file name has been changedxudan1-1/+1
The original refstack results file is refstack/refstack.log For Fraser, it has been changed into refstack/tempest.log Change-Id: I7d40abe0c5a366b5fbb1912ec58d15109f22c0d0 Signed-off-by: xudan <xudan16@huawei.com>
2018-04-17Switch to OPNFV Fraser releasexudan37-50/+44
1. Use Functest Fraser docker images whose tag is fraser 2. Use Yardstick stable docker images 3. Functest fraser has moved testcases.yaml file from functest/ci to xtesting/ci 4. Functest fraser has removed 'functest env prepare' command 5. Functest uses rally to execute osinterop test cases instead of refstack_client Change-Id: I81ebefe6ea959ca77c2c87f1d75ef56ac18c165f Signed-off-by: xudan <xudan16@huawei.com>
2018-04-17Implementation of API validaton exemption in dovetailGeorg Kunz6-3/+1683
This patch adds the ability to Dovetail to disable strict API response validation in Tempest-based test cases run by Functest. Corresponding updates of the web portal and backporting of this patch to OVP 1.0.0 (Danube) will follow. JIRA: DOVETAIL-633 Change-Id: Iace99ea1b6224ea907a2c3af8676e9285e6ad3ee Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-04-10Add functest healthcheck test suiteEddie Arrage6-0/+100
- Add Functest healthcheck tier to Dovetail as new test suite - Intention is that these test cases are not part of mandatory test cases in compliance program - Use for debug and SUT pre-flight check - May consider running tc001 (connection_check) at other test execution stages to determine if SUT is still functioning, as this test takes < 10 sec (more important as test complexity increases such as in vnf test tier) - Replacement for patch: https://gerrit.opnfv.org/gerrit/#/c/52461/ - Removed CI filters, as suggested Change-Id: I2e72931b27e43359d6250f6959a17af8526c5094 Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-04-10Decoupling Yardstick Docker config items from source codexudan5-71/+34
There are some special Docker setting items when creating Yardstick Docker Containers. 1. Remove the extra settings which are required by Yardstick Danube. 2. Remove the settings for pushing results to DB because Dovetail plan to disable this function. 3. Move the other special Functest settings to config files. JIRA: DOVETAIL-640 Change-Id: Ic2d90393e227d7886b14d805eaa307b7aef56e4a Signed-off-by: xudan <xudan16@huawei.com>
2018-04-10Support to specify docker image for each test casexudan11-29/+49
1. Currently each type of test cases will use one docker image. 2. For example, Functest test cases use opnfv/functest-restapi:euphrates Yardstick uses opnfv/yardstick:opnfv-5.1.0 3. It needs to support to specify the docker image for each test case. 4. For example, vping test cases use opnfv/functest-smoke:euphrates sdnvpn uses opnfv/functest-features:euphrates 5. The benefit of this is that these docker images are well verified by Functest CI and from Functest plan, they may not support functest-restapi docker image in its future release. JIRA: DOVETAIL-638 Change-Id: I8a30eea2233aeff809af2e241b6c939194397832 Signed-off-by: xudan <xudan16@huawei.com>