summaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)AuthorFilesLines
2018-05-15Updating the list of OS interop testsGeorg Kunz1-190/+184
Updating the list of test cases used in the OpenStack Interop test case to match i) the tests used in the 2017.09 guideline, and ii) the version of Tempest used by Functest We keep the explicit list of test cases in the test case definition file for now to enable Dovetail to verify the list of executed tests corresponds to the tests we want to run. JIRA: DOVETAIL-644 Change-Id: Ic627ef2bbad1dcfa5dd8773301c68dffb545a3b2 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-15Renaming test cases for better readabilityGeorg Kunz85-785/+304
This patchset proposes a new naming scheme for test cases. The ides is to replace the current numbered scheme <testarea>.tc<number> with: <testarea>.<capability/test case description> The motivation for this is that a the numbering does not convey any information about what the test case is doing and hides to which test case implementation in Functest or Yardstick a test case spec refers. Note: this patchset has not yet been tested in Dovetail itself. Its goal is start a discussion about the naming scheme. JIRA: DOVETAIL-647 Change-Id: If33b4bf88e451b372c2af10895ff92a87d9d961d Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-15Split tempest_smoke test casexudan10-135/+184
1. Total number of sub test cases is 119 2. Remove all 14 object_storage test cases which need swift 3. Remove 26 test cases already included in osinterop test area 4. Remove 21 test cases already included in ipv6 test area 5. Remove 2 scenario test cases already included in tempest test area 6. Split other 56 test cases into 8 groups Change-Id: I9eef20593ebf5f69347a2ec3a612c9ebc5a753db Signed-off-by: xudan <xudan16@huawei.com>
2018-05-14Switching to functest:fraser container tagGeorg Kunz1-1/+1
switching to functest:fraser from functest:opnfv-6.0.0 to incorporate bug fixes which have been backported to stable/fraser. Change-Id: Ieab9064ac0187b3460b09fe74861fc844b0b0195 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-11Adding documentation for API validation exemptionGeorg Kunz3-4/+29
Updating the user guide and OVP addendum to include references to the detailed document describing the API response validation exemption. JIRA: DOVETAIL-651 Change-Id: I4026f84f021704f0986b45680471e8fb711fc314 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-11Adding Neutron trunk port tests to proposed_tests suiteGeorg Kunz5-0/+32
Adding the Neutron trunk port tempest tests to Dovetail and the proposed_tests test suite. JIRA: DOVETAIL-648 Change-Id: Ia77980a245fd1bca6e5f0e567467432e12bf974e Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-11Fix the versions of openstack python client in Dovetail containerxudan1-2/+10
1. Now the openstack python clients installed are always the latest. 2. Fix them to certain versions according to the openstack requirements. 3. The first version openstack requirements provided is newton. https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=stable/newton JIRA: DOVETAIL-615 Change-Id: I107f8d3a4dbd95af3441c0ba410de37ba8128be3 Signed-off-by: xudan <xudan16@huawei.com>
2018-05-11Fixing SDNVPN tempest API testsGeorg Kunz2-2/+2
The code to run Tempest tests has changed in SDNVPN between Euphrates and Fraser, causing these tests to fail. Moreover, due to a bug in SDNVPN, not all bgpvpn API tests are currently being run. This is being addressed by https://gerrit.opnfv.org/gerrit/#/c/57299/ and needs to be cherrypicked to Fraser 6.1 for inclusion in OVP 2.0. Change-Id: I0a7575e8a9ebc23af5edaa26c1a9ce1e8149919c Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-08Improve the CLI of Dovetail toolxudan4-73/+86
The CLI of dovetail tool needs some improvement. 1. Add an option '--testcase' to support specify test case. 2. Add an option '--no-clean' to keep functest/yardstick/bottlenecks containers for debugging after running the test cases. 3. Improve the local report section. a. Remove some info that is not necessary b. Make the info more clearly JIRA: DOVETAIL-588 Change-Id: Ic769a6e2788b407c3df08ccd593a0ecdaa35e5ea Signed-off-by: xudan <xudan16@huawei.com>
2018-05-07Adding a tox environment for locally building docsGeorg Kunz4-1/+299
Run tox -edocs to locally build the Dovetail docs. This is quite useful because the default OPNFV docs preview build toolchain is broken for Dovetail. Change-Id: I0a55a701b96437a20aa09edc47bc3e49e32c0b87 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-07Decoupling Bottlenecks Docker config items from source codexudan5-106/+88
There are some special Docker setting items when creating Bottlenecks Docker Cntainers. Move the special settings to config files. Disable the function to push results to DB. JIRA: DOVETAIL-645 Change-Id: Icc033222edf74c0fe4853dede2794f4226fd4252 Signed-off-by: xudan <xudan16@huawei.com>
2018-05-04Remove local DBxudan15-521/+50
Remove local DB and can just report without launch local DB. Need to do a little modification with the web portal code. JIRA: DOVETAIL-593 Change-Id: I595db39cce4156596ee0522d4b7e642a23d78ab9 Signed-off-by: xudan <xudan16@huawei.com>
2018-05-01Integrating Functest Patrole test caseGeorg Kunz6-1/+36
Adding the new Patrole test case of Functest to Dovetail and to the proposed_tests test suite. JIRA: DOVETAIL-649 Change-Id: I2c3f583cdbd3af75cec020b0fe97f287d23498a5 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-04-26Extend web portal to show if API exemption was used during a test runchenjiankun3-1/+35
JIRA: DOVETAIL-634 The OVP web port needs to read from the test results if the strict API response validation exemption was used or not and show it correspondingly. Detailed description: As discussed on the Dovetail call on 2018-04-11, the web portal should show if API response validation was enabled or disabled. The information if API response validation was enabled or disabled is to be determined based on the dovetail.log file. If API response validation was disabled,** the log file contains the line "Strict API response validation DISABLED.". Upon uploading a result tar-package, the web portal should extract this information from the log file, store it in the database, and display it as follows: This indication shall be displayed along with the test summary on the Test Run Results page as either: "API response validation enabled" or "API response validation disabled" For clarification: API response validation is enabled by default. API response validation is disabled only if the --no-api-validation command line option is given. Change-Id: Ifacb1eb2127cc8dc6364c98dead1217aeedfd359 Signed-off-by: chenjiankun <chenjiankun1@huawei.com>
2018-04-24The refstack file name has been changedxudan1-1/+1
The original refstack results file is refstack/refstack.log For Fraser, it has been changed into refstack/tempest.log Change-Id: I7d40abe0c5a366b5fbb1912ec58d15109f22c0d0 Signed-off-by: xudan <xudan16@huawei.com>
2018-04-17Switch to OPNFV Fraser releasexudan37-50/+44
1. Use Functest Fraser docker images whose tag is fraser 2. Use Yardstick stable docker images 3. Functest fraser has moved testcases.yaml file from functest/ci to xtesting/ci 4. Functest fraser has removed 'functest env prepare' command 5. Functest uses rally to execute osinterop test cases instead of refstack_client Change-Id: I81ebefe6ea959ca77c2c87f1d75ef56ac18c165f Signed-off-by: xudan <xudan16@huawei.com>
2018-04-17Implementation of API validaton exemption in dovetailGeorg Kunz6-3/+1683
This patch adds the ability to Dovetail to disable strict API response validation in Tempest-based test cases run by Functest. Corresponding updates of the web portal and backporting of this patch to OVP 1.0.0 (Danube) will follow. JIRA: DOVETAIL-633 Change-Id: Iace99ea1b6224ea907a2c3af8676e9285e6ad3ee Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-04-10Add functest healthcheck test suiteEddie Arrage6-0/+100
- Add Functest healthcheck tier to Dovetail as new test suite - Intention is that these test cases are not part of mandatory test cases in compliance program - Use for debug and SUT pre-flight check - May consider running tc001 (connection_check) at other test execution stages to determine if SUT is still functioning, as this test takes < 10 sec (more important as test complexity increases such as in vnf test tier) - Replacement for patch: https://gerrit.opnfv.org/gerrit/#/c/52461/ - Removed CI filters, as suggested Change-Id: I2e72931b27e43359d6250f6959a17af8526c5094 Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-04-10Decoupling Yardstick Docker config items from source codexudan5-71/+34
There are some special Docker setting items when creating Yardstick Docker Containers. 1. Remove the extra settings which are required by Yardstick Danube. 2. Remove the settings for pushing results to DB because Dovetail plan to disable this function. 3. Move the other special Functest settings to config files. JIRA: DOVETAIL-640 Change-Id: Ic2d90393e227d7886b14d805eaa307b7aef56e4a Signed-off-by: xudan <xudan16@huawei.com>
2018-04-10Support to specify docker image for each test casexudan11-29/+49
1. Currently each type of test cases will use one docker image. 2. For example, Functest test cases use opnfv/functest-restapi:euphrates Yardstick uses opnfv/yardstick:opnfv-5.1.0 3. It needs to support to specify the docker image for each test case. 4. For example, vping test cases use opnfv/functest-smoke:euphrates sdnvpn uses opnfv/functest-features:euphrates 5. The benefit of this is that these docker images are well verified by Functest CI and from Functest plan, they may not support functest-restapi docker image in its future release. JIRA: DOVETAIL-638 Change-Id: I8a30eea2233aeff809af2e241b6c939194397832 Signed-off-by: xudan <xudan16@huawei.com>
2018-04-08Removing outdated patch for Yardstick HA testsGeorg Kunz1-133/+0
This patch was needed to modify the timeout values of the HA test cases in Yardstick for the Danube release. In Euphrates, the timeout values have been fixed upstream, so the patch is not needed anymore. The code for applying the patch was already removed in https://gerrit.opnfv.org/gerrit/#/c/51997/ Change-Id: I67cceb57c063378d77f6cbc720bc6464d2be4aed Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-04-06Fixed typos in the license headerrpaik1-1/+1
Change-Id: Ib2aae63dd7576a8458f068250e9b6dda8678a1fd Signed-off-by: rpaik <rpaik@linuxfoundation.org>
2018-04-04Decoupling Functest Docker config items from source codexudan10-43/+24
There are some special Docker setting item when creating Functest Containers. 1. Remove the extra settings which are required by Functest Danube. 2. Remove the settings for pushing results to DB because Dovetail plans to disable this function. 3. Move the other special Functest settings to config files. JIRA: DOVETAIL-639 Change-Id: Ibc4090b5c0aafdf9ed64e65a9a92e34a12a16df5 Signed-off-by: xudan <xudan16@huawei.com>
2018-03-29Integrate dovetail with ONAPMoshe13-3/+244
Change-Id: I6a1fa5f27df2f5127eb00156b3135dc79caf5e77 Issue-ID: DOVETAIL-629 Signed-off-by: Moshe <moshehoa@amdocs.com>
2018-03-20Exemption process for disabling strict API validationGeorg Kunz1-0/+183
This document describes an exemption process for disabling strict API response validation. JIRA: DOVETAIL-617 Change-Id: I690efdcae7dd15a6c0781a837a62677904836fca Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-03-15Disable restricts of sdnvpn test casesxudan7-8/+12
1. The original testcases.yaml file of Functest restricts sdnvpn test cases to run on Fuel, Apex or Netvirt on bgpvpn scenario. https://github.com/opnfv/functest/blob/master/functest/ci/testcases.yaml#L278 2. Dovetail has a testcases.yaml file to allow sdnvpn test cases run on any commercial SUTs. https://github.com/opnfv/dovetail/blob/master/etc/userconfig/testcases.yaml#L113 Change-Id: I0d5ed5b22e55dc3dc0fe584dc57b28239334b7de Signed-off-by: xudan <xudan16@huawei.com>
2018-03-15substitute image copy with docker volumeSerenaFeng7-23/+50
Change-Id: Ie470d03bd168c909fd02c316523049c67a6bb095 Signed-off-by: SerenaFeng <feng.xiaowei@zte.com.cn>
2018-03-14Bugfix: Failed to get the hardware infoxudan1-3/+4
The directory of userconfig has changed from /home/opnfv/dovetail/dovetail/userconfig to /etc/dovetail/userconfig. Change-Id: Id3e8b1f3e9299e9263a4adf3456b6733fdb62e11 Signed-off-by: xudan <xudan16@huawei.com>
2018-03-14Add process info to pod.yaml to specify the info of a special processxudan5-8/+125
1. Many HA test cases try to kill special processes with their name. 2. Then they check corresponding openstack services and the status of processes' recovery. 3. For different SUTs, the names of the processes and the hosts' names may be different. 4. So there is a requirement for Dovetail tool to provide one way to allow users to specify all the infos about process name and its host name. 5. All process infos can be added into file DOVETAIL_HOME/user_config/pod.yaml 6. The infos include 'attack_process' and 'attack_host' for each HA test cases. 7. If not given in this file, will use Yardtsick default values. JIRA: DOVETAIL-627 Change-Id: I83cee991f72a8685080ed562597c70d73002623a Signed-off-by: xudan <xudan16@huawei.com>
2018-03-06Separate valid docker tags for functest and yardstickGeorg Kunz4-19/+22
Specifying separate valid docker tags for functest and yardstick for input validation. DOVETAIL-592 Change-Id: I3365cdbd9ab521bf1368750c3f34b6863b3f7ec8 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-03-06Bugfix: it crashes when check_tc_result returns Nonexudan1-2/+6
The function 'check_tc_result' may return 'None'. In this case it will throw an exception because of https://github.com/opnfv/dovetail/blob/master/dovetail/run.py#L65 The exception is like Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dovetail/run.py", line 339, in <module> main() File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 716, in _call_ return self.main(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 696, in main rv = self.invoke(ctx) File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 889, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 534, in invoke return callback(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/dovetail/run.py", line 304, in main duration = run_test(testsuite_yaml, testarea, logger, kwargs) File "/usr/local/lib/python2.7/dist-packages/dovetail/run.py", line 65, in run_test if stop_on_fail['criteria'] == "FAIL" and kwargs['stop']: TypeError: 'NoneType' object has no attribute '_getitem_' JIRA: DOVETAIL-622 Change-Id: I9e612b380c3dfb148692074a3bf8eb31875213ac Signed-off-by: xudan <xudan16@huawei.com>
2018-03-06simplify file existence check with os.path.isfile()SerenaFeng1-16/+11
Change-Id: I89d15e18c588c27ab41596119d5288cba6e4e02d Signed-off-by: SerenaFeng <feng.xiaowei@zte.com.cn>
2018-03-06leverage data_files to eliminate so many hardcoded abs_path_seekingsSerenaFeng87-56/+61
1. mv conf/ userconfig/ compliance/ patch to etc/ 2. after installation etc/ will be put under /etc/dovetail 3. leveraging /etc/dovetail in coding Change-Id: Icf9cb00ff7fc7cf35755f60d3e258b23e2c70839 Signed-off-by: SerenaFeng <feng.xiaowei@zte.com.cn>
2018-03-02Adding SDNVPN tempest test cases to proposed_tests suiteGeorg Kunz3-0/+23
Adding the SDNVPN Tempest API test cases of the SDNVPN project to the proposed_tests suite to allow for running these tests in the CI pipeline for validating their suitability for inclusion in the OVP Euphrates release. JIRA: DOVETAIL-608 Change-Id: Ic12db1584a2540a14979094c65fc128c6020ecea Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-03-01Remove Whitespace from CVP Docker READMETrevor Bramwell1-2/+2
Change-Id: Ic880258ef8ffbcbdcf3452419dd385227567a5a3 Signed-off-by: Trevor Bramwell <tbramwell@linuxfoundation.org>
2018-03-01Bugfix: Dovetail tool will crash when hosts.yaml file with incorrect formatxudan2-16/+35
If the hosts.yaml file is like, — hosts_info: 192.168.141.101: it will raise an exception, File "/usr/local/lib/python2.7/dist-packages/dovetail/utils/dovetail_utils.py", line 227, in add_hosts_info names=hostnames) File "/usr/local/lib/python2.7/dist-packages/python_hosts/hosts.py", line 54, in _init_ raise Exception('Address and Name(s) must be specified.') Exception: Address and Name(s) must be specified. If the hosts.yaml file is empty, it will crash with, File "/usr/local/lib/python2.7/dist-packages/dovetail/container.py", line 174, in create if hosts_yaml['hosts_info']: TypeError: 'NoneType' object has no attribute '_getitem_' JIRA: DOVETAIL-621 Change-Id: I547928e514b3294c048379459c07df89879fbb03 Signed-off-by: xudan <xudan16@huawei.com>
2018-02-28Removing old INFO fileGeorg Kunz1-26/+0
The INFO file has been replaced by the INFO.yaml file. Change-Id: I818697134cbb380a318237fdf593d7b6bf95ec04 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-02-28Removal of Wenjing Chu from the list of committersGeorg Kunz1-4/+3
https://lists.opnfv.org/pipermail/opnfv-tech-discuss/2018-February/020497.html Change-Id: Ic2b536551d919e9b66f41ba00dd6dc8b71d93f7b Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-02-28Update CVP Docker Compose ConfigurationTrevor Bramwell5-11/+44
- Uses extra files config.env, vhost.env to hold environment variables for docker configuration. - Provides samples for these config that can be copied and modified to fit deployment requirements. - Updates the docker images to the newest releases. Change-Id: Ibe9b8597edfb38940297433708497c244200d0af Signed-off-by: Trevor Bramwell <tbramwell@linuxfoundation.org>
2018-02-24Adapt Bottlenecks Euphrates frameworkxudan6-16/+35
Bottlenecks stress test needs some adaptions about Bottlenecks Euphrates release. JIRA: DOVETAIL-587 Change-Id: Ia447143c8f08924ac1b190ee0173f39e462c5dfa Signed-off-by: xudan <xudan16@huawei.com>
2018-02-24Removal of Dave Neary from the list of committersGeorg Kunz1-4/+3
Change-Id: I06b1e53bca44f5486a64977edef8b0e53072c625 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-02-24Add Functest cloudify_ims test case into proposed_tests test suitexudan4-8/+27
Add cloudify_ims test case into proposed_tests test suite. Then this test case can be run by Dovetail on CI. JIRA: DOVETAIL-603 Change-Id: I5ab832002388f0916e04beef9d630de2cb37df59 Signed-off-by: xudan <xudan16@huawei.com>
2018-02-23Update OVP logo and branding guide linkEddie Arrage2-1/+1
- Update the OPNFV Verified logo for OVP portal home page - Update branding guide link in OVP portal left side navigation Change-Id: I77ead3e68b34d30d55fd07f0e4a176c5ae71852e Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-02-23Committer promotion Dan Xu and removal of Leo WangGeorg Kunz1-4/+10
Contributions Dan Xu: Dan Xu has been almost exclusively doing the development for the Dovetail tool, CI integration, and the practical validation of test case candidates. https://gerrit.opnfv.org/gerrit/#/q/owner:%22Dan+Xu+%253Cxudan16%2540huawei.com%253E%22 Leo Wang has announced that he is stepping down as committer from the project. Thank you for your contributions to the project. https://lists.opnfv.org/pipermail/opnfv-tsc/2018-February/004130.html Change-Id: I14032657b057bdd3da1f9c3514b3c27e34dbafc0 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-02-19Add 3 new HA test cases into proposed_tests test suitexudan7-24/+63
Add 3 new HA test cases into proposed_tests test suite and running them on CI. For the current 8 HA test cases and 3 other HA test cases plans to add, do some adaptions for the Yardstick Euphrates release. 1. Don't need the bugfix patch anymore. 2. The data format has changed, and has done some adaptions. 3. For Yardstick Euphrates, if setting OS_INSECURE=True, it needs to set OS_CACERT=False 4. Test case tc011 needs a image and flavor to create the stack. JIRA: DOVETAIL-614 JIRA: DOVETAIL-586 Change-Id: I9da8c8316246b9edee0d81ad18422ae6018becaf Signed-off-by: xudan <xudan16@huawei.com>
2018-02-16Add a function to check the cacert filexudan2-20/+29
1. Currently it won't pass the cacert file to other containers if the OS_AUTH_URL is http not https. 2. However, even though the OS_AUTH_URL is http, it still needs cacert file somewhere for Functest and Rally. 3. This problem is found on Fuel Euphrates. 4. Add a function to check whether the cacert file exists. 5. If exists, pass it to testing project containers. JIRA: DOVETAIL-616 Change-Id: Ied7bcc72e8f1c738bbce32c18096ca13641d3cd7 Signed-off-by: xudan <xudan16@huawei.com>
2018-02-16Add Functest test case snaps_smoke into proposed_tests test suitexudan4-56/+21
Add snaps_smoke into proposed_tests test suite. Then this test case can be run by Dovetail on CI. JIRA: DOVETAIL-596 Change-Id: I8fc4bb656bcac551ae4b693dbc4d817b665c6111 Signed-off-by: xudan <xudan16@huawei.com>
2018-02-14Add stop on failure logic and CLI optionEddie Arrage2-7/+18
- Improve test execution control by adding stop on failure flag - Modify test check function to return value regardless of whether report or local file option is employed - Modify run test loop to check for failure after every test case if stop flag set Change-Id: I27a4f3aaaa8b8cb8c52eb3f092f52048980abb44 Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-02-11Auto Generated INFO.yaml fileagardner1-0/+73
INFO.yaml meant to be machine readable. This file was auto generated from an ldap lookup and your projects INFO file. Meeting times are not currently filled out. please ammend this patchset to complete Going forward comitter promotions should be done against this file. PTLs: Must be merged for you to gain delete access on testresults.opnfv.org If project_lead: is missing (my script could not parse) please amend with same from commiters section eg: project_lead: &opnfv_dovetail_ptl name: 'Lead Name' email: 'lead email' id: 'lead lfid' company: 'lead company' Change-Id: Ie151ba96dd6b6c7d0d8e74e1462e35133a7c235f Signed-off-by: agardner <agardner@linuxfoundation.org>
2018-02-08Add ablity to config EXTERNAL_NETWORK in env_config.shEddie Arrage1-23/+28
- EXTERNAL_NETWORK env variable can be explicitly set - If set, the value will be used in Yardstack configuration - Otherwise, previous method of using output from 'openstack network list' will still be employed Change-Id: I9d8b4a69ff3f5093c034823cc95e92e151bfade6 Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>