summaryrefslogtreecommitdiffstats
path: root/dovetail/testcase.py
AgeCommit message (Collapse)AuthorFilesLines
2019-09-16Move the proj to python3Panagiotis Karalis1-8/+9
The aim of this patch is to upgrade python version of this project from python2 to python3. Signed-off-by: Panagiotis Karalis <panos.pkaralis@gmail.com> Change-Id: I3d3ef01176fda1b23a0542a24625be2f3368c40e
2019-09-02Add portal_key_file into all test case yml filesxudan1-0/+3
Change-Id: If111a8cc56c31ec3e22b090aca92ff56d41f7e01 Signed-off-by: xudan <xudan16@huawei.com>
2019-06-12Add RESTful APIs into Dovetail containerxudan1-0/+2
JIRA: DOVETAIL-774 Change-Id: Ifb934eaeda52c1d316d80d9bb9c1890ffb71049c Signed-off-by: xudan <xudan16@huawei.com>
2019-06-10Bugfix: Fix Dovetail 2 bugs found with CI resultsxudan1-4/+0
There are 2 bugs found with CI results of functest.tempest.neutron_tempest_plugin_api and all HA test cases. Yardtsick conf file does't mapping the keyfile to Yardstick container, so all HA test cases got SSHERROR. Also remove some useless debug msg to make dovetail.log more concise. Change-Id: I63c12d1b91ed5f8f8a893689384685ca9215f367 Signed-off-by: xudan <xudan16@huawei.com>
2019-04-17Failed to upload results to ONAP portalPanagiotis Karalis1-0/+3
The results.json file generated by running 'dovetail run --testsuite xxx' is different from the one generated by running 'dovetail run --testcase xxx'. The former can be uploaded to ONAP portal successfully but the latter one can't. The main reason is that the latter one lacking of 'vnf_type'. Keep the results.json with the same format for both running one test suite or several test cases. JIRA: DOVETAIL-769 Change-Id: I46af84f9f229f9e601439f68c9ed9df9477d002b Signed-off-by: Panagiotis Karalis <pkaralis@intracom-telecom.com>
2019-02-20Remove deprecated ONAP lifecycle test caseStamatis Katsaounis1-10/+0
This patch removes the ONAP lifecycle deprecated test case. Furthermore, it clears helper functions as well as unwanted configuration. Change-Id: Ib958457cf120ffb641a72be028d3a031741c44c6 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-02-01Add ONAP VVP Heat Template validation testsStamatis Katsaounis1-1/+11
Change-Id: I9a3d56932ce41191901381831013768d1c57a749 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-01-03Add ONAP VNF SDK verification testsxudan1-1/+11
Please refer to env_config.sh.onap.sample when testing it. The guide of preparing local env for onap package validation can be found here https://gerrit.onap.org/r/#/c/75119 Change-Id: I39d6b11544847756126623a7eb3b953dc800c470 Signed-off-by: xudan <xudan16@huawei.com>
2018-12-20Simplify project conf filesxudan1-13/+11
1. use copy commands instead of pre_copy 2. remove redundant items in project conf files 3. make the volume mapping clearer Change-Id: I6c6aa58fac65d7e40105e0a54f6544ee5c47db31 Signed-off-by: xudan <xudan16@huawei.com>
2018-12-11Enable ShellRunnerxudan1-2/+0
1. fix the bug 'ShellRunner' object has no attribute 'archive_logs' 2. remove the testarea_supported because it's not very necessary for the test cases themselves and can make it a little more simple to add a test case with new test area JIRA: DOVETAIL-750 Change-Id: I71ee74615200376adca2a0db040753e5fce329bc Signed-off-by: xudan <xudan16@huawei.com>
2018-11-23Integrate k8s test cassesxudan1-2/+12
In order to integrate k8s test cases, Dovetail framework should do some refactor and make it more general for k8s test cases as well as ONAP ones. Integrate 2 k8s test cases. JIRA: DOVETAIL-748 Change-Id: Ibd87754ffb5fb29f6b4ce79232af860c2ed2da9c Signed-off-by: xudan <xudan16@huawei.com>
2018-11-15Add missing unit tests for testcase fileStamatis Katsaounis1-2/+1
JIRA: DOVETAIL-724 This patch adds unit tests for Tescases classes methods of Dovetail which were missing. Change-Id: Ie1b665249ce8afaa76b59b0dbe7f132e404a5569 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2018-11-01Update classes in testcase.pyxudan1-45/+5
1. Remove the functions which are not called anywhere 2. Change some class methods to be static methods and instance methods Change-Id: I2e754cdff61aa34ffe899e352653ff767d4ac69d Signed-off-by: xudan <xudan16@huawei.com>
2018-10-31CI verify jobs failed due to the update of toxxudan1-2/+2
The LF pod used to run the CI verify jobs have updated the version of tox. So the CI verify jobs failed. See https://build.opnfv.org/ci/view/dovetail/job/dovetail-merge-master/476/console Fix the python format errors Ignore error 'W503 line break before binary operator'. Change-Id: Ic2df070447bbea39c877ce7a394d733a95064bb1 Signed-off-by: xudan <xudan16@huawei.com>
2018-07-24Making the results file a proper json fileGeorg Kunz1-1/+13
The results.json file is currently not a properly formated json file, but contains one json structure per line/testcase. Whit this change, all test results are being collected at runtime and then written to a properly formated json file at the end of a test run. Moreover, this patch introduces 'mandatory' and 'optional' sections to the test suites definition files in order to clearly indicate if a test is part of the corresponding test scope. The web portal needs to be adapted accordingly to be able to read the new results file. JIRA: DOVETAIL-672 Change-Id: Id0706c7e82622fc9c2c2fece26b2d6c07c1580be Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-07-22Remove functest_testsuite and functest_testcase in dovetail_config.ymlxudan1-4/+2
The 2 sections 'functest_testsuite' and 'functest_testcase' are used by all Functest test cases. They aim to parse the results of Tempest sub test cases. Remove them, and check the sub test case list, 1. if it's empty, that means no need to check sub test cases 2. if it is not empty, check the sub test cases one by one. Additional, remove some useless keys in dovetail_config.yml JIRA: DOVETAIL-658 Change-Id: Ic829264b0c8eaedd8e2b3fa55b38a34644482c20 Signed-off-by: xudan <xudan16@huawei.com>
2018-07-17Update test suite ovp.nextxudan1-12/+35
Update test suite ovp.next according to the latest OVP scope. Modify Dovetail daily jobs to run all the scope. There should be 2 jobs. One for all mandatory test cases and the other for all optional ones. That's mainly because of that the total executed time will be too large (more than 300 minutes). Then the job will always failed because of time out. It's hard to enlarge the time because it's already larger than common 3 hours. Split it into 2 jobs can avoid the time out issue and make the results clearer for reviewing. JIRA: DOVETAIL-694 Change-Id: Ie0ea6221868941781af1477f7c7719f7cb4351a4 Signed-off-by: xudan <xudan16@huawei.com>
2018-06-29Update licence headers to comply with guidelinesGeorg Kunz1-1/+2
Updating the licence headers to comply with the community guidelines [1] and anteater checks. [1] https://wiki.opnfv.org/display/DEV/Contribution+Guidelines#ContributionGuidelines-License Change-Id: Idec787e32daf76c0b1bd6108977b78828125da5e Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-27Archive all test case result filesxudan1-1/+1
1. All detailed tempest test cases results are recorded with file tempest-report.html rather than tempest.log now. 2. Archive all results files include functest.log, yardstick.log ... for each test case. Otherwise, they will all be in one log file. 3. Support to set the source files and dest files need to be archived with the test case config files rather than hardcode with the source code. 4. Give the file which used to be parse the results (PASS/FAIL) with the test case config files rather than hardcode with the source code. Change-Id: I30f435d91ab90cf20def47007f177fe98187484d Signed-off-by: xudan <xudan16@huawei.com>
2018-05-18Got an exception when test suite is emptyxudan1-1/+5
1. Return empty test case list when test suite is empty. 2. Don't quite understand why comment out all proposed_tests. 3. Move stress ping test case at the end of the list, because it may broken the message queue. Change-Id: Ib717fc91e3763be03f243f82e53679462ecdae04 Signed-off-by: xudan <xudan16@huawei.com>
2018-05-07Decoupling Bottlenecks Docker config items from source codexudan1-8/+0
There are some special Docker setting items when creating Bottlenecks Docker Cntainers. Move the special settings to config files. Disable the function to push results to DB. JIRA: DOVETAIL-645 Change-Id: Icc033222edf74c0fe4853dede2794f4226fd4252 Signed-off-by: xudan <xudan16@huawei.com>
2018-04-17Implementation of API validaton exemption in dovetailGeorg Kunz1-0/+19
This patch adds the ability to Dovetail to disable strict API response validation in Tempest-based test cases run by Functest. Corresponding updates of the web portal and backporting of this patch to OVP 1.0.0 (Danube) will follow. JIRA: DOVETAIL-633 Change-Id: Iace99ea1b6224ea907a2c3af8676e9285e6ad3ee Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-03-29Integrate dovetail with ONAPMoshe1-0/+10
Change-Id: I6a1fa5f27df2f5127eb00156b3135dc79caf5e77 Issue-ID: DOVETAIL-629 Signed-off-by: Moshe <moshehoa@amdocs.com>
2018-03-06leverage data_files to eliminate so many hardcoded abs_path_seekingsSerenaFeng1-9/+6
1. mv conf/ userconfig/ compliance/ patch to etc/ 2. after installation etc/ will be put under /etc/dovetail 3. leveraging /etc/dovetail in coding Change-Id: Icf9cb00ff7fc7cf35755f60d3e258b23e2c70839 Signed-off-by: SerenaFeng <feng.xiaowei@zte.com.cn>
2017-09-25Support to run mandatory or optional test cases separatelyxudan1-0/+38
1. Currently, dovetail can run a test suite or a test area in that test suite. 2. The test areas in one test suite are not divided into optional and mandatory. 3. Split them and support to run just mandatory or optional. 4. Support to run multiple test areas. JIRA: DOVETAIL-505 Change-Id: I42cd7b4e11c3e3674c806e9bc999b782bf5c85c6 Signed-off-by: xudan <xudan16@huawei.com>
2017-08-28Bugfix: functest only keeps the last test case's logxudan1-0/+1
JIRA: DOVETAIL-490 1. Dovetail just keeps the last tempest/defcore/ipv6/bgpvpn test's log file. 2. All log files should be kept, so users and developers can know the reason of failed test cases. 3. tempest logs will be stored in directory tempest_logs/ 4. defcore logs will be stored in directory defcore_logs/ 5. IPv6 logs will be stored in directory ipv6_logs/ 6. bgpvpn logs will be stored in directory bgpvpn_logs/ 7. All vping logs are in functest.log Change-Id: I79eae79cb32d65ce1ada3dd6f4c6dfd3945fc512 Signed-off-by: xudan <xudan16@huawei.com>
2017-08-03Add resiliency test caes into proposed_testsxudan1-1/+19
JIRA: DOVETAIL-474 Add Bottlenecks resiliency test case "posca_factor_ping" into proposed_tests. Change-Id: I76aea651014fb4722ddae07559f417d3f840eee2 Signed-off-by: xudan <xudan16@huawei.com>
2017-06-28Modify Dovetail logsxudan1-13/+16
JIRA: DOVETAIL-450 Change-Id: I795679c9559d1bead3aecefff28d21b6567aac2f Signed-off-by: xudan <xudan16@huawei.com>
2017-05-24SDNVPN test cases failed in CIxudan1-21/+7
JIRA: DOVETAIL-439 1. SDNVPN test cases run in CI failed, because the file /home/opnfv/repos/sdnvpn/sdnvpn/test/functest/config.yaml is empty. 2. copy the file userconfig/sdnvpn_config_testcase1/2/4/8.yaml to /home/opnfv/repos/sdnvpn/sdnvpn/test/functest/config.yaml in Functest container Change-Id: Ie904f83c755efc0d9e0abe5dcd44fe4a5056123e Signed-off-by: xudan <xudan16@huawei.com>
2017-04-21use Functest tempest_custom test cases to run all Ipv6 test casesxudan1-1/+1
JIRA: DOVETAIL-401 1. modify all Ipv6 test cases, change tempest_smoke_serial/tempest_full_parallel into tempest_custom, add ID of each sub-testcases. 2. Some sub-testcases in ipv6.tc002, ipv6.tc003, ipv6.tc004 and ipv6.tc005 can not be found in the full tempest test case list. 2. disable testcases' script_result_acquired to enable tempest_custom run multiple times. 3. use build_tag-testcase_name(such as daily-master-79a6e392-28bb-4c40-9d88-a03bed99da2d-dovetail.ipv6.tc025) as the unique ID of each testcase. 4. comment out the progress bar, because it has some problems when print the report text. 5. Functest has changed the TEST_DB_URL from http://<testapi>:<port>/apv/v1 to http://<testapi>:<port>/apv/v1/results. 6. Functest has changed the TEST_DB_URL to file://<file_path>/<file_name>. see https://gerrit.opnfv.org/gerrit/#/c/33511/ Change-Id: I524f6702377f215f526158214842481eb44565be Signed-off-by: xudan <xudan16@huawei.com>
2017-04-12dovetail tool: bugfix: test cases are erroneously judgedxudan1-4/+5
JIRA: DOVETAIL-375 JIRA: DOVETAIL-390 1. defcore test cases are erroneously judged because of: a) defcore results json data lack some keys b) some sub-testcases' names are the same as the prefix of another sub-testcases. 2. this patch removes the extra keys and judge sub-testcases accurately. 3. remove "store_type" in functest_config.yml and yardstick_config.yml. 4. change some log levels from info to debug. Change-Id: I5f126e70309409cac66ef9a0909f1573d325098b Signed-off-by: xudan <xudan16@huawei.com>
2017-04-01refstack result parseMatthewLi1-0/+11
JIRA: DOVETAIL-375 1, together with other testsuite from functest, such as tempest etc. this should merge after https://gerrit.opnfv.org/gerrit/#/c/31313/ 2, delete some unnecessary definition 3, bugfix of docker cp inside container, see https://build.opnfv.org/ci/view/dovetail/job/dovetail-compass-baremetal-debug-master/361/console 4, disable tempest testcaes since changed to refstack_defcore Change-Id: I67503e348af4a3f697f1b4106bfe81c05add8aa0 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-03-30dovetail tool: refstack testcase integrationMatthewLi1-0/+25
JIRA: DOVETAIL-370 Change-Id: I60c9f431358f848e24abcb865c4b2ca8c3b2d843 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-02-17dovetail tool: skip testcase status addedMatthewLi1-1/+1
JIRA: DOVETAIL-344 1) testcase status, True/False --> PASS/SKIP/FAIL, so skipped testcase in tempest can be figured out details can be seen in https://etherpad.opnfv.org/p/dovetail_report 2) some variable name changed more readable, such as passed-->testcase_passed failed--> prepare_failed 3) "dovetail report" minor amend according to 1) Change-Id: Ibe8e50693fc474d78e7d0fb89257f34703248579 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-02-08Merge "dovetail tool: use python new-style classes which inherit from object"hongbo tian1-1/+1
2017-02-03dovetail tool: bugfix: cannot load pre/post condition from config filexudan1-10/+20
JIRA: DOVETAIL-342 Change-Id: If98905b6724c0a990d789a7ce914aeecac72d7c4 Signed-off-by: xudan <xudan16@huawei.com>
2017-01-20dovetail tool: use python new-style classes which inherit from objectxudan1-1/+1
JIRA: DOVETAIL-187 Change-Id: Id6fa50a3a3cec7a18d27ca8feb383b53836a339b Signed-off-by: xudan <xudan16@huawei.com>
2017-01-13dovetail tool: command lineMatthewLi1-1/+5
JIRA: DOVETAIL-173 details please see https://wiki.opnfv.org/display/dovetail/Dovetail+Command+Line Change-Id: Iff04b0df8c4e6310d35a45b9c8ba3c7b3b5e1105 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-12-26dovetail tool: reorganize function prepare_cmd in testcase.pyxudan1-25/+23
1. move the common statements in prepare_cmd of class FunctestTestcase and Testcase into parse_cmd. JIRA: DOVETAIL-163 Change-Id: I61a0a24b47cfc9e8d49ca9c62fafbac7cce21111 Signed-off-by: xudan <xudan16@huawei.com>
2016-12-22[dovetail tool]check and get results for each cmdLeo Wang1-7/+23
JIRA: DOVETAIL-166 Check the results of each cmds executed in test case 1. the results of pre_condition, post_condition, cmds need to be checked, so it can get a quick fail, dont need to go through the next step 2. it's more accurate to show where error occurred as early as possible 3. get results from shell scripts Change-Id: I5c1e59839c55b92de0e83e7e1eb552aa364b3f80 Signed-off-by: Leo Wang <grakiss.wanglei@huawei.com>
2016-12-15dovetail tool: bugfix about setup dovetailxudan1-4/+8
1. change the entry_points of setup.cfg into dovetail.run:main 2. after setup the command 'dovetail -d true --testsuite debug' can just be successfully executed under folder dovetail/dovetail. That is mainly because of the load functions of testcase.py using relative paths. 3. using absolute path instead of relative path can support the command successfully executed under any path. JIRA: DOVETAIL-154 Change-Id: I4a2ad39a1a60e7e63efc55515781d27b823aa894 Signed-off-by: xudan <xudan16@huawei.com>
2016-12-15dovetail tool: move conf/dovetail_config.py into utils/xudan1-1/+1
1. move conf/dovetail_config.py into utils/ 2. change the open file path in dovetail_config.py 3. do some modification about all 'from conf.dovetail_config import ***' JIRA: DOVETAIL-158 Change-Id: I7670e6de993899e4e2a604cbc501dc3339896ef1 Signed-off-by: xudan <xudan16@huawei.com>
2016-12-14[dovetail tool] support shell scripts for testcase validationLeo Wang1-6/+12
JIRA: DOVETAIL-46 1. for now a testcase has two kinds of validation types(functest, yardstick), and it is not enough to check the complete funcionality 2. add new validation type(shell) for extra validation of the test case to make result more accurate and more convincing. Change-Id: I5c049a71f11cca71a7914f8af704c1983aba3dca Signed-off-by: Leo Wang <grakiss.wanglei@huawei.com>
2016-12-14[dovetail tool] support shell scripts for testcase validationLeo Wang1-4/+5
JIRA: DOVETAIL-46 1. for now a testcase has two kinds of validation types(functest, yardstick), and it is not enough to check the complete funcionality 2. add new validation type(shell) for extra validation of the test case to make result more accurate and more convincing. Change-Id: I60c1b54335b94e0cb150232432adb958b8f5a143 Signed-off-by: Leo Wang <grakiss.wanglei@huawei.com>
2016-12-13[dovetail tool] support shell scripts for testcase validationLeo Wang1-53/+119
JIRA: DOVETAIL-46 1. for now a testcase has two kinds of validation types(functest, yardstick), and it is not enough to check the complete funcionality 2. add new validation type(shell) for extra validation of the test case to make result more accurate and more convincing. Change-Id: I45dca6b8dbd888757da163189d261f6e4dba5034 Signed-off-by: Leo Wang <grakiss.wanglei@huawei.com>
2016-12-13bugfix for flake8 formatMatthewLi1-2/+2
error log see https://build.opnfv.org/ci/view/dovetail/job/dovetail-merge-master/58/console Change-Id: I7f5a7f290d82e79249561322259aa5e72186e126 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-12-13Merge "[WIP] Migrate to tox based environment"hongbo tian1-3/+3
2016-12-13[WIP] Migrate to tox based environmentLeif Madsen1-3/+3
Migrate the custom setup.py to a more formal setup environment using tox, in the same way that OpenStack configures their projects. Adds metadata to tox.ini from the setup.py file, and breaks out the dependencies from setup.py into requirements.txt and test-requirements.txt. Executes flake8 testing for pep8 coverage. Defaults to using Python 2 for the environment. Change-Id: Ieaea7d9f3dc227febd8fb1dc9142d478ba197a92 Signed-off-by: Leif Madsen <lmadsen@redhat.com> Signed-off-by: zshi <zshi@redhat.com>
2016-12-09[dovetail tool]: move config info from .py to config .yml fileMatthewLi1-6/+8
JIRA: DOVETAIL-145 Change-Id: Ia9b3c678f1a491e5daf2d280f05904ead3b0a296 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-11-29dovetail tool: package all logger initializationxudan1-5/+15
1. For container.py, parser.py, report.py and testcase.py, they use functions in classes to create loggers. So each class will have its own logger. 2. run.py will create all the loggers by create_logs() function before the use of them. So logs will not be initialized until the call of create_logs(). 3. Move logger in run.py into function main(). JIRA: DOVETAIL-56 Change-Id: I53de090df270cc7a34762ff99cbc9115cfb09465 Signed-off-by: xudan <xudan16@huawei.com>