summaryrefslogtreecommitdiffstats
path: root/dovetail/report.py
AgeCommit message (Collapse)AuthorFilesLines
2019-09-16Move the proj to python3Panagiotis Karalis1-4/+4
The aim of this patch is to upgrade python version of this project from python2 to python3. Signed-off-by: Panagiotis Karalis <panos.pkaralis@gmail.com> Change-Id: I3d3ef01176fda1b23a0542a24625be2f3368c40e
2019-09-02Add portal_key_file into all test case yml filesxudan1-0/+2
Change-Id: If111a8cc56c31ec3e22b090aca92ff56d41f7e01 Signed-off-by: xudan <xudan16@huawei.com>
2019-08-27Add validation disabled/enabled to results.jsonxudan1-0/+2
Change-Id: I918f85c5ac89cf2745efa8513946a2b5ab8638b1 Signed-off-by: xudan <xudan16@huawei.com>
2019-06-10Bugfix: Fix Dovetail 2 bugs found with CI resultsxudan1-2/+0
There are 2 bugs found with CI results of functest.tempest.neutron_tempest_plugin_api and all HA test cases. Yardtsick conf file does't mapping the keyfile to Yardstick container, so all HA test cases got SSHERROR. Also remove some useless debug msg to make dovetail.log more concise. Change-Id: I63c12d1b91ed5f8f8a893689384685ca9215f367 Signed-off-by: xudan <xudan16@huawei.com>
2019-05-16Fix the verdict of sub test casesxudan1-1/+1
Error to group skipped sub test cases to passed one. Example: Skipped sub test case will be parsed as passed in following case. skipped sub test case: tempest.api.network.test_routers.RoutersTest.test_create_show_list_update_delete_router passed sub test case list includes this one: tempest.api.network.test_routers.RoutersTest.test_add_remove_router_interface_with_subnet_id Then the skipped one will be checked as pass. JIRA: DOVETAIL-775 Change-Id: I449f4968c23a9def1177a7c9645e5dcb89b2ec30 Signed-off-by: xudan <xudan16@huawei.com>
2019-04-26Change HEAT package from .tag.gz to be .zipxudan1-1/+1
As Trevor Lovett suggests, the Heat file is a zip file. So users need to package it different only for Dovetail testing. It will be better to change it to be .zip to keep it the same as what they do. JIRA: DOVETAIL-770 Change-Id: I5338720c5e1363081e1f8cdce33391a969f1cd97 Signed-off-by: xudan <xudan16@huawei.com>
2019-04-17Failed to upload results to ONAP portalPanagiotis Karalis1-0/+7
The results.json file generated by running 'dovetail run --testsuite xxx' is different from the one generated by running 'dovetail run --testcase xxx'. The former can be uploaded to ONAP portal successfully but the latter one can't. The main reason is that the latter one lacking of 'vnf_type'. Keep the results.json with the same format for both running one test suite or several test cases. JIRA: DOVETAIL-769 Change-Id: I46af84f9f229f9e601439f68c9ed9df9477d002b Signed-off-by: Panagiotis Karalis <pkaralis@intracom-telecom.com>
2019-03-28Fix VVP test caseStamatis Katsaounis1-2/+2
This patch adds check for no environment variables provided to container configuration. Furthermore, it changes the check results file to a list of check result files. Change-Id: I2112ee1fd4b3156608d8a27dd3b9c90dbe35a24c Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com> Co-Authored-By: Panagiotis Karalis <pkaralis@intracom-telecom.com>
2019-03-13Calculate checksum for input VNFStamatis Katsaounis1-0/+21
This patch adds checksum information inside ONAP related test case run results. The checksum is produced by the VNF input which can either be a CSAR file or an archive of Heat templates. Change-Id: I0ed58bdc9cc4031da08fd2ac220ef294520ef447 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-03-08Add Functest Rally test casesStamatis Katsaounis1-12/+26
This patch adds all Rally testcases of Functest to Dovetail. Change-Id: I8b20bce9966e924fd5aac41bde5a30442394db05 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-03-07Fix the verdict of TC in sub-testcase list resultPanagiotis Karalis1-0/+6
When the sub-test case list is defined, in the dovetail results, each test case is appeared with the its verdict next to its name. Once a sub-testcase is passed, the verdict PASS is appeared next to the name of sub testcase. Once a sub-testcase is skipped or failed, the verdict FAIL is appeared next to the name of sub-testcase. This fix updates the code so that the verdict of skipped and failed sub test cases to be appeared respectively next to its sub-tsetcase name. JIRA: DOVETAIL-763 Change-Id: Ib05b3793a57acbaa67b89321a840ae11251c2417 Signed-off-by: Panagiotis Karalis <pkaralis@intracom-telecom.com>
2019-03-07Provide a list of check results files instead of oneStamatis Katsaounis1-39/+47
This patchs gives the ability to provide more than one check results file. Furthermore, it does not alter the behavior of the existing tests but prepares dovetail to be able to fetch information from many sources. Change-Id: I23cd23d2908ad6bdddd94b977813d0e81dc0c05a Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-02-20Remove deprecated ONAP lifecycle test caseStamatis Katsaounis1-53/+1
This patch removes the ONAP lifecycle deprecated test case. Furthermore, it clears helper functions as well as unwanted configuration. Change-Id: Ib958457cf120ffb641a72be028d3a031741c44c6 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-02-14Dynamic results version and vnf type recognitionStamatis Katsaounis1-1/+4
This patch makes it able to fetch the version from the testsuite. Furthermore, in case of ONAP test cases it adds the vnf_type to the produced results.json file in order to be parsed from the results portal. Finally, it creates a testuite for ONAP TOSCA related tests. Change-Id: I613358c445cfe62730b98fee5e8c6fba7d322e90 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-02-01Add ONAP VVP Heat Template validation testsStamatis Katsaounis1-15/+73
Change-Id: I9a3d56932ce41191901381831013768d1c57a749 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-01-03Add ONAP VNF SDK verification testsxudan1-3/+79
Please refer to env_config.sh.onap.sample when testing it. The guide of preparing local env for onap package validation can be found here https://gerrit.onap.org/r/#/c/75119 Change-Id: I39d6b11544847756126623a7eb3b953dc800c470 Signed-off-by: xudan <xudan16@huawei.com>
2018-12-11Enable ShellRunnerxudan1-13/+6
1. fix the bug 'ShellRunner' object has no attribute 'archive_logs' 2. remove the testarea_supported because it's not very necessary for the test cases themselves and can make it a little more simple to add a test case with new test area JIRA: DOVETAIL-750 Change-Id: I71ee74615200376adca2a0db040753e5fce329bc Signed-off-by: xudan <xudan16@huawei.com>
2018-11-23Integrate k8s test cassesxudan1-3/+29
In order to integrate k8s test cases, Dovetail framework should do some refactor and make it more general for k8s test cases as well as ONAP ones. Integrate 2 k8s test cases. JIRA: DOVETAIL-748 Change-Id: Ibd87754ffb5fb29f6b4ce79232af860c2ed2da9c Signed-off-by: xudan <xudan16@huawei.com>
2018-11-20Add missing unit tests for report fileStamatis Katsaounis1-19/+20
JIRA: DOVETAIL-724 This patch adds unit tests for report file methods of Dovetail which were missing. Change-Id: I2ba7c8f9fcfb962988b2904b01538e9f89a0d849 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2018-11-06Update classes in report.pyxudan1-33/+28
Change-Id: Ia17bf7ef3e304eef7b3f0b2f7aa04e3be75bc8c3 Signed-off-by: xudan <xudan16@huawei.com>
2018-10-31CI verify jobs failed due to the update of toxxudan1-4/+3
The LF pod used to run the CI verify jobs have updated the version of tox. So the CI verify jobs failed. See https://build.opnfv.org/ci/view/dovetail/job/dovetail-merge-master/476/console Fix the python format errors Ignore error 'W503 line break before binary operator'. Change-Id: Ic2df070447bbea39c877ce7a394d733a95064bb1 Signed-off-by: xudan <xudan16@huawei.com>
2018-10-01Bugfix: There is an exception when crawling Yardstick resultsxudan1-1/+2
If there is no pod.yaml, the results data looks like, {"status": 2, "result": "[Errno 2] No such file or directory: u'/home/opnfv/userconfig/pre_config/pod.yaml'"} It will get an exception when reading value from file, File "/usr/local/lib/python2.7/dist-packages/dovetail/report.py", line 327, in crawl_from_file criteria = data['result']['criteria'] TypeError: string indices must be integers https://build.opnfv.org/ci/view/dovetail/job/dovetail-apex-baremetal-default-mandatory-master/52/console Change-Id: I73ca3e5ae73433a2dc139789df4d67e8c583be08 Signed-off-by: xudan <xudan16@huawei.com>
2018-09-13Update code base and docs to 2018.09xudan1-1/+1
The official name of the next release is going to be 2018.09. We need to rename all occurrences of 2018.08 in the code base, the documentation, and the web portal. JIRA: DOVETAIL-740 Change-Id: Ie0e532d6671b7590454e9afd960a2934f313a022 Signed-off-by: xudan <xudan16@huawei.com>
2018-07-24Making the results file a proper json fileGeorg Kunz1-31/+28
The results.json file is currently not a properly formated json file, but contains one json structure per line/testcase. Whit this change, all test results are being collected at runtime and then written to a properly formated json file at the end of a test run. Moreover, this patch introduces 'mandatory' and 'optional' sections to the test suites definition files in order to clearly indicate if a test is part of the corresponding test scope. The web portal needs to be adapted accordingly to be able to read the new results file. JIRA: DOVETAIL-672 Change-Id: Id0706c7e82622fc9c2c2fece26b2d6c07c1580be Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-07-22Remove functest_testsuite and functest_testcase in dovetail_config.ymlxudan1-8/+4
The 2 sections 'functest_testsuite' and 'functest_testcase' are used by all Functest test cases. They aim to parse the results of Tempest sub test cases. Remove them, and check the sub test case list, 1. if it's empty, that means no need to check sub test cases 2. if it is not empty, check the sub test cases one by one. Additional, remove some useless keys in dovetail_config.yml JIRA: DOVETAIL-658 Change-Id: Ic829264b0c8eaedd8e2b3fa55b38a34644482c20 Signed-off-by: xudan <xudan16@huawei.com>
2018-06-29Update licence headers to comply with guidelinesGeorg Kunz1-1/+4
Updating the licence headers to comply with the community guidelines [1] and anteater checks. [1] https://wiki.opnfv.org/display/DEV/Contribution+Guidelines#ContributionGuidelines-License Change-Id: Idec787e32daf76c0b1bd6108977b78828125da5e Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-27Archive all test case result filesxudan1-30/+41
1. All detailed tempest test cases results are recorded with file tempest-report.html rather than tempest.log now. 2. Archive all results files include functest.log, yardstick.log ... for each test case. Otherwise, they will all be in one log file. 3. Support to set the source files and dest files need to be archived with the test case config files rather than hardcode with the source code. 4. Give the file which used to be parse the results (PASS/FAIL) with the test case config files rather than hardcode with the source code. Change-Id: I30f435d91ab90cf20def47007f177fe98187484d Signed-off-by: xudan <xudan16@huawei.com>
2018-05-08Improve the CLI of Dovetail toolxudan1-27/+8
The CLI of dovetail tool needs some improvement. 1. Add an option '--testcase' to support specify test case. 2. Add an option '--no-clean' to keep functest/yardstick/bottlenecks containers for debugging after running the test cases. 3. Improve the local report section. a. Remove some info that is not necessary b. Make the info more clearly JIRA: DOVETAIL-588 Change-Id: Ic769a6e2788b407c3df08ccd593a0ecdaa35e5ea Signed-off-by: xudan <xudan16@huawei.com>
2018-05-04Remove local DBxudan1-63/+31
Remove local DB and can just report without launch local DB. Need to do a little modification with the web portal code. JIRA: DOVETAIL-593 Change-Id: I595db39cce4156596ee0522d4b7e642a23d78ab9 Signed-off-by: xudan <xudan16@huawei.com>
2018-04-17Switch to OPNFV Fraser releasexudan1-3/+3
1. Use Functest Fraser docker images whose tag is fraser 2. Use Yardstick stable docker images 3. Functest fraser has moved testcases.yaml file from functest/ci to xtesting/ci 4. Functest fraser has removed 'functest env prepare' command 5. Functest uses rally to execute osinterop test cases instead of refstack_client Change-Id: I81ebefe6ea959ca77c2c87f1d75ef56ac18c165f Signed-off-by: xudan <xudan16@huawei.com>
2018-03-29Integrate dovetail with ONAPMoshe1-2/+67
Change-Id: I6a1fa5f27df2f5127eb00156b3135dc79caf5e77 Issue-ID: DOVETAIL-629 Signed-off-by: Moshe <moshehoa@amdocs.com>
2018-02-19Add 3 new HA test cases into proposed_tests test suitexudan1-8/+10
Add 3 new HA test cases into proposed_tests test suite and running them on CI. For the current 8 HA test cases and 3 other HA test cases plans to add, do some adaptions for the Yardstick Euphrates release. 1. Don't need the bugfix patch anymore. 2. The data format has changed, and has done some adaptions. 3. For Yardstick Euphrates, if setting OS_INSECURE=True, it needs to set OS_CACERT=False 4. Test case tc011 needs a image and flavor to create the stack. JIRA: DOVETAIL-614 JIRA: DOVETAIL-586 Change-Id: I9da8c8316246b9edee0d81ad18422ae6018becaf Signed-off-by: xudan <xudan16@huawei.com>
2018-02-05Adapt Functest E release Frameworkxudan1-7/+0
Use opnfv/functest-restapi:euphrates. Adapt Functest E framework for all Functest test cases. 1. All path of functest and sdnvpn has been changed. 2. Sdnvpn can clean resources by itself with E release, no need to clean again. 3. Add a file testcases.yaml to overwrite the file in functest container, then it can support to run tempest_custom i.e. ipv6 and tempest in OVP. JIRA: DOVETAIL-584 Change-Id: I3ba9aa8e80becaa99cc89c8db4b39acbb89dfda4 Signed-off-by: xudan <xudan16@huawei.com>
2017-12-25[Bugfix] Local summary with wrong results when Tempest failed at the SetupClassxudan1-11/+14
When Tempest test cases failed at the SetupClass, the record is the class name of the test cases rather than the name of the test cases. The local summary will say they are passed wrongly. JIRA: DOVETAIL-575 Change-Id: Iec879378770017eddd37e9a99346d980355bb6e2 Signed-off-by: xudan <xudan16@huawei.com>
2017-11-20Include full date in name of compressed result fileGeorg Kunz1-1/+1
The name of the compressed result file included a timestamp according to the pattern "day_hour_minute". This is a rather uncommon timestamp and can cause confusion. This patch changes the timestamp to a more common "YearMonthDay-HourMinute" format. Please note: Please check if this change has an impact on the web portal. Change-Id: Ife41050bc1d5ca6942706b756d03a1d0d251ae41 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2017-09-29Merge "Bugfix: sdnvpn test cases failed due to wrong module name"Leo wang1-1/+2
2017-09-28Bugfix: sdnvpn test cases failed due to wrong module namexudan1-1/+2
SDNVPN test cases failed due to wrong module name. The module name needs to be changed into "sdnvpn.test.functest.testcase_1" Refer to patch https://gerrit.opnfv.org/gerrit/#/c/41385/ Currently Dovetail uses functest:cvp.0.5.0 which uses module name "testcase_1". JIRA: DOVETAIL-510 Change-Id: I3232760d3af34125c7da71a5d2c62ca32693c82b Signed-off-by: xudan <xudan16@huawei.com>
2017-09-29Merge "Support to run mandatory or optional test cases separately"Leo wang1-5/+2
2017-09-28Merge "Make Yardstick log level can be set by CLI"Leo wang1-1/+1
2017-09-26Bugfix: tar results dir cannot be parsed by web portalxudan1-3/+9
The directory tree of results tar file is wrong, so web portal cannot parse it. Reset the tar file. JIRA: DOVETAIL-508 Change-Id: I8475323294e84ca51242b700570926d5c0177390 Signed-off-by: xudan <xudan16@huawei.com>
2017-09-26Make Yardstick log level can be set by CLIxudan1-1/+1
1. Currently, the log level of yardstick is DEBUG and cannot be modified. 2. Make it can be changed from CLI. 3. When use dovetail run --debug, can see all Yardstick logs (including DEBUG) in dovetail.log. 4. When run without debug option, cannot see Yardstick debug logs in dovetail.logs. 5. yardstick.log always doesn't include debug logs. JIRA: DOVETAIL-506 Change-Id: Iaafb61232a22e16fff2fa4605bf7e262b022085e Signed-off-by: xudan <xudan16@huawei.com>
2017-09-25Support to run mandatory or optional test cases separatelyxudan1-5/+2
1. Currently, dovetail can run a test suite or a test area in that test suite. 2. The test areas in one test suite are not divided into optional and mandatory. 3. Split them and support to run just mandatory or optional. 4. Support to run multiple test areas. JIRA: DOVETAIL-505 Change-Id: I42cd7b4e11c3e3674c806e9bc999b782bf5c85c6 Signed-off-by: xudan <xudan16@huawei.com>
2017-08-15Adapt Functest cvp.0.5.0 docker imagexudan1-6/+8
In order to use functest cvp.0.5.0, Dovetail tool needs some adaptions. Change-Id: I9d0a23791c246a6c4961f229eaeb1f52f9b4995a Signed-off-by: xudan <xudan16@huawei.com>
2017-08-10create a tar file for logsgrakiss1-1/+0
JIRA: DOVETAIL-479 User may want to upload all logs along with the results, so others may check the running logs to make sure what' s going on during the test. Change-Id: Ice906264bbdf091d3353d6d2e146653aeb057f5f Signed-off-by: grakiss <grakiss.wanglei@huawei.com>
2017-08-08create a tar file for logsgrakiss1-0/+13
JIRA: DOVETAIL-479 User may want to upload all logs along with the results, so others may check the running logs to make sure what' s going on during the test. Change-Id: Ia5862f99c129b8207aa91a391567ffdd5ee0829a Signed-off-by: grakiss <grakiss.wanglei@huawei.com>
2017-08-03Add resiliency test caes into proposed_testsxudan1-1/+68
JIRA: DOVETAIL-474 Add Bottlenecks resiliency test case "posca_factor_ping" into proposed_tests. Change-Id: I76aea651014fb4722ddae07559f417d3f840eee2 Signed-off-by: xudan <xudan16@huawei.com>
2017-06-28Modify Dovetail logsxudan1-22/+25
JIRA: DOVETAIL-450 Change-Id: I795679c9559d1bead3aecefff28d21b6567aac2f Signed-off-by: xudan <xudan16@huawei.com>
2017-05-25change the name of Yardstick results file and some log msgxudan1-2/+2
JIRA: DOVETAIL-428 1. Change the name of Yardstick result file from opnfv_yardstick_tc019.out to dovetail.ha.tc001.out. Because opnfv_yardstick_tc019 is the name defined in Yardstick and dovetail.ha.tc001 is the name defined in Dovetail. 2. Check whether the file exists before reporting the results are stored in file. 3. Delete test cases vping_ssh, vping_userdata and sdnvpn.tc003 from the testsuite proposed_tests. Change-Id: I4cb16947729479616a03abd9e91b76f9a41a949c Signed-off-by: xudan <xudan16@huawei.com>
2017-05-08fix ha testcase PASS/FAIL judgementMatthewLi1-1/+6
JIRA: DOVETAIL-425 Change-Id: Ifee2473956e0a1a3ad9c98beff92d94327251614 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-04-25Bugfix: KeyError when tempest_custom runs wrongxudan1-8/+12
JIRA: DOVETAIL-406 1. When there are something wrong with the test case tempest_custom, it's details json data are "details": {}, So it will throw KeyError exception in this situation. 2. Catch this exception and set all sub_testcases FAIL. 3. Disable the exceed_max_retry_times(). Change-Id: I6bdfc5944a535625b2211a3bd8096a13b81b339e Signed-off-by: xudan <xudan16@huawei.com>