summaryrefslogtreecommitdiffstats
path: root/dovetail/report.py
AgeCommit message (Collapse)AuthorFilesLines
2018-11-06Update classes in report.pyxudan1-33/+28
Change-Id: Ia17bf7ef3e304eef7b3f0b2f7aa04e3be75bc8c3 Signed-off-by: xudan <xudan16@huawei.com>
2018-10-31CI verify jobs failed due to the update of toxxudan1-4/+3
The LF pod used to run the CI verify jobs have updated the version of tox. So the CI verify jobs failed. See https://build.opnfv.org/ci/view/dovetail/job/dovetail-merge-master/476/console Fix the python format errors Ignore error 'W503 line break before binary operator'. Change-Id: Ic2df070447bbea39c877ce7a394d733a95064bb1 Signed-off-by: xudan <xudan16@huawei.com>
2018-10-01Bugfix: There is an exception when crawling Yardstick resultsxudan1-1/+2
If there is no pod.yaml, the results data looks like, {"status": 2, "result": "[Errno 2] No such file or directory: u'/home/opnfv/userconfig/pre_config/pod.yaml'"} It will get an exception when reading value from file, File "/usr/local/lib/python2.7/dist-packages/dovetail/report.py", line 327, in crawl_from_file criteria = data['result']['criteria'] TypeError: string indices must be integers https://build.opnfv.org/ci/view/dovetail/job/dovetail-apex-baremetal-default-mandatory-master/52/console Change-Id: I73ca3e5ae73433a2dc139789df4d67e8c583be08 Signed-off-by: xudan <xudan16@huawei.com>
2018-09-13Update code base and docs to 2018.09xudan1-1/+1
The official name of the next release is going to be 2018.09. We need to rename all occurrences of 2018.08 in the code base, the documentation, and the web portal. JIRA: DOVETAIL-740 Change-Id: Ie0e532d6671b7590454e9afd960a2934f313a022 Signed-off-by: xudan <xudan16@huawei.com>
2018-07-24Making the results file a proper json fileGeorg Kunz1-31/+28
The results.json file is currently not a properly formated json file, but contains one json structure per line/testcase. Whit this change, all test results are being collected at runtime and then written to a properly formated json file at the end of a test run. Moreover, this patch introduces 'mandatory' and 'optional' sections to the test suites definition files in order to clearly indicate if a test is part of the corresponding test scope. The web portal needs to be adapted accordingly to be able to read the new results file. JIRA: DOVETAIL-672 Change-Id: Id0706c7e82622fc9c2c2fece26b2d6c07c1580be Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-07-22Remove functest_testsuite and functest_testcase in dovetail_config.ymlxudan1-8/+4
The 2 sections 'functest_testsuite' and 'functest_testcase' are used by all Functest test cases. They aim to parse the results of Tempest sub test cases. Remove them, and check the sub test case list, 1. if it's empty, that means no need to check sub test cases 2. if it is not empty, check the sub test cases one by one. Additional, remove some useless keys in dovetail_config.yml JIRA: DOVETAIL-658 Change-Id: Ic829264b0c8eaedd8e2b3fa55b38a34644482c20 Signed-off-by: xudan <xudan16@huawei.com>
2018-06-29Update licence headers to comply with guidelinesGeorg Kunz1-1/+4
Updating the licence headers to comply with the community guidelines [1] and anteater checks. [1] https://wiki.opnfv.org/display/DEV/Contribution+Guidelines#ContributionGuidelines-License Change-Id: Idec787e32daf76c0b1bd6108977b78828125da5e Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-27Archive all test case result filesxudan1-30/+41
1. All detailed tempest test cases results are recorded with file tempest-report.html rather than tempest.log now. 2. Archive all results files include functest.log, yardstick.log ... for each test case. Otherwise, they will all be in one log file. 3. Support to set the source files and dest files need to be archived with the test case config files rather than hardcode with the source code. 4. Give the file which used to be parse the results (PASS/FAIL) with the test case config files rather than hardcode with the source code. Change-Id: I30f435d91ab90cf20def47007f177fe98187484d Signed-off-by: xudan <xudan16@huawei.com>
2018-05-08Improve the CLI of Dovetail toolxudan1-27/+8
The CLI of dovetail tool needs some improvement. 1. Add an option '--testcase' to support specify test case. 2. Add an option '--no-clean' to keep functest/yardstick/bottlenecks containers for debugging after running the test cases. 3. Improve the local report section. a. Remove some info that is not necessary b. Make the info more clearly JIRA: DOVETAIL-588 Change-Id: Ic769a6e2788b407c3df08ccd593a0ecdaa35e5ea Signed-off-by: xudan <xudan16@huawei.com>
2018-05-04Remove local DBxudan1-63/+31
Remove local DB and can just report without launch local DB. Need to do a little modification with the web portal code. JIRA: DOVETAIL-593 Change-Id: I595db39cce4156596ee0522d4b7e642a23d78ab9 Signed-off-by: xudan <xudan16@huawei.com>
2018-04-17Switch to OPNFV Fraser releasexudan1-3/+3
1. Use Functest Fraser docker images whose tag is fraser 2. Use Yardstick stable docker images 3. Functest fraser has moved testcases.yaml file from functest/ci to xtesting/ci 4. Functest fraser has removed 'functest env prepare' command 5. Functest uses rally to execute osinterop test cases instead of refstack_client Change-Id: I81ebefe6ea959ca77c2c87f1d75ef56ac18c165f Signed-off-by: xudan <xudan16@huawei.com>
2018-03-29Integrate dovetail with ONAPMoshe1-2/+67
Change-Id: I6a1fa5f27df2f5127eb00156b3135dc79caf5e77 Issue-ID: DOVETAIL-629 Signed-off-by: Moshe <moshehoa@amdocs.com>
2018-02-19Add 3 new HA test cases into proposed_tests test suitexudan1-8/+10
Add 3 new HA test cases into proposed_tests test suite and running them on CI. For the current 8 HA test cases and 3 other HA test cases plans to add, do some adaptions for the Yardstick Euphrates release. 1. Don't need the bugfix patch anymore. 2. The data format has changed, and has done some adaptions. 3. For Yardstick Euphrates, if setting OS_INSECURE=True, it needs to set OS_CACERT=False 4. Test case tc011 needs a image and flavor to create the stack. JIRA: DOVETAIL-614 JIRA: DOVETAIL-586 Change-Id: I9da8c8316246b9edee0d81ad18422ae6018becaf Signed-off-by: xudan <xudan16@huawei.com>
2018-02-05Adapt Functest E release Frameworkxudan1-7/+0
Use opnfv/functest-restapi:euphrates. Adapt Functest E framework for all Functest test cases. 1. All path of functest and sdnvpn has been changed. 2. Sdnvpn can clean resources by itself with E release, no need to clean again. 3. Add a file testcases.yaml to overwrite the file in functest container, then it can support to run tempest_custom i.e. ipv6 and tempest in OVP. JIRA: DOVETAIL-584 Change-Id: I3ba9aa8e80becaa99cc89c8db4b39acbb89dfda4 Signed-off-by: xudan <xudan16@huawei.com>
2017-12-25[Bugfix] Local summary with wrong results when Tempest failed at the SetupClassxudan1-11/+14
When Tempest test cases failed at the SetupClass, the record is the class name of the test cases rather than the name of the test cases. The local summary will say they are passed wrongly. JIRA: DOVETAIL-575 Change-Id: Iec879378770017eddd37e9a99346d980355bb6e2 Signed-off-by: xudan <xudan16@huawei.com>
2017-11-20Include full date in name of compressed result fileGeorg Kunz1-1/+1
The name of the compressed result file included a timestamp according to the pattern "day_hour_minute". This is a rather uncommon timestamp and can cause confusion. This patch changes the timestamp to a more common "YearMonthDay-HourMinute" format. Please note: Please check if this change has an impact on the web portal. Change-Id: Ife41050bc1d5ca6942706b756d03a1d0d251ae41 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2017-09-29Merge "Bugfix: sdnvpn test cases failed due to wrong module name"Leo wang1-1/+2
2017-09-28Bugfix: sdnvpn test cases failed due to wrong module namexudan1-1/+2
SDNVPN test cases failed due to wrong module name. The module name needs to be changed into "sdnvpn.test.functest.testcase_1" Refer to patch https://gerrit.opnfv.org/gerrit/#/c/41385/ Currently Dovetail uses functest:cvp.0.5.0 which uses module name "testcase_1". JIRA: DOVETAIL-510 Change-Id: I3232760d3af34125c7da71a5d2c62ca32693c82b Signed-off-by: xudan <xudan16@huawei.com>
2017-09-29Merge "Support to run mandatory or optional test cases separately"Leo wang1-5/+2
2017-09-28Merge "Make Yardstick log level can be set by CLI"Leo wang1-1/+1
2017-09-26Bugfix: tar results dir cannot be parsed by web portalxudan1-3/+9
The directory tree of results tar file is wrong, so web portal cannot parse it. Reset the tar file. JIRA: DOVETAIL-508 Change-Id: I8475323294e84ca51242b700570926d5c0177390 Signed-off-by: xudan <xudan16@huawei.com>
2017-09-26Make Yardstick log level can be set by CLIxudan1-1/+1
1. Currently, the log level of yardstick is DEBUG and cannot be modified. 2. Make it can be changed from CLI. 3. When use dovetail run --debug, can see all Yardstick logs (including DEBUG) in dovetail.log. 4. When run without debug option, cannot see Yardstick debug logs in dovetail.logs. 5. yardstick.log always doesn't include debug logs. JIRA: DOVETAIL-506 Change-Id: Iaafb61232a22e16fff2fa4605bf7e262b022085e Signed-off-by: xudan <xudan16@huawei.com>
2017-09-25Support to run mandatory or optional test cases separatelyxudan1-5/+2
1. Currently, dovetail can run a test suite or a test area in that test suite. 2. The test areas in one test suite are not divided into optional and mandatory. 3. Split them and support to run just mandatory or optional. 4. Support to run multiple test areas. JIRA: DOVETAIL-505 Change-Id: I42cd7b4e11c3e3674c806e9bc999b782bf5c85c6 Signed-off-by: xudan <xudan16@huawei.com>
2017-08-15Adapt Functest cvp.0.5.0 docker imagexudan1-6/+8
In order to use functest cvp.0.5.0, Dovetail tool needs some adaptions. Change-Id: I9d0a23791c246a6c4961f229eaeb1f52f9b4995a Signed-off-by: xudan <xudan16@huawei.com>
2017-08-10create a tar file for logsgrakiss1-1/+0
JIRA: DOVETAIL-479 User may want to upload all logs along with the results, so others may check the running logs to make sure what' s going on during the test. Change-Id: Ice906264bbdf091d3353d6d2e146653aeb057f5f Signed-off-by: grakiss <grakiss.wanglei@huawei.com>
2017-08-08create a tar file for logsgrakiss1-0/+13
JIRA: DOVETAIL-479 User may want to upload all logs along with the results, so others may check the running logs to make sure what' s going on during the test. Change-Id: Ia5862f99c129b8207aa91a391567ffdd5ee0829a Signed-off-by: grakiss <grakiss.wanglei@huawei.com>
2017-08-03Add resiliency test caes into proposed_testsxudan1-1/+68
JIRA: DOVETAIL-474 Add Bottlenecks resiliency test case "posca_factor_ping" into proposed_tests. Change-Id: I76aea651014fb4722ddae07559f417d3f840eee2 Signed-off-by: xudan <xudan16@huawei.com>
2017-06-28Modify Dovetail logsxudan1-22/+25
JIRA: DOVETAIL-450 Change-Id: I795679c9559d1bead3aecefff28d21b6567aac2f Signed-off-by: xudan <xudan16@huawei.com>
2017-05-25change the name of Yardstick results file and some log msgxudan1-2/+2
JIRA: DOVETAIL-428 1. Change the name of Yardstick result file from opnfv_yardstick_tc019.out to dovetail.ha.tc001.out. Because opnfv_yardstick_tc019 is the name defined in Yardstick and dovetail.ha.tc001 is the name defined in Dovetail. 2. Check whether the file exists before reporting the results are stored in file. 3. Delete test cases vping_ssh, vping_userdata and sdnvpn.tc003 from the testsuite proposed_tests. Change-Id: I4cb16947729479616a03abd9e91b76f9a41a949c Signed-off-by: xudan <xudan16@huawei.com>
2017-05-08fix ha testcase PASS/FAIL judgementMatthewLi1-1/+6
JIRA: DOVETAIL-425 Change-Id: Ifee2473956e0a1a3ad9c98beff92d94327251614 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-04-25Bugfix: KeyError when tempest_custom runs wrongxudan1-8/+12
JIRA: DOVETAIL-406 1. When there are something wrong with the test case tempest_custom, it's details json data are "details": {}, So it will throw KeyError exception in this situation. 2. Catch this exception and set all sub_testcases FAIL. 3. Disable the exceed_max_retry_times(). Change-Id: I6bdfc5944a535625b2211a3bd8096a13b81b339e Signed-off-by: xudan <xudan16@huawei.com>
2017-04-22Bugfix about the ipv6 results checkerxudan1-0/+6
JIRA: DOVETAIL-404 1. IPV6 sub-testcase's name includes [id], which will affects the regex. 2. Remove the [id] from the name of the sub_testcases. Change-Id: I51b567e8382fa4e701f8b32ba7ea195d49fd31eb Signed-off-by: xudan <xudan16@huawei.com>
2017-04-21use Functest tempest_custom test cases to run all Ipv6 test casesxudan1-6/+15
JIRA: DOVETAIL-401 1. modify all Ipv6 test cases, change tempest_smoke_serial/tempest_full_parallel into tempest_custom, add ID of each sub-testcases. 2. Some sub-testcases in ipv6.tc002, ipv6.tc003, ipv6.tc004 and ipv6.tc005 can not be found in the full tempest test case list. 2. disable testcases' script_result_acquired to enable tempest_custom run multiple times. 3. use build_tag-testcase_name(such as daily-master-79a6e392-28bb-4c40-9d88-a03bed99da2d-dovetail.ipv6.tc025) as the unique ID of each testcase. 4. comment out the progress bar, because it has some problems when print the report text. 5. Functest has changed the TEST_DB_URL from http://<testapi>:<port>/apv/v1 to http://<testapi>:<port>/apv/v1/results. 6. Functest has changed the TEST_DB_URL to file://<file_path>/<file_name>. see https://gerrit.opnfv.org/gerrit/#/c/33511/ Change-Id: I524f6702377f215f526158214842481eb44565be Signed-off-by: xudan <xudan16@huawei.com>
2017-04-18put dovetail test result back in log and not generte report fileMatthewLi1-1/+1
Change-Id: I2da5e7844f5ce034d4c0cfaa4592e6cd47002049 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-04-12dovetail tool: bugfix: test cases are erroneously judgedxudan1-61/+59
JIRA: DOVETAIL-375 JIRA: DOVETAIL-390 1. defcore test cases are erroneously judged because of: a) defcore results json data lack some keys b) some sub-testcases' names are the same as the prefix of another sub-testcases. 2. this patch removes the extra keys and judge sub-testcases accurately. 3. remove "store_type" in functest_config.yml and yardstick_config.yml. 4. change some log levels from info to debug. Change-Id: I5f126e70309409cac66ef9a0909f1573d325098b Signed-off-by: xudan <xudan16@huawei.com>
2017-04-01refstack result parseMatthewLi1-30/+27
JIRA: DOVETAIL-375 1, together with other testsuite from functest, such as tempest etc. this should merge after https://gerrit.opnfv.org/gerrit/#/c/31313/ 2, delete some unnecessary definition 3, bugfix of docker cp inside container, see https://build.opnfv.org/ci/view/dovetail/job/dovetail-compass-baremetal-debug-master/361/console 4, disable tempest testcaes since changed to refstack_defcore Change-Id: I67503e348af4a3f697f1b4106bfe81c05add8aa0 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-03-30dovetail tool: set a unique build_tag for each DoveTail testxudan1-3/+2
JIRA: DOVETAIL-379 1. create a uuid for each test, and set build_tag=daily-master-<uuid> 2. the reason it starts with "daily_master" is that it's the requirement of functest when push data to DB. Change-Id: I9a59d5695a9f0d83bf4330379f1949a2a04f876c Signed-off-by: xudan <xudan16@huawei.com>
2017-03-25dovetail tool: Add an option cmd for pushing results into DBxudan1-3/+3
JIRA: DOVETAIL-373 1. add an option --report/-r to choose the report type. 2. "-r file" will store results with files. 3. "-r http://192.168.135.2:8000/api/v1" will push results into DB. The DB url can replaced by user's local DB or official DB. 4. Default report type is defined as "report_dest" in dovetail_config.yml. 5. Yardstick can't push results of one testcase into DB now, so both "-r file" and "-r http://..." will be regarded as "file". Change-Id: Ieaa017ab59c5b15235bf306ef57cdc56beb22af8 Signed-off-by: xudan <xudan16@huawei.com>
2017-02-20dovetail tool: bugfix for dovetail report and docker buildMatthewLi1-2/+2
JIRA: DOVETAIL-353 JIRA: DOVETAIL-354 related patch: https://gerrit.opnfv.org/gerrit/#/c/29049/ Change-Id: Ic871abbda62dead7cf5e23e6117c9885d637c7fb Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-02-18Merge "dovetail tool: skip testcase status added"hongbo tian1-40/+58
2017-02-17dovetail tool: change the way to get the functest result filexudan1-5/+8
Functest has changed it's way to generate the local result file. see Gerrit: https://gerrit.opnfv.org/gerrit/#/c/28513/ Dovetail needs to do some adaptions to fit this change. JIRA: DOVETAIL-350 Change-Id: I681f74832f582f26c8874f7d2726c8588f430612 Signed-off-by: xudan <xudan16@huawei.com>
2017-02-17dovetail tool: skip testcase status addedMatthewLi1-40/+58
JIRA: DOVETAIL-344 1) testcase status, True/False --> PASS/SKIP/FAIL, so skipped testcase in tempest can be figured out details can be seen in https://etherpad.opnfv.org/p/dovetail_report 2) some variable name changed more readable, such as passed-->testcase_passed failed--> prepare_failed 3) "dovetail report" minor amend according to 1) Change-Id: Ibe8e50693fc474d78e7d0fb89257f34703248579 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-02-08Merge "dovetail tool: use python new-style classes which inherit from object"hongbo tian1-10/+10
2017-02-07dovetail tool: report.py bugfix and tempest skipped testcase addedMatthewLi1-11/+6
JIRA: DOVETAIL-343 1, bugfix https://build.opnfv.org/ci/view/dovetail/job/dovetail-compass-virtual-debug-master/lastSuccessfulBuild/artifact/results/dovetail_report.txt test result error 2, skipped testcase in tempest added 3, two testcase in tempest full testsuite added in debug.yml to debug Change-Id: I6bf371d3384e3755d6a41d951cfc9d1a55e00ab3 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-01-25dovetail tool: functest different testsuite support and yardstick testcase addedMatthewLi1-20/+44
JIRA: DOVETAIL-186 Change-Id: Id0eec6652c9595c660436dec89a309a77b16bc03 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-01-20dovetail tool: use python new-style classes which inherit from objectxudan1-10/+10
JIRA: DOVETAIL-187 Change-Id: Id6fa50a3a3cec7a18d27ca8feb383b53836a339b Signed-off-by: xudan <xudan16@huawei.com>
2017-01-11dovetail tool: switch logging to proper usageMatthewLi1-14/+14
JIRA: DOVETAIL-178 Change-Id: Ica814d4ff634c9159e4ecca888b8cc0f6c16e624 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-01-05release tag supportMatthewLi1-7/+9
JIRA: DOVETAIL-170 Change-Id: Id56e30fdb3fb5b652e2145dd406b71986997f200 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-12-22[dovetail tool]check and get results for each cmdLeo Wang1-24/+61
JIRA: DOVETAIL-166 Check the results of each cmds executed in test case 1. the results of pre_condition, post_condition, cmds need to be checked, so it can get a quick fail, dont need to go through the next step 2. it's more accurate to show where error occurred as early as possible 3. get results from shell scripts Change-Id: I5c1e59839c55b92de0e83e7e1eb552aa364b3f80 Signed-off-by: Leo Wang <grakiss.wanglei@huawei.com>
2016-12-20Merge "bugfix for duration missed"hongbo tian1-1/+1