summaryrefslogtreecommitdiffstats
path: root/dovetail/report.py
AgeCommit message (Collapse)AuthorFilesLines
2017-04-22Bugfix about the ipv6 results checkerxudan1-0/+6
JIRA: DOVETAIL-404 1. IPV6 sub-testcase's name includes [id], which will affects the regex. 2. Remove the [id] from the name of the sub_testcases. Change-Id: I51b567e8382fa4e701f8b32ba7ea195d49fd31eb Signed-off-by: xudan <xudan16@huawei.com>
2017-04-21use Functest tempest_custom test cases to run all Ipv6 test casesxudan1-6/+15
JIRA: DOVETAIL-401 1. modify all Ipv6 test cases, change tempest_smoke_serial/tempest_full_parallel into tempest_custom, add ID of each sub-testcases. 2. Some sub-testcases in ipv6.tc002, ipv6.tc003, ipv6.tc004 and ipv6.tc005 can not be found in the full tempest test case list. 2. disable testcases' script_result_acquired to enable tempest_custom run multiple times. 3. use build_tag-testcase_name(such as daily-master-79a6e392-28bb-4c40-9d88-a03bed99da2d-dovetail.ipv6.tc025) as the unique ID of each testcase. 4. comment out the progress bar, because it has some problems when print the report text. 5. Functest has changed the TEST_DB_URL from http://<testapi>:<port>/apv/v1 to http://<testapi>:<port>/apv/v1/results. 6. Functest has changed the TEST_DB_URL to file://<file_path>/<file_name>. see https://gerrit.opnfv.org/gerrit/#/c/33511/ Change-Id: I524f6702377f215f526158214842481eb44565be Signed-off-by: xudan <xudan16@huawei.com>
2017-04-18put dovetail test result back in log and not generte report fileMatthewLi1-1/+1
Change-Id: I2da5e7844f5ce034d4c0cfaa4592e6cd47002049 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-04-12dovetail tool: bugfix: test cases are erroneously judgedxudan1-61/+59
JIRA: DOVETAIL-375 JIRA: DOVETAIL-390 1. defcore test cases are erroneously judged because of: a) defcore results json data lack some keys b) some sub-testcases' names are the same as the prefix of another sub-testcases. 2. this patch removes the extra keys and judge sub-testcases accurately. 3. remove "store_type" in functest_config.yml and yardstick_config.yml. 4. change some log levels from info to debug. Change-Id: I5f126e70309409cac66ef9a0909f1573d325098b Signed-off-by: xudan <xudan16@huawei.com>
2017-04-01refstack result parseMatthewLi1-30/+27
JIRA: DOVETAIL-375 1, together with other testsuite from functest, such as tempest etc. this should merge after https://gerrit.opnfv.org/gerrit/#/c/31313/ 2, delete some unnecessary definition 3, bugfix of docker cp inside container, see https://build.opnfv.org/ci/view/dovetail/job/dovetail-compass-baremetal-debug-master/361/console 4, disable tempest testcaes since changed to refstack_defcore Change-Id: I67503e348af4a3f697f1b4106bfe81c05add8aa0 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-03-30dovetail tool: set a unique build_tag for each DoveTail testxudan1-3/+2
JIRA: DOVETAIL-379 1. create a uuid for each test, and set build_tag=daily-master-<uuid> 2. the reason it starts with "daily_master" is that it's the requirement of functest when push data to DB. Change-Id: I9a59d5695a9f0d83bf4330379f1949a2a04f876c Signed-off-by: xudan <xudan16@huawei.com>
2017-03-25dovetail tool: Add an option cmd for pushing results into DBxudan1-3/+3
JIRA: DOVETAIL-373 1. add an option --report/-r to choose the report type. 2. "-r file" will store results with files. 3. "-r http://192.168.135.2:8000/api/v1" will push results into DB. The DB url can replaced by user's local DB or official DB. 4. Default report type is defined as "report_dest" in dovetail_config.yml. 5. Yardstick can't push results of one testcase into DB now, so both "-r file" and "-r http://..." will be regarded as "file". Change-Id: Ieaa017ab59c5b15235bf306ef57cdc56beb22af8 Signed-off-by: xudan <xudan16@huawei.com>
2017-02-20dovetail tool: bugfix for dovetail report and docker buildMatthewLi1-2/+2
JIRA: DOVETAIL-353 JIRA: DOVETAIL-354 related patch: https://gerrit.opnfv.org/gerrit/#/c/29049/ Change-Id: Ic871abbda62dead7cf5e23e6117c9885d637c7fb Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-02-18Merge "dovetail tool: skip testcase status added"hongbo tian1-40/+58
2017-02-17dovetail tool: change the way to get the functest result filexudan1-5/+8
Functest has changed it's way to generate the local result file. see Gerrit: https://gerrit.opnfv.org/gerrit/#/c/28513/ Dovetail needs to do some adaptions to fit this change. JIRA: DOVETAIL-350 Change-Id: I681f74832f582f26c8874f7d2726c8588f430612 Signed-off-by: xudan <xudan16@huawei.com>
2017-02-17dovetail tool: skip testcase status addedMatthewLi1-40/+58
JIRA: DOVETAIL-344 1) testcase status, True/False --> PASS/SKIP/FAIL, so skipped testcase in tempest can be figured out details can be seen in https://etherpad.opnfv.org/p/dovetail_report 2) some variable name changed more readable, such as passed-->testcase_passed failed--> prepare_failed 3) "dovetail report" minor amend according to 1) Change-Id: Ibe8e50693fc474d78e7d0fb89257f34703248579 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-02-08Merge "dovetail tool: use python new-style classes which inherit from object"hongbo tian1-10/+10
2017-02-07dovetail tool: report.py bugfix and tempest skipped testcase addedMatthewLi1-11/+6
JIRA: DOVETAIL-343 1, bugfix https://build.opnfv.org/ci/view/dovetail/job/dovetail-compass-virtual-debug-master/lastSuccessfulBuild/artifact/results/dovetail_report.txt test result error 2, skipped testcase in tempest added 3, two testcase in tempest full testsuite added in debug.yml to debug Change-Id: I6bf371d3384e3755d6a41d951cfc9d1a55e00ab3 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-01-25dovetail tool: functest different testsuite support and yardstick testcase addedMatthewLi1-20/+44
JIRA: DOVETAIL-186 Change-Id: Id0eec6652c9595c660436dec89a309a77b16bc03 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-01-20dovetail tool: use python new-style classes which inherit from objectxudan1-10/+10
JIRA: DOVETAIL-187 Change-Id: Id6fa50a3a3cec7a18d27ca8feb383b53836a339b Signed-off-by: xudan <xudan16@huawei.com>
2017-01-11dovetail tool: switch logging to proper usageMatthewLi1-14/+14
JIRA: DOVETAIL-178 Change-Id: Ica814d4ff634c9159e4ecca888b8cc0f6c16e624 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-01-05release tag supportMatthewLi1-7/+9
JIRA: DOVETAIL-170 Change-Id: Id56e30fdb3fb5b652e2145dd406b71986997f200 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-12-22[dovetail tool]check and get results for each cmdLeo Wang1-24/+61
JIRA: DOVETAIL-166 Check the results of each cmds executed in test case 1. the results of pre_condition, post_condition, cmds need to be checked, so it can get a quick fail, dont need to go through the next step 2. it's more accurate to show where error occurred as early as possible 3. get results from shell scripts Change-Id: I5c1e59839c55b92de0e83e7e1eb552aa364b3f80 Signed-off-by: Leo Wang <grakiss.wanglei@huawei.com>
2016-12-20Merge "bugfix for duration missed"hongbo tian1-1/+1
2016-12-20bugfix for duration missedMatthewLi1-1/+1
JIRA: DOVETAIL-162 Change-Id: Idc78790e49ab3e3552d45b902cfdf8cc37fd2847 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-12-15dovetail tool: move conf/dovetail_config.py into utils/xudan1-1/+1
1. move conf/dovetail_config.py into utils/ 2. change the open file path in dovetail_config.py 3. do some modification about all 'from conf.dovetail_config import ***' JIRA: DOVETAIL-158 Change-Id: I7670e6de993899e4e2a604cbc501dc3339896ef1 Signed-off-by: xudan <xudan16@huawei.com>
2016-12-13[dovetail tool] support shell scripts for testcase validationLeo Wang1-11/+14
JIRA: DOVETAIL-46 1. for now a testcase has two kinds of validation types(functest, yardstick), and it is not enough to check the complete funcionality 2. add new validation type(shell) for extra validation of the test case to make result more accurate and more convincing. Change-Id: I45dca6b8dbd888757da163189d261f6e4dba5034 Signed-off-by: Leo Wang <grakiss.wanglei@huawei.com>
2016-12-13Merge "[WIP] Migrate to tox based environment"hongbo tian1-12/+16
2016-12-13[WIP] Migrate to tox based environmentLeif Madsen1-12/+16
Migrate the custom setup.py to a more formal setup environment using tox, in the same way that OpenStack configures their projects. Adds metadata to tox.ini from the setup.py file, and breaks out the dependencies from setup.py into requirements.txt and test-requirements.txt. Executes flake8 testing for pep8 coverage. Defaults to using Python 2 for the environment. Change-Id: Ieaea7d9f3dc227febd8fb1dc9142d478ba197a92 Signed-off-by: Leif Madsen <lmadsen@redhat.com> Signed-off-by: zshi <zshi@redhat.com>
2016-12-09[dovetail tool]: move config info from .py to config .yml fileMatthewLi1-2/+3
JIRA: DOVETAIL-145 Change-Id: Ia9b3c678f1a491e5daf2d280f05904ead3b0a296 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-12-02[WIP] dovetail report improvementMatthewLi1-44/+93
JIRA: DOVETAIL-49 1)details please refer to https://wiki.opnfv.org/display/dovetail/Dovetail+Reports 2)results please refer to one example https://etherpad.opnfv.org/p/dovetail_report Change-Id: I63a1ba3b6aa1ffe5fa043a11cd3777dd0e8d52d9 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-11-29dovetail tool: package all logger initializationxudan1-25/+53
1. For container.py, parser.py, report.py and testcase.py, they use functions in classes to create loggers. So each class will have its own logger. 2. run.py will create all the loggers by create_logs() function before the use of them. So logs will not be initialized until the call of create_logs(). 3. Move logger in run.py into function main(). JIRA: DOVETAIL-56 Change-Id: I53de090df270cc7a34762ff99cbc9115cfb09465 Signed-off-by: xudan <xudan16@huawei.com>
2016-11-26dovetail tool: testarea argument addedMatthewLi1-6/+10
JIRA: DOVETAIL-71 1) argument testarea is added 2) terminology "scenario" change to "testsuite" 3) related documents amended 4) all word "certification" changed to "compliance" Change-Id: I038a9e04bf83cfdac20f0c59adb1841bf5470584 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-11-17dovetail tool: reorganize dovetail_config.py and modify related filesxudan1-7/+11
1. move all sentencs and functions into class DovetailConfig 2. delete dict container_config for it has been replaced by dovetail_config 2. modify files that import dovetail_config, including container.py, parser.py, report.py, run.py, dovetail_logger.py and testcase.py JIRA: DOVETAIL-55 Change-Id: Ib5ca3cadd751075c879f39e8cb2eb2e0adcef6aa Signed-off-by: xudan <xudan16@huawei.com>
2016-11-15dovetail tool: replace the hard-coded "Tescase" and modify some classmethodLinghui Zeng1-11/+14
JIRA: DOVETAIL-45 1. In the testcase.py file, we replace the hard-coded "Tesecase" with "cls" or "self" based on the specific contexts. 2. For all the four committed files, we modify some classmethod into staticmethod because each method is basically just a function, called syntactically like a method, but without access to the object and its' internals (attributes and other methods). Change-Id: Ieb452f476a2d33ae9aca4c904ae7d2c92b68689e Signed-off-by: Linghui Zeng <linghui.zeng@huawei.com>
2016-11-15generate report data for reuseLeo Wang1-24/+48
JIRA: DOVETAIL-22 1. report text is string, can not be reused by other module, 2. generate report data first, then this data can be uploaded to database or be translated to report text Change-Id: Ieecd87c827537c825c87a0db4ff170ae490d75ae Signed-off-by: Leo Wang <grakiss.wanglei@huawei.com>
2016-11-10dovetail tool: 2 config files, compliance_set and proposed_testsMatthewLi1-1/+1
JIRA: DOVETAIL-50 1)'proposed_tests.yml' is added, which runs as the total tests run. The 'compliance_set.yml' will run as dovetail certification tests; 2)ipv6 test cases added accroding to 'docs/testsuites/ipv6/ipv6_all_testcases', tc026 missed temporarily,yardstick ipv6tc026 only supports 2 installers by now according to https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Cases 3)vimops001-006 added for VIM images operation test cases 4)nfvi001-002 added for vping ssh/userdata in fucntest 5)more test cases can be added in future in proposed_tests.yml, besides, the test cases can be adjusted in future in proposed_tests.yml Change-Id: I8a8634583549a802f6f64fbea651e7be4efa361c Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-11-08dovetail tool: ensure sub_testcase_list and testcase are not Nonezshi1-9/+18
JIRA: DOVETAIL-52 1) save return value of testcase.sub_testcase() in sub_testcase_list for multiple calls later 2) ensure sub_testcase_list is not called with None value 3) ensure testcase is not called with None value 4) identify None testcase with 'Undefined' in dovetail_report.txt Change-Id: Ifcf0c0846dc788575310f29932a4757752f1e3a3 Signed-off-by: zshi <zshi@redhat.com>
2016-10-21dovetail tool: rename report_file_path variablezshi1-4/+4
the value assigned to report_file_path is a file name, not a path. Change-Id: I338ae2bb5e836d26c711bfa9144f16d06c08deb7 Signed-off-by: zshi <zshi@redhat.com>
2016-10-14[dovetail_tool] Fix the python style scanned by flake8xudan1-26/+48
JIRA: DOVETAIL-34 Change-Id: Ib9a6dd254f1d77b0cbbfa66e8e1e44df1ce14a4b Signed-off-by: xudan <xudan16@huawei.com>
2016-10-12preparation for setup.pyMatthewLi1-0/+239
JIRA: DOVETAIL-26 1)change the directory scripts to dovetail, to make it unified in OPNFV and for setup.py 2)change the paths accordingly 3)this should be first merged, since docker is changed, then to add setup.py and run the local test Change-Id: I65974bde17f310ebbfe3c025532b05dae8752945 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>