summaryrefslogtreecommitdiffstats
path: root/dovetail/run.py
AgeCommit message (Collapse)AuthorFilesLines
2017-05-09Support Yardstick to push results to databasexudan1-12/+7
JIRA: DOVETAIL-421 1. Yardstick is working on supporting push results to mongoDB. see https://gerrit.opnfv.org/gerrit/#/c/34091/ 2. make Dovetail supporting push results to database too when user choose to report to database. Change-Id: Iee06ad6be59dad62ac4f985f4a982fb5ff93a592 Signed-off-by: xudan <xudan16@huawei.com>
2017-05-04Get results json data from databasexudan1-3/+6
JIRA: DOVETAIL-417 1. If report results with files, all results will be stored in local files. 2. If report to database, will get results from database and stored with file results.json Change-Id: Ic30037bd66cd37042f82b75fac2043a876e21c6f Signed-off-by: xudan <xudan16@huawei.com>
2017-05-01Tips on update docker to right version, > 1.8grkiss1-0/+1
JIRA: DOVETAIL-414 Dovetail requires docker at version > 1.8, so add some validation in code, and also versions of client and server should match with each other, or there will be unpredictable errors. Gives a notice to user to decide whether to update the docker to the latest version(1.17) Change-Id: I1331f5c19943a73e8fe25f76bd21628233e556f3 Signed-off-by: grakiss <grakiss.wanglei@huawei.com>
2017-04-25Bugfix: KeyError when tempest_custom runs wrongxudan1-2/+2
JIRA: DOVETAIL-406 1. When there are something wrong with the test case tempest_custom, it's details json data are "details": {}, So it will throw KeyError exception in this situation. 2. Catch this exception and set all sub_testcases FAIL. 3. Disable the exceed_max_retry_times(). Change-Id: I6bdfc5944a535625b2211a3bd8096a13b81b339e Signed-off-by: xudan <xudan16@huawei.com>
2017-04-21use Functest tempest_custom test cases to run all Ipv6 test casesxudan1-3/+3
JIRA: DOVETAIL-401 1. modify all Ipv6 test cases, change tempest_smoke_serial/tempest_full_parallel into tempest_custom, add ID of each sub-testcases. 2. Some sub-testcases in ipv6.tc002, ipv6.tc003, ipv6.tc004 and ipv6.tc005 can not be found in the full tempest test case list. 2. disable testcases' script_result_acquired to enable tempest_custom run multiple times. 3. use build_tag-testcase_name(such as daily-master-79a6e392-28bb-4c40-9d88-a03bed99da2d-dovetail.ipv6.tc025) as the unique ID of each testcase. 4. comment out the progress bar, because it has some problems when print the report text. 5. Functest has changed the TEST_DB_URL from http://<testapi>:<port>/apv/v1 to http://<testapi>:<port>/apv/v1/results. 6. Functest has changed the TEST_DB_URL to file://<file_path>/<file_name>. see https://gerrit.opnfv.org/gerrit/#/c/33511/ Change-Id: I524f6702377f215f526158214842481eb44565be Signed-off-by: xudan <xudan16@huawei.com>
2017-04-21Merge "put dovetail test result back in log and not generte report file"hongbo tian1-5/+5
2017-04-19set dovetail.log level by command linexudan1-0/+2
JIRA: DOVETAIL-400 1. set both the dovetail.log and screen log level by command --debug. 2. if --report http://testapi_url:port/api/v1/, change it into --report http://testapi_url:port/api/v1 Change-Id: I7cf697d4ac9072fc2d302a67ccde0f452d84be8f Signed-off-by: xudan <xudan16@huawei.com>
2017-04-18put dovetail test result back in log and not generte report fileMatthewLi1-5/+5
Change-Id: I2da5e7844f5ce034d4c0cfaa4592e6cd47002049 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-04-13Merge "dovetail tool: bugfix: test cases are erroneously judged"hongbo tian1-8/+19
2017-04-12dovetail tool: bugfix: test cases are erroneously judgedxudan1-8/+19
JIRA: DOVETAIL-375 JIRA: DOVETAIL-390 1. defcore test cases are erroneously judged because of: a) defcore results json data lack some keys b) some sub-testcases' names are the same as the prefix of another sub-testcases. 2. this patch removes the extra keys and judge sub-testcases accurately. 3. remove "store_type" in functest_config.yml and yardstick_config.yml. 4. change some log levels from info to debug. Change-Id: I5f126e70309409cac66ef9a0909f1573d325098b Signed-off-by: xudan <xudan16@huawei.com>
2017-04-10refstack: add manually method to let user can set its tempest.conf and ↵MatthewLi1-0/+7
defcore.txt JIRA: DOVETAIL-394 for refstack, userguide is listed in https://gerrit.opnfv.org/gerrit/#/c/33049/ in CI, use the automate way to generate tempest.conf to let refstack run against SUT, i.e., defcore.tc001.yml will provide a way to let user can let refstack run against SUT manually, i.e. defcore.tc002.yml since the tempest.conf autometely generated may not satisfied with SUT, users can adjust its tempest.conf to let the defcore testcases pass. Change-Id: I5ed212c9c6d864d352d1c497ea039cc553461158 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-04-01Merge "dovetail tool: check push_to_db success or fail"hongbo tian1-9/+22
2017-03-31dovetail tool: change results dir into $DOVETAIL_HOME/resultsxudan1-0/+7
JIRA: DOVETAIL-381 Now DoveTail's results dir is defined in $DOVETAIL_HOME/dovetail/conf/dovetail_config.yml, and its default value is /home/opnfv/dovetail/results. Since customer's jumphost may not have this dir and they don't want to create it, Dovetail needs to change this dir into $DOVETAIL_HOME/results. Change-Id: I616fb2da87bcfaabfd78d8762897e378168f694d Signed-off-by: xudan <xudan16@huawei.com>
2017-03-31dovetail tool: check push_to_db success or failxudan1-9/+22
JIRA: DOVETAIL-380 If Functest fails to push results to db, there is just an error log in functest.log Dovetail needs to know the failure. Dovetail will get the results from db with keys, build_tag and testcase. If it can get the results data, it means push_to_db success. Fix the duration time wrong. Change-Id: If03bdf0ceb2f1a9d422381cdf06e1fd207358bd5 Signed-off-by: xudan <xudan16@huawei.com>
2017-03-30dovetail tool: set a unique build_tag for each DoveTail testxudan1-0/+4
JIRA: DOVETAIL-379 1. create a uuid for each test, and set build_tag=daily-master-<uuid> 2. the reason it starts with "daily_master" is that it's the requirement of functest when push data to DB. Change-Id: I9a59d5695a9f0d83bf4330379f1949a2a04f876c Signed-off-by: xudan <xudan16@huawei.com>
2017-03-29dovetail tool: offline supportMatthewLi1-0/+5
JIRA: DOVETAIL-164 usage: dovetail run --testsuite <testsuite> --offline 1,if --offline given, can run offline by using local docker images (if there is no images, will get an error in log, images can be pulled from dockerhub or downloaded from artifacts.opnfv.org ) 2,if --offline not given, will run online, which means pull docker image from dockerhub Change-Id: I87ab35c82e59fb13a7a88c38d233e89285efc4d3 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-03-25dovetail tool: Add an option cmd for pushing results into DBxudan1-3/+19
JIRA: DOVETAIL-373 1. add an option --report/-r to choose the report type. 2. "-r file" will store results with files. 3. "-r http://192.168.135.2:8000/api/v1" will push results into DB. The DB url can replaced by user's local DB or official DB. 4. Default report type is defined as "report_dest" in dovetail_config.yml. 5. Yardstick can't push results of one testcase into DB now, so both "-r file" and "-r http://..." will be regarded as "file". Change-Id: Ieaa017ab59c5b15235bf306ef57cdc56beb22af8 Signed-off-by: xudan <xudan16@huawei.com>
2017-03-21dovetail tool: remove installer and CI related variablesxudan1-12/+1
JIRA: DOVETAIL-371 1. remove all the envs in functest_config.yml and yardstick_config.yml 2. remove all relative cmds in cmd_config.yml 3. remove all the envs when running a new functest/yardstick container 4. add some variables just for functest pushing results to db 5. get the external network name for yardstick 6. make --debug/-d just control the log level shown on the screen 7. set requests=2.10.0 since python-openstackclient needs 'requests!=2.12.2,!=2.13.0,>=2.10.0' Change-Id: I9f941e09d067fc95a14c3c9be1374f41df4a9a16 Signed-off-by: xudan <xudan16@huawei.com>
2017-01-11dovetail tool: switch logging to proper usageMatthewLi1-7/+7
JIRA: DOVETAIL-178 Change-Id: Ica814d4ff634c9159e4ecca888b8cc0f6c16e624 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-12-22[dovetail tool]check and get results for each cmdLeo Wang1-2/+2
JIRA: DOVETAIL-166 Check the results of each cmds executed in test case 1. the results of pre_condition, post_condition, cmds need to be checked, so it can get a quick fail, dont need to go through the next step 2. it's more accurate to show where error occurred as early as possible 3. get results from shell scripts Change-Id: I5c1e59839c55b92de0e83e7e1eb552aa364b3f80 Signed-off-by: Leo Wang <grakiss.wanglei@huawei.com>
2016-12-20Merge "bugfix for duration missed"hongbo tian1-0/+4
2016-12-20bugfix for duration missedMatthewLi1-0/+4
JIRA: DOVETAIL-162 Change-Id: Idc78790e49ab3e3552d45b902cfdf8cc37fd2847 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-12-15dovetail tool: move conf/dovetail_config.py into utils/xudan1-1/+1
1. move conf/dovetail_config.py into utils/ 2. change the open file path in dovetail_config.py 3. do some modification about all 'from conf.dovetail_config import ***' JIRA: DOVETAIL-158 Change-Id: I7670e6de993899e4e2a604cbc501dc3339896ef1 Signed-off-by: xudan <xudan16@huawei.com>
2016-12-14dovetail tool: improve cli to update non-envs automaticallyxudan1-40/+66
1. modify cmd_config.yml a. change 'env' and 'non-env' into 'config' and 'control' b. add subsection 'path' for each block in 'config' c. delete 'DEPLOY_SCENARIO' and 'DEPLOY_TYPE' d. use 'func_tag' and 'yard_tag' to control container's tag 2. if cmd in config a. if its path is envs, it will be used to update envs in dovetail_config b. if its path is non_envs, it will be used to update other value in the dict JIRA: DOVETAIL-70 Change-Id: I1cf9bfcd1a19294a390a85bce6e458ce50672b7f Signed-off-by: xudan <xudan16@huawei.com>
2016-12-13[dovetail tool] support shell scripts for testcase validationLeo Wang1-26/+6
JIRA: DOVETAIL-46 1. for now a testcase has two kinds of validation types(functest, yardstick), and it is not enough to check the complete funcionality 2. add new validation type(shell) for extra validation of the test case to make result more accurate and more convincing. Change-Id: I45dca6b8dbd888757da163189d261f6e4dba5034 Signed-off-by: Leo Wang <grakiss.wanglei@huawei.com>
2016-12-13Merge "dovetail tool: add debug option to control the log level showing in ↵hongbo tian1-0/+2
screen"
2016-12-09[dovetail tool]: move config info from .py to config .yml fileMatthewLi1-14/+15
JIRA: DOVETAIL-145 Change-Id: Ia9b3c678f1a491e5daf2d280f05904ead3b0a296 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-12-05dovetail tool: add debug option to control the log level showing in screenxudan1-0/+2
1. add debug option in cmd_config.yml for controling log showing in screen. 2. rename existent DEBUG as CON_DEBUG because it is used for functest/yardstick container to control the log level. 3. fix the bug in function update_config_envs(). If one key is not in system's envs, os.environ[key] get an error with ErrorKey. JIRA: DOVETAIL-148 Change-Id: If3d75109f51e2e91743b8f1134a76da42e57383a Signed-off-by: xudan <xudan16@huawei.com>
2016-12-02Merge "dovetail tool: reorganize statements in class DovetailConfig"hongbo tian1-0/+2
2016-12-02[WIP] dovetail report improvementMatthewLi1-2/+9
JIRA: DOVETAIL-49 1)details please refer to https://wiki.opnfv.org/display/dovetail/Dovetail+Reports 2)results please refer to one example https://etherpad.opnfv.org/p/dovetail_report Change-Id: I63a1ba3b6aa1ffe5fa043a11cd3777dd0e8d52d9 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-11-30Merge "dovetail tool: run.py execution always wipes of results files"hongbo tian1-4/+16
2016-11-29dovetail tool: run.py execution always wipes of results filesxudan1-4/+16
1. move function clean_results_dir() into run.py 2. call this function at the beginning of function main() 3. this can only work correctly after all log initializations packaged, see JIRA: DOVETAIL-56 and gerrit https://gerrit.opnfv.org/gerrit/#/c/24059/ JIRA: DOVETAIL-80 Change-Id: I1cdda8f6d43f21c45c0537868cb7ed10d14ea9df Signed-off-by: xudan <xudan16@huawei.com>
2016-11-29dovetail tool: reorganize statements in class DovetailConfigxudan1-0/+2
1. move some statements in class DovetailConfig into function load_config_files() 2. call load_config_files() in run.py 3. this can only work correctly after moving clean_results_dir() into run.py, see JIRA:DOVETAIL-80 and gerrit https://gerrit.opnfv.org/gerrit/#/c/24929/ JIRA: DOVETAIL-55 Change-Id: I217c1c21dc6ba8849da7c00564f62d3c22cdc391 Signed-off-by: xudan <xudan16@huawei.com>
2016-11-29dovetail tool: To exec testsuite loading after command line arguments is TrueMatthewLi1-2/+2
JIRA: DOVETAIL-142 Change-Id: I1d8a444580755b4f4d9a0f7098b4b17ea66d6f58 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-11-29dovetail tool: package all logger initializationxudan1-8/+24
1. For container.py, parser.py, report.py and testcase.py, they use functions in classes to create loggers. So each class will have its own logger. 2. run.py will create all the loggers by create_logs() function before the use of them. So logs will not be initialized until the call of create_logs(). 3. Move logger in run.py into function main(). JIRA: DOVETAIL-56 Change-Id: I53de090df270cc7a34762ff99cbc9115cfb09465 Signed-off-by: xudan <xudan16@huawei.com>
2016-11-26dovetail tool: testarea argument addedMatthewLi1-14/+32
JIRA: DOVETAIL-71 1) argument testarea is added 2) terminology "scenario" change to "testsuite" 3) related documents amended 4) all word "certification" changed to "compliance" Change-Id: I038a9e04bf83cfdac20f0c59adb1841bf5470584 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-11-17dovetail tool: reorganize dovetail_config.py and modify related filesxudan1-13/+11
1. move all sentencs and functions into class DovetailConfig 2. delete dict container_config for it has been replaced by dovetail_config 2. modify files that import dovetail_config, including container.py, parser.py, report.py, run.py, dovetail_logger.py and testcase.py JIRA: DOVETAIL-55 Change-Id: Ib5ca3cadd751075c879f39e8cb2eb2e0adcef6aa Signed-off-by: xudan <xudan16@huawei.com>
2016-11-15Fix option handling in case of no '--tag' option.Tomofumi Hayashi1-2/+2
In case of default, 'None' is in kwargs['tag'] so need to check it in option handling. The code is to fix it. Change-Id: Iaadf7e6f0f064dda819970346a810a6a8b47c74c Signed-off-by: Tomofumi Hayashi <tohayash@redhat.com>
2016-11-15Merge "Add "docker tag" option for each testcase containers."hongbo tian1-0/+22
2016-11-10Add "docker tag" option for each testcase containers.Tomofumi Hayashi1-0/+22
JIRA: DOVETAIL-69 This commit is to add new option to specify docker tags for each test containers, such as '--TAG "functest:stable"'. Change-Id: Ica11e16abfb21649a3eea9c7d7d5cd31502d2e21 Signed-off-by: Tomofumi Hayashi <tohayash@redhat.com>
2016-11-10dovetail tool: 2 config files, compliance_set and proposed_testsMatthewLi1-1/+1
JIRA: DOVETAIL-50 1)'proposed_tests.yml' is added, which runs as the total tests run. The 'compliance_set.yml' will run as dovetail certification tests; 2)ipv6 test cases added accroding to 'docs/testsuites/ipv6/ipv6_all_testcases', tc026 missed temporarily,yardstick ipv6tc026 only supports 2 installers by now according to https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Cases 3)vimops001-006 added for VIM images operation test cases 4)nfvi001-002 added for vping ssh/userdata in fucntest 5)more test cases can be added in future in proposed_tests.yml, besides, the test cases can be adjusted in future in proposed_tests.yml Change-Id: I8a8634583549a802f6f64fbea651e7be4efa361c Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-11-08dovetail tool: ensure sub_testcase_list and testcase are not Nonezshi1-0/+4
JIRA: DOVETAIL-52 1) save return value of testcase.sub_testcase() in sub_testcase_list for multiple calls later 2) ensure sub_testcase_list is not called with None value 3) ensure testcase is not called with None value 4) identify None testcase with 'Undefined' in dovetail_report.txt Change-Id: Ifcf0c0846dc788575310f29932a4757752f1e3a3 Signed-off-by: zshi <zshi@redhat.com>
2016-11-02dovetail tool: Add cli mechanism for changing envs at run-timexudan1-5/+39
JIRA: DOVETAIL-23 1. Add file cmd_config.yml for defining arguments and options. 2. Using python click module to add and parse cmds in cmd_config.yml 3. Using envs options to update envs in yardstick/functest config file Change-Id: Ie513c0f0a1ed76f29408419a4bcbbcf21334bb3f Signed-off-by: xudan <xudan16@huawei.com>
2016-10-21Simplify getting pre_condition cmdszshi1-2/+1
Change-Id: I8ca04cbdc4236e9ff754bad816bb7282673b5b55 Signed-off-by: zshi <zshi@redhat.com>
2016-10-14[dovetail_tool] Fix the python style scanned by flake8xudan1-10/+12
JIRA: DOVETAIL-34 Change-Id: Ib9a6dd254f1d77b0cbbfa66e8e1e44df1ce14a4b Signed-off-by: xudan <xudan16@huawei.com>
2016-10-12preparation for setup.pyMatthewLi1-0/+84
JIRA: DOVETAIL-26 1)change the directory scripts to dovetail, to make it unified in OPNFV and for setup.py 2)change the paths accordingly 3)this should be first merged, since docker is changed, then to add setup.py and run the local test Change-Id: I65974bde17f310ebbfe3c025532b05dae8752945 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>