summaryrefslogtreecommitdiffstats
path: root/dovetail/test_runner.py
AgeCommit message (Collapse)AuthorFilesLines
2017-08-28Merge "Bugfix: functest only keeps the last test case's log"Leo wang1-0/+24
2017-08-28Bugfix: functest only keeps the last test case's logxudan1-0/+24
JIRA: DOVETAIL-490 1. Dovetail just keeps the last tempest/defcore/ipv6/bgpvpn test's log file. 2. All log files should be kept, so users and developers can know the reason of failed test cases. 3. tempest logs will be stored in directory tempest_logs/ 4. defcore logs will be stored in directory defcore_logs/ 5. IPv6 logs will be stored in directory ipv6_logs/ 6. bgpvpn logs will be stored in directory bgpvpn_logs/ 7. All vping logs are in functest.log Change-Id: I79eae79cb32d65ce1ada3dd6f4c6dfd3945fc512 Signed-off-by: xudan <xudan16@huawei.com>
2017-08-23Bugfix: pre_condition cmds is just executed oncexudan1-11/+7
JIRA: DOVETAIL-489 1. All the cmds defined in pre_condition is just executed once for each type i.e. functest, yardstick, bottlenecks... 2. The copy cmds defined in each functest test cases are just executed for the first test case rather than all test cases. 3. Make the cmds to be executed for each test case. Change-Id: I571f0eb9dd3cab77fc8597e5a9b5246398804d3e Signed-off-by: xudan <xudan16@huawei.com>
2017-08-03Add resiliency test caes into proposed_testsxudan1-1/+9
JIRA: DOVETAIL-474 Add Bottlenecks resiliency test case "posca_factor_ping" into proposed_tests. Change-Id: I76aea651014fb4722ddae07559f417d3f840eee2 Signed-off-by: xudan <xudan16@huawei.com>
2017-06-28Modify Dovetail logsxudan1-18/+18
JIRA: DOVETAIL-450 Change-Id: I795679c9559d1bead3aecefff28d21b6567aac2f Signed-off-by: xudan <xudan16@huawei.com>
2017-05-30sdnvpn offline support in dovetailMatthewLi1-1/+10
JIRA: DOVETAIL-441 Change-Id: I709c62a36c65ef0b4da69c5508f7a8273e68dc2b Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-05-25Change the directories of results and some files to DOVETAIL_HOMExudan1-1/+1
JIRA: DOVETAIL-438 1. Need an env variable DOVETAIL_HOME in env_config.sh 2. env_config.sh, pod.yaml and id_rsa should be put in path DOVETAIL_HOME/pre_config 3. the results will also be put in this path 4. no need to give the path of openstack credential file while running a testsuite 5. if running with the code, source the env_config.sh file before running source $DOVETAIL_HOME/env_config.sh dovetail run --testsuite <testsuite_name> 6. if running with Dovetail docker image docker run -it --privileged=true -e DOVETAIL_HOME=<dovetail_home_path> -v <dovetail_home_path>:<dovetail_home_path> -v /var/run/docker.sock:/var/run/docker.sock opnfv/dovetail:latest /bin/bash Change-Id: Ia23078f27a34e293d0075e49c673930f41069cb6 Signed-off-by: xudan <xudan16@huawei.com>
2017-05-24SDNVPN test cases failed in CIxudan1-6/+26
JIRA: DOVETAIL-439 1. SDNVPN test cases run in CI failed, because the file /home/opnfv/repos/sdnvpn/sdnvpn/test/functest/config.yaml is empty. 2. copy the file userconfig/sdnvpn_config_testcase1/2/4/8.yaml to /home/opnfv/repos/sdnvpn/sdnvpn/test/functest/config.yaml in Functest container Change-Id: Ie904f83c755efc0d9e0abe5dcd44fe4a5056123e Signed-off-by: xudan <xudan16@huawei.com>
2017-04-21use Functest tempest_custom test cases to run all Ipv6 test casesxudan1-1/+2
JIRA: DOVETAIL-401 1. modify all Ipv6 test cases, change tempest_smoke_serial/tempest_full_parallel into tempest_custom, add ID of each sub-testcases. 2. Some sub-testcases in ipv6.tc002, ipv6.tc003, ipv6.tc004 and ipv6.tc005 can not be found in the full tempest test case list. 2. disable testcases' script_result_acquired to enable tempest_custom run multiple times. 3. use build_tag-testcase_name(such as daily-master-79a6e392-28bb-4c40-9d88-a03bed99da2d-dovetail.ipv6.tc025) as the unique ID of each testcase. 4. comment out the progress bar, because it has some problems when print the report text. 5. Functest has changed the TEST_DB_URL from http://<testapi>:<port>/apv/v1 to http://<testapi>:<port>/apv/v1/results. 6. Functest has changed the TEST_DB_URL to file://<file_path>/<file_name>. see https://gerrit.opnfv.org/gerrit/#/c/33511/ Change-Id: I524f6702377f215f526158214842481eb44565be Signed-off-by: xudan <xudan16@huawei.com>
2017-04-01refstack result parseMatthewLi1-3/+4
JIRA: DOVETAIL-375 1, together with other testsuite from functest, such as tempest etc. this should merge after https://gerrit.opnfv.org/gerrit/#/c/31313/ 2, delete some unnecessary definition 3, bugfix of docker cp inside container, see https://build.opnfv.org/ci/view/dovetail/job/dovetail-compass-baremetal-debug-master/361/console 4, disable tempest testcaes since changed to refstack_defcore Change-Id: I67503e348af4a3f697f1b4106bfe81c05add8aa0 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-03-30Merge "dovetail tool: change function pull_image"hongbo tian1-3/+4
2017-03-30dovetail tool: refstack testcase integrationMatthewLi1-0/+5
JIRA: DOVETAIL-370 Change-Id: I60c9f431358f848e24abcb865c4b2ca8c3b2d843 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-03-30dovetail tool: change function pull_imagexudan1-3/+4
JIRA: DOVETAIL-378 1. test_runner checks the result of cmd "pull image". If fail to pull, then break the running. 2. pull_image will return None when there are errors, otherwise return image name. 3. if there is an old image named the same as the new one, the old one's tag will be changed into <none>. In this case, the old image needs to be removed. 4. if there are some containers using this old image, the remove will be skipped rather than logging error. Change-Id: I940cb18911112d5e13f3c14f87ddbdbaced3538a Signed-off-by: xudan <xudan16@huawei.com>
2017-03-29dovetail tool: offline supportMatthewLi1-2/+10
JIRA: DOVETAIL-164 usage: dovetail run --testsuite <testsuite> --offline 1,if --offline given, can run offline by using local docker images (if there is no images, will get an error in log, images can be pulled from dockerhub or downloaded from artifacts.opnfv.org ) 2,if --offline not given, will run online, which means pull docker image from dockerhub Change-Id: I87ab35c82e59fb13a7a88c38d233e89285efc4d3 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-03-20dovetail tool: change name of credentials from creds to openrcxudan1-1/+1
1. change file name from openstack.creds to openrc.sh 2. change the cmd --creds to --openrc/-o 3. now the path of --openrc can be relative path no restrict to abspath 4. remove the useless "work_dir" in dovetail_config.yml JIRA: DOVETAIL-369 JIRA: DOVETAIL-367 Change-Id: I2592438eef060f84132ff7fa226aea93ea947362 Signed-off-by: xudan <xudan16@huawei.com>
2017-02-17dovetail tool: skip testcase status addedMatthewLi1-10/+10
JIRA: DOVETAIL-344 1) testcase status, True/False --> PASS/SKIP/FAIL, so skipped testcase in tempest can be figured out details can be seen in https://etherpad.opnfv.org/p/dovetail_report 2) some variable name changed more readable, such as passed-->testcase_passed failed--> prepare_failed 3) "dovetail report" minor amend according to 1) Change-Id: Ibe8e50693fc474d78e7d0fb89257f34703248579 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-01-11dovetail tool: switch logging to proper usageMatthewLi1-1/+1
JIRA: DOVETAIL-178 Change-Id: Ica814d4ff634c9159e4ecca888b8cc0f6c16e624 Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2016-12-26dovetail tool: reorganize function prepare_cmd in testcase.pyxudan1-3/+3
1. move the common statements in prepare_cmd of class FunctestTestcase and Testcase into parse_cmd. JIRA: DOVETAIL-163 Change-Id: I61a0a24b47cfc9e8d49ca9c62fafbac7cce21111 Signed-off-by: xudan <xudan16@huawei.com>
2016-12-22[dovetail tool]check and get results for each cmdLeo Wang1-11/+67
JIRA: DOVETAIL-166 Check the results of each cmds executed in test case 1. the results of pre_condition, post_condition, cmds need to be checked, so it can get a quick fail, dont need to go through the next step 2. it's more accurate to show where error occurred as early as possible 3. get results from shell scripts Change-Id: I5c1e59839c55b92de0e83e7e1eb552aa364b3f80 Signed-off-by: Leo Wang <grakiss.wanglei@huawei.com>
2016-12-14[dovetail tool] support shell scripts for testcase validationLeo Wang1-1/+1
JIRA: DOVETAIL-46 1. for now a testcase has two kinds of validation types(functest, yardstick), and it is not enough to check the complete funcionality 2. add new validation type(shell) for extra validation of the test case to make result more accurate and more convincing. Change-Id: I5c049a71f11cca71a7914f8af704c1983aba3dca Signed-off-by: Leo Wang <grakiss.wanglei@huawei.com>
2016-12-13[dovetail tool] support shell scripts for testcase validationLeo Wang1-0/+100
JIRA: DOVETAIL-46 1. for now a testcase has two kinds of validation types(functest, yardstick), and it is not enough to check the complete funcionality 2. add new validation type(shell) for extra validation of the test case to make result more accurate and more convincing. Change-Id: I45dca6b8dbd888757da163189d261f6e4dba5034 Signed-off-by: Leo Wang <grakiss.wanglei@huawei.com>