aboutsummaryrefslogtreecommitdiffstats
path: root/functest/ci/run_tests.py
AgeCommit message (Collapse)AuthorFilesLines
2017-08-18Merge "Fix summary in run_tests.py"Cedric Ollivier1-6/+12
2017-08-18Fix summary in run_tests.pyCédric Ollivier1-6/+12
It fixes summary() when several tests have not been executed after a blocking test in failure. Change-Id: I3c0af4c8083ffa76f786756d409c09e018bbecc8 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-08-18Put domain name in CONSTAlex Yang1-0/+2
Domain name is not obtained in source_rc_file() but used in the patch https://gerrit.opnfv.org/gerrit/#/c/39109/. Change-Id: Iceaf352f7ad5e15f15906f0e3aa1c5ab5ae30a42 Signed-off-by: Alex Yang <yangyang1@zte.com.cn>
2017-08-16Clean run_tests.py and the related utCédric Ollivier1-78/+62
It enhances run_tests.py as proposed in [1]. It also prints all skipped tests in summary and fixes copyright headers. All the related unit tests have been updated too. [1] https://jira.opnfv.org/browse/FUNCTEST-832 JIRA: FUNCTEST-832 Change-Id: I59b96422bc7942ecd6270c45ab7a3fb603c13ccb Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-07-02Apply restrictive file permissionsCédric Ollivier1-3/+0
It conforms with [1] by creating a new venv which checks the unix permissions. As jjobs call Functest console scripts [2], all perms can be 644. Dockerfiles are updated as well. [1] https://security.openstack.org/guidelines/dg_apply-restrictive-file-permissions.html [2] https://gerrit.opnfv.org/gerrit/#/c/36805/ Depends-On: I9209e6efa1b493e24135402a46df72aaa14115d1 Change-Id: I31bc7f12b775928845e23b6b40288b0a50b87219 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-06-30Define console_scriptsCédric Ollivier1-2/+5
Only prepare_env.py and run_tests.py remain executable to avoid breaking the gate. As soon as the jenkins jobs are updated to run them from $PATH, tox will be updated to protect against chmod a+x py files. Dockerfile will be simplified as well. Change-Id: Iac1a7ceb6b344e800177226ea39919826f6d3735 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-06-26Don't display functest report for single test casesjose.lausuch1-10/+12
When executing a single test case, it is not neccessary to show duplicated information about the result of the test case. The functest report only makes sense when running multiple tests. e.g. functest testcase run vping_ssh The report is still showed when doing: functest testcase run all Change-Id: I92d97d6111ecd9484a14c72cf4bb0c2bbe3ae6bf Signed-off-by: jose.lausuch <jose.lausuch@ericsson.com>
2017-06-20Remove all references to /home/opnfv/repos/functestCédric Ollivier1-3/+4
It also removes relative (and incomplete) references which could be defined in modules instead of this global configuration. Change-Id: Ie20dc3547e49b6224aa8100cd380d37b87ece5a9 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-30Merge "Define create_snapshot() and clean() in TestCase"Cedric Ollivier1-27/+23
2017-05-30Define create_snapshot() and clean() in TestCaseCédric Ollivier1-27/+23
They replace the former calls in run_tests which are not suitable for all test cases. Now any test case can define how to clean its resources. If the snapshot cannot be created, the test case is considered as failed. Only a message is printed if any failure during cleaning. It also defines a new class called OSGCTestCase useful for test cases which don't clean their OpenStack resources. All test cases located in opnfv_tests/openstack inherit from it to keep the global behavior unchanged. It also deletes exit instructions in openstack_clean and openstack_snapshot, removes clean flags in testcases.yaml and updates the related utils. All Docs are modified as well. JIRA: FUNCTEST-438 Change-Id: I8938e6255708012380389763a24059ace4ce45d8 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-28Switch shebangs to /usr/bin/env pythonCédric Ollivier1-1/+1
It modified the shebang to /usr/bin/env which is considered as more flexible. Change-Id: I9c8cdb4243d41a83d524499cc91060d3700e008b Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-23Bugfix: fix two typosLinda Wang1-1/+1
1. In run_tests.py, test case name is shown before running: logger.error(kwargs['test']) -> logger.info(kwargs['test']) 2. In tempest.py, Verication -> Verification Change-Id: I117838cdb4b0f920c4cf855645d910e623e06868 Signed-off-by: Linda Wang <wangwulin@huawei.com>
2017-05-19Print stack when catching exceptions in run_tests.pyCédric Ollivier1-7/+8
Change-Id: Id94475dd519ce04b87da470f6e6bb5a1c1aa2d75 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-16Define Runner classCédric Ollivier1-202/+202
It simply converts run_tests.py's functions to Runner's methods. This proposal could be enhanced in a second step to fully leverage on object programming. It defines as instance variables the former static variables to allow multiple runs (it's mandatory for unit testing). All false positives in unit tests are fixed but run_tests.py is still not fully covered. Change-Id: I8f91c132aa8b2248041f31e46238dd5598344d34 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-16Add enabled flag in testcases.yamljose.lausuch1-0/+7
This will allow enable/disable test cases directly using true/false. Change-Id: Iaa7d051ebb8dd9e5a6f835a823f58f1e17a87b8d Signed-off-by: jose.lausuch <jose.lausuch@ericsson.com>
2017-05-16Switch from generate_report to PrettyTableCédric Ollivier1-26/+24
run_tests.py now relies on PrettyTable as most of the openstack clients. generate_report.py and its related unit tests are simply removed. It sets padding_width=5 in testcase.py too to conform with run_tests.py. Now report is printed in every case. Change-Id: Id9ce93f984503f25d6a2150482f397853fa3dd64 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-16Fix how to get tier_nameCédric Ollivier1-1/+1
The tier name was incorrect when passing a test case as run_test.py arg. Change-Id: I6f439fc7b90facb3f0553798bd1fe06ac9f09475 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-06Implement TestCase __str__()Cédric Ollivier1-2/+1
It's designed to be called by run_tests.py when printing the results of test cases and when generating the global report. Change-Id: If4a6f023ef2344bbc4f940d07dde4b776dce5d68 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-05Merge "Print the real testcase duration"Jose Lausuch1-11/+7
2017-05-05Print the real testcase durationCédric Ollivier1-11/+7
TestCase offers a new public method to calculate and print the duration. It also adds the related unit tests and adapts run_tests.py. Change-Id: Ib6a7e637dafacb9027146199aeb033c2dcb986c6 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-05Replace CONST.* by getattribute/setattrjose.lausuch1-11/+12
Directories affected: - ci - core and respective unit tests Change-Id: I6a3d5aa68de29fc5a37ae543a067ff797eba33e6 Signed-off-by: jose.lausuch <jose.lausuch@ericsson.com>
2017-05-04Delete functest.utils.functest_loggerCédric Ollivier1-4/+6
It deletes functest.utils.functest_logger and the related unit tests. Then it modifies all functest modules to get all loggers via logging.getLogger(). __name__ is mainly used as getLogger arg as proposed by logging [1]. All loggers and handlers are now defined via functest/ci/logging.ini instead of a dict loaded by an external json file. Now only warn messages and info messages from ci and core packages are printed in console. [1] https://docs.python.org/2/library/logging.html Change-Id: Ic192855e0f9bf94825d8f7ec73549a0f3b8d44c5 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-04-26Switch TestCase attribute criteria to resultCédric Ollivier1-1/+1
It mainly avoids mixing input and output. Criteria is now an input set in functest/ci/testcases.yaml and then must be passed as __init__() args (which will be proposed in an additional change). Then it also renames the related TestCase method to check_result(). Change-Id: Ifc3c8e3ea6cde7e3edf7174bed4bf2bf0894e8e3 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-04-11Modify TestCase constructor attributesCédric Ollivier1-1/+2
Every feature/testcase now allows receiving the data defined in testcases.yaml as args (name is renamed to case_name). From the time being, only project and case names are handled. Next pending patches will add criteria, cmd and repo to this list. It keeps the default values for case names except for features which will be aggregated into Feature. Change-Id: Id742d100b8183d7f10894c24ae6879d1b2b60ac9 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-04-04Add case_name as constructor argCédric Ollivier1-1/+1
It allows managing multiple TestCase names with only one TestCase module. It is mainly required by odl which implements: - odl, - odl_netvirt, - fds. It also renames case to case_name in Features to conform with TestCases. JIRA: FUNCTEST-762 Change-Id: Ie254f754a0ea3077a8afda1c470528d38c79478f Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-03-27Rename testcase_base to testcaseCédric Ollivier1-3/+3
It simply removes the useless suffix. Change-Id: Ib9a682b57d8f33518359d817a593bb3786577390 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-03-27Switch TestcaseBase to TestCaseCédric Ollivier1-2/+2
It partially reverts "Rename TestCasesBase into TestcaseBase" [1] and removes the useless suffix too. [1] https://gerrit.opnfv.org/gerrit/#/c/25391/ Change-Id: Iddf2ec8d6c1dafe6e4c6713c2e69969613317471 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-03-07Merge "delete ft_constants and file functest_constants.py"Morgan Richomme1-5/+0
2017-03-07delete ft_constants and file functest_constants.pySerenaFeng1-5/+0
delete no longer used ft_constants in run_test.py substitute ft_constants with CONST in foundation.py and remove useless str() finally delete not-used file functest_constants.py Change-Id: I109a5409e501b610f41aae7d75c55ca745a3d4c1 Signed-off-by: SerenaFeng <feng.xiaowei@zte.com.cn>
2017-03-02Merge "Remove exec_test.sh"Jose Lausuch1-14/+3
2017-03-01Remove exec_test.shjose.lausuch1-14/+3
After the patch https://gerrit.opnfv.org/gerrit/#/c/29403/ all the test cases in functest have been adapted to the new framework using modules/classes. This patch removes the need for having the script exec_tests.sh which is not used any more by the framework. Change-Id: I7547c908d2eca60edf8c867db920c7da4f23d312 Signed-off-by: jose.lausuch <jose.lausuch@ericsson.com>
2017-03-01Add clean_flag to test casesjose.lausuch1-2/+4
Some test cases such as tempest or those using SNAPS have a built-in cleaup or teardown mechanism. This patch helps to save some time by bypassing the openstack snapshot and cleanup when running those kind of tests cases. Change-Id: I50e7a17937a813532b3756474158416d3e85b922 Signed-off-by: jose.lausuch <jose.lausuch@ericsson.com>
2017-02-23Use new-style classes which inherit from objectLinda Wang1-1/+1
JIRA: FUNCTEST-707 Change-Id: Ieaa888375136eddbbe003a24b32bf09fd0f92923 Signed-off-by: Linda Wang <wangwulin@huawei.com>
2017-02-14Merge changes from topic 'revert_write_to_file'Jose Lausuch1-1/+1
* changes: Add can_dump_request_to_file decorator Revert "write test results to a local file"
2017-02-13Revert "write test results to a local file"Cédric Ollivier1-1/+1
This reverts commit ee21af78fbfc93e888acda121f08d2b216dd0159. Change-Id: If7c8d3b645073574862eb67225b1e11b0dd7a021 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-02-09Refactoring of args and parser variableashishk19941-33/+43
It also fixes one issue when printing summary for tiers and others induced by [1]. [1] https://gerrit.opnfv.org/gerrit/#/c/26587/ JIRA: FUNCTEST-438 Co-Authored-By: Cédric Ollivier <cedric.ollivier@orange.com> Change-Id: I821efaa7d62b208d797d19563f573d1e28da9df4 Signed-off-by: ashishk1994 <ashishk.iiit@gmail.com> Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-02-09Fix source_credentials in openstack_utilsCédric Ollivier1-2/+2
It fixes issues raised when an env var contains '=' (e.g. LS_COLORS). It now exports the vars listed in rc_file without creating a new process. Change-Id: I6e6a4e3ab9c45afe4d40015831acb283acbe564c Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-02-02Define how to pass args to TestcaseBase.runCédric Ollivier1-1/+5
Here is an exemple of functest/ci/testcases.yaml run: module: 'functest.opnfv_tests.sdn.odl.odl' class: 'ODLTests' args: suites: - /home/opnfv/repos/odl_test/csit/suites/integration/basic - /home/opnfv/repos/odl_test/csit/suites/openstack/neutron Change-Id: I37b0ffa18f626bd8bb52ed460ae38053bd12c8ee Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-01-25Refactor Rally to adopt OOhelenyao1-4/+0
JIRA: FUNCTEST-541 Change-Id: Ife04f58dc31ca9ba6316cc31577f81a6bf0e0405 Signed-off-by: helenyao <yaohelan@huawei.com>
2017-01-19Merge "write test results to a local file"Morgan Richomme1-1/+1
2017-01-17Merge "Refactoring of args and parser variable in ci/run_tests, prepare_env"Morgan Richomme1-24/+37
2017-01-17write test results to a local filewu.zhihui1-1/+1
Write test result to a file or push it to DB depends on the value format of test_db_url which is defined in config_functest.yaml. Meanwhile, test_db_url can be set by os envrion value "RESULT_STORE". If test_db_url is a url, e.g. http:// or https://, then push result to DB. If test_db_url is a file location, e.g. file:///, then write results to a file with json data. One result record, one line. JIRA: FUNCTEST-657 Change-Id: I579087cd2c24d61a79142b5d67003fb486b6c723 Signed-off-by: wu.zhihui <wu.zhihui1@zte.com.cn>
2017-01-12Bugfix: fix the wrong return codehelenyao1-3/+4
The bug was introduced by patch[1] [1] https://gerrit.opnfv.org/gerrit/#/c/24619/ Change-Id: Icf6c267aba26fcb5b129a611423dcf2200f95277 Signed-off-by: helenyao <yaohelan@huawei.com>
2017-01-09Refactoring of args and parser variable in ci/run_tests, prepare_envashishk19941-24/+37
Can't import run_tests in test_run_tests.py (new unit test file). It is throwing error: nosetests: error: unrecognized arguments It is because of declaration of parser and args which are global in file, So to allow safe import of this file, we need to declare it into __main__ branch. Change-Id: Ie11ebcfd8a1b8e692bd8bf4260a54f752b67fd5e Signed-off-by: ashishk1994 <ashishk.iiit@gmail.com>
2016-12-21Refactor test run related modules using new constants providerSerenaFeng1-20/+17
JIRA: FUNCTEST-673 Change-Id: Ia37fb5a08c8fbc4a8acb666565ea887f9ae48b83 Signed-off-by: SerenaFeng <feng.xiaowei@zte.com.cn>
2016-12-15Add check_criteria() in testcase_baseCédric Ollivier1-3/+4
It allows checking testcase results and then returning an error value to CI. Now run() should return EX_OK even if criteria is FAILED. It also fixes an issue induced by [1]. [1] https://gerrit.opnfv.org/gerrit/#/c/24561/ Change-Id: Iaa095696f9dcb87f8cc95179b4f404b092d1e23e Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2016-12-14Authenticate clients with keystoneauth1.sessionhelenyao1-0/+1
JIRA: FUNCTEST-529 1. use keystoneauth1.session to initialize the client for each service The keystoneauth1.session.Session class was introduced into keystoneauth1 as an attempt to bring a unified interface to the various OpenStack clients that share common authentication and request parameters between a variety of services. 2. update ODL case to leverage session to get the endpoint info Change-Id: Ic8c01b9b7ed86d3bdd9f5125504bc47f46a37700 Signed-off-by: helenyao <yaohelan@huawei.com>
2016-12-07abstract umbrella part to make integration code simplerSerenaFeng1-2/+2
implement an abstract class Umbrella, and take parser for instance JIRA: FUNCTEST-618 Change-Id: Iadf71ad49abd901b9659cb71135fc902a3b616d1 Signed-off-by: SerenaFeng <feng.xiaowei@zte.com.cn>
2016-12-02Rename TestCasesBase into TestcaseBasehelenyao1-3/+3
JIRA: FUNCTEST-628 1. rename the TestCasesBase into TestcaseBase 2. updated the module name to testcase_base.py Change-Id: Ic6b5d8209098c136613a2419c3790668fdbd493e Signed-off-by: helenyao <yaohelan@huawei.com>
2016-11-30Revert "solve failure testcases are not pushed to testDB problem"Morgan Richomme1-4/+3
This reverts commit a41264c46d583a68d48de041663213752dc62bf3. Change-Id: Id62c5342fffb5d75672127d9765d4cc882e3f4df Signed-off-by: Morgan Richomme <morgan.richomme@orange.com>