aboutsummaryrefslogtreecommitdiffstats
path: root/functest/ci/run_tests.py
AgeCommit message (Collapse)AuthorFilesLines
2018-02-28Rename all Functest refs to XtestingCédric Ollivier1-302/+0
It mainly renames python modules and then the related documentation config files. Change-Id: I186010bb88d3d39afe7b8fd1ebcef9c690cc1282 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-02-24Print env vars and env_fileCédric Ollivier1-11/+4
It simply eases detecting wrong inputs [1]. [1] https://build.opnfv.org/ci/view/functest/job/functest-apex-baremetal-daily-master/796/console Change-Id: I0784c29265c69a6ba4a049226196bcdd70b0a572 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-02-22Use constants for Functest config fileCédric Ollivier1-2/+2
env_file can't be defined in a config file simply because it's an entry point (Jenkins jobs, end users) Change-Id: Ie23c0ef90efc839d60f0f3a9754c58746b1f3a00 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-02-16Get properly env vars or their default valuesCédric Ollivier1-8/+9
It defines env.get() as an unique way to get Functest env vars or their default values. It can be considered as a wrapper above os.environ. It enforces backward compatibility via CONST which mustn't be used for that purpose. It should be noted that it also stops using CONST for getting OpenStack env vars. Change-Id: I333dc1afbc0123166a7eaff8b551370098efa341 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-02-10Run all daily testcases if CI_LOOP is unsetCédric Ollivier1-2/+2
Then CI_LOOP is no longer mandatory and it bypasses the current jenkins jobs's issue [1][2][3]. [1] https://gerrit.opnfv.org/gerrit/#/c/51981/ [2] https://build.opnfv.org/ci/view/functest/job/functest-apex-baremetal-daily-master/763/console [3] https://build.opnfv.org/ci/view/functest/job/functest-apex-baremetal-daily-master/764/console Change-Id: I73629079d70e0f43e609d3861490573e10fd216c Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-02-09Read the env file only if existsCédric Ollivier1-0/+4
Change-Id: I8e718e583a5a47a089755c31e623a44732e5ba1e Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-02-09Remove one obsolete varCédric Ollivier1-5/+0
Else it simply forces the config file into xtesting. Change-Id: I33153997f2aa422e50cea011f6a4e7eb567a83d7 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-02-09Unlink run_tests from constantsCédric Ollivier1-11/+11
run_tests.py doesn't read any functest config file what is mandatory for xtesting. Change-Id: I49f672c54a606acead75f573d1ce496c81fb235a Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-02-09Move get_dict_by_test() into run_tests.pyCédric Ollivier1-3/+16
It also removes functest_utils.get_criteria_by_test() Change-Id: I3f265642acd053755e32f8e92f1086b93517c247 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-02-08Move source_credentials() into run_tests.pyCédric Ollivier1-2/+17
It's also renamed source_envfile(). Change-Id: I0e7c38c2def125961f86dc2bc9a63cfb6ad87c03 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-02-01Rename a common credential file for OS and K8SLinda Wang1-11/+3
Change-Id: Ib8d24be4b29ab1de00d5dd5e3442146b2437cb94 Signed-off-by: Linda Wang <wangwulin@huawei.com>
2018-01-21Partially rewrite test_run_tests.pyCédric Ollivier1-13/+13
It mainly fixes pylint issues and leverages on decorators. It should be noted that run_tests.py is not fully covered (see xtesting requirements). Change-Id: I114b67c3c5bfe61d72d004829af513e2014ad8b8 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-12-19Delete TestCase.create_snapshot()Cédric Ollivier1-3/+0
OSGC has been removed then this method can be safely removed. It also updates run_tests.py. Change-Id: I4030b7f0351d24d095313ae990b08e65d8f69115 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-12-18Merge "Inject all envs in openrc into CONST"Cedric Ollivier1-13/+0
2017-12-15Inject all envs in openrc into CONSTLinda Wang1-13/+0
After "env_prepare" is removed, some specific vars in openrc (eg,."EXTERNAL_NETWORK") loaded to os.environ should be injected into CONST in one python process. Change-Id: Iaa5c6047033dc2a30c533174e2016111759aec31 Signed-off-by: Linda Wang <wangwulin@huawei.com>
2017-12-14Improve the pylint score of functest-coreLinda Wang1-41/+67
It also modifies the file list to ensure the rating of ci modules. Change-Id: Ia1e414be5364cb3da3d54882db428024ed6bd99f Signed-off-by: Linda Wang <wangwulin@huawei.com>
2017-12-12All patching config files in memoryLinda Wang1-35/+0
It adds the yaml content as Config attribute and allow patching it without modifying the file (and reimport constants or config). It also removes obsolete data in config_patch.yaml and moves several functions in the right modules (Config) JIRA: ARMBAND-334 Co-Authored-By: Cédric Ollivier <cedric.ollivier@orange.com> Change-Id: I5c9a574d3283828063154c977cdfbc2abfdc6777 Signed-off-by: Linda Wang <wangwulin@huawei.com> Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-12-11Add Python3 support in functest/ciCédric Ollivier1-1/+2
Change-Id: If13bfaa196b8dfab1e83feb28c5a4719c7acd8c7 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-11-30Stop updating test_db_url in config file by env varLinda Wang1-13/+0
If TEST_DB_URL is declared in env variable, use it before considering the var in config file. Change-Id: I656ad8765287c48d60af7ab88b421805d76fb231 Signed-off-by: Linda Wang <wangwulin@huawei.com>
2017-11-22Remove prepare_envJose Lausuch1-0/+51
After moving the rally installation out of prepare_env It doesn't much sense to keep this script as it doesn't do useful things any more. Change-Id: I9ab3b2dd30c8ec0fbb825ee4302a83cce80f1cbe Signed-off-by: Jose Lausuch <jalausuch@suse.com>
2017-09-18Return Functest status to JenkinsCédric Ollivier1-9/+13
It avoids mixing Functest and Features CI issues by simply exiting the status of Functest testcases. The result of third-party testcases are printed into console but are no longer taken into account into the global status. Change-Id: I040ff54780db2ddb6d81993839fee09f5e472323 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-09-13Disable urllib3 warningsLinda Wang1-0/+1
1. When running in compass OSA, lots of SubjectAltNameWarnings are shown in console, so just disable the warnings here. See https://github.com/shazow/urllib3/issues/497 for details 2. Redirect warnings issued by the warnings module to the logging system via logging.captureWarnings(True) 3. But urllib3 warning is still there for two cmds about rally/tempest. Change-Id: Ic22cf8e2babc10c9d55a7ed46b841312a4e430a7 Signed-off-by: Linda Wang <wangwulin@huawei.com>
2017-08-25Fix args parsing in run_testsLinda Wang1-2/+2
By default, 'report' and 'noclean' are both False, so self.clean_flag shoule be True and self.report_flag should be False. Change-Id: I4935f70bd124fbb996dcaa73e01e0e61cd0184e1 Signed-off-by: Linda Wang <wangwulin@huawei.com>
2017-08-24Prevent KeyError during main args parsingMartin Kulhavy1-3/+3
Change-Id: Ia340c79c82b891e921379c7c0186eaa238117b13 Signed-off-by: Martin Kulhavy <martin.kulhavy@nokia.com>
2017-08-18Merge "Fix summary in run_tests.py"Cedric Ollivier1-6/+12
2017-08-18Fix summary in run_tests.pyCédric Ollivier1-6/+12
It fixes summary() when several tests have not been executed after a blocking test in failure. Change-Id: I3c0af4c8083ffa76f786756d409c09e018bbecc8 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-08-18Put domain name in CONSTAlex Yang1-0/+2
Domain name is not obtained in source_rc_file() but used in the patch https://gerrit.opnfv.org/gerrit/#/c/39109/. Change-Id: Iceaf352f7ad5e15f15906f0e3aa1c5ab5ae30a42 Signed-off-by: Alex Yang <yangyang1@zte.com.cn>
2017-08-16Clean run_tests.py and the related utCédric Ollivier1-78/+62
It enhances run_tests.py as proposed in [1]. It also prints all skipped tests in summary and fixes copyright headers. All the related unit tests have been updated too. [1] https://jira.opnfv.org/browse/FUNCTEST-832 JIRA: FUNCTEST-832 Change-Id: I59b96422bc7942ecd6270c45ab7a3fb603c13ccb Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-07-02Apply restrictive file permissionsCédric Ollivier1-3/+0
It conforms with [1] by creating a new venv which checks the unix permissions. As jjobs call Functest console scripts [2], all perms can be 644. Dockerfiles are updated as well. [1] https://security.openstack.org/guidelines/dg_apply-restrictive-file-permissions.html [2] https://gerrit.opnfv.org/gerrit/#/c/36805/ Depends-On: I9209e6efa1b493e24135402a46df72aaa14115d1 Change-Id: I31bc7f12b775928845e23b6b40288b0a50b87219 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-06-30Define console_scriptsCédric Ollivier1-2/+5
Only prepare_env.py and run_tests.py remain executable to avoid breaking the gate. As soon as the jenkins jobs are updated to run them from $PATH, tox will be updated to protect against chmod a+x py files. Dockerfile will be simplified as well. Change-Id: Iac1a7ceb6b344e800177226ea39919826f6d3735 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-06-26Don't display functest report for single test casesjose.lausuch1-10/+12
When executing a single test case, it is not neccessary to show duplicated information about the result of the test case. The functest report only makes sense when running multiple tests. e.g. functest testcase run vping_ssh The report is still showed when doing: functest testcase run all Change-Id: I92d97d6111ecd9484a14c72cf4bb0c2bbe3ae6bf Signed-off-by: jose.lausuch <jose.lausuch@ericsson.com>
2017-06-20Remove all references to /home/opnfv/repos/functestCédric Ollivier1-3/+4
It also removes relative (and incomplete) references which could be defined in modules instead of this global configuration. Change-Id: Ie20dc3547e49b6224aa8100cd380d37b87ece5a9 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-30Merge "Define create_snapshot() and clean() in TestCase"Cedric Ollivier1-27/+23
2017-05-30Define create_snapshot() and clean() in TestCaseCédric Ollivier1-27/+23
They replace the former calls in run_tests which are not suitable for all test cases. Now any test case can define how to clean its resources. If the snapshot cannot be created, the test case is considered as failed. Only a message is printed if any failure during cleaning. It also defines a new class called OSGCTestCase useful for test cases which don't clean their OpenStack resources. All test cases located in opnfv_tests/openstack inherit from it to keep the global behavior unchanged. It also deletes exit instructions in openstack_clean and openstack_snapshot, removes clean flags in testcases.yaml and updates the related utils. All Docs are modified as well. JIRA: FUNCTEST-438 Change-Id: I8938e6255708012380389763a24059ace4ce45d8 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-28Switch shebangs to /usr/bin/env pythonCédric Ollivier1-1/+1
It modified the shebang to /usr/bin/env which is considered as more flexible. Change-Id: I9c8cdb4243d41a83d524499cc91060d3700e008b Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-23Bugfix: fix two typosLinda Wang1-1/+1
1. In run_tests.py, test case name is shown before running: logger.error(kwargs['test']) -> logger.info(kwargs['test']) 2. In tempest.py, Verication -> Verification Change-Id: I117838cdb4b0f920c4cf855645d910e623e06868 Signed-off-by: Linda Wang <wangwulin@huawei.com>
2017-05-19Print stack when catching exceptions in run_tests.pyCédric Ollivier1-7/+8
Change-Id: Id94475dd519ce04b87da470f6e6bb5a1c1aa2d75 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-16Define Runner classCédric Ollivier1-202/+202
It simply converts run_tests.py's functions to Runner's methods. This proposal could be enhanced in a second step to fully leverage on object programming. It defines as instance variables the former static variables to allow multiple runs (it's mandatory for unit testing). All false positives in unit tests are fixed but run_tests.py is still not fully covered. Change-Id: I8f91c132aa8b2248041f31e46238dd5598344d34 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-16Add enabled flag in testcases.yamljose.lausuch1-0/+7
This will allow enable/disable test cases directly using true/false. Change-Id: Iaa7d051ebb8dd9e5a6f835a823f58f1e17a87b8d Signed-off-by: jose.lausuch <jose.lausuch@ericsson.com>
2017-05-16Switch from generate_report to PrettyTableCédric Ollivier1-26/+24
run_tests.py now relies on PrettyTable as most of the openstack clients. generate_report.py and its related unit tests are simply removed. It sets padding_width=5 in testcase.py too to conform with run_tests.py. Now report is printed in every case. Change-Id: Id9ce93f984503f25d6a2150482f397853fa3dd64 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-16Fix how to get tier_nameCédric Ollivier1-1/+1
The tier name was incorrect when passing a test case as run_test.py arg. Change-Id: I6f439fc7b90facb3f0553798bd1fe06ac9f09475 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-06Implement TestCase __str__()Cédric Ollivier1-2/+1
It's designed to be called by run_tests.py when printing the results of test cases and when generating the global report. Change-Id: If4a6f023ef2344bbc4f940d07dde4b776dce5d68 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-05Merge "Print the real testcase duration"Jose Lausuch1-11/+7
2017-05-05Print the real testcase durationCédric Ollivier1-11/+7
TestCase offers a new public method to calculate and print the duration. It also adds the related unit tests and adapts run_tests.py. Change-Id: Ib6a7e637dafacb9027146199aeb033c2dcb986c6 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-05Replace CONST.* by getattribute/setattrjose.lausuch1-11/+12
Directories affected: - ci - core and respective unit tests Change-Id: I6a3d5aa68de29fc5a37ae543a067ff797eba33e6 Signed-off-by: jose.lausuch <jose.lausuch@ericsson.com>
2017-05-04Delete functest.utils.functest_loggerCédric Ollivier1-4/+6
It deletes functest.utils.functest_logger and the related unit tests. Then it modifies all functest modules to get all loggers via logging.getLogger(). __name__ is mainly used as getLogger arg as proposed by logging [1]. All loggers and handlers are now defined via functest/ci/logging.ini instead of a dict loaded by an external json file. Now only warn messages and info messages from ci and core packages are printed in console. [1] https://docs.python.org/2/library/logging.html Change-Id: Ic192855e0f9bf94825d8f7ec73549a0f3b8d44c5 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-04-26Switch TestCase attribute criteria to resultCédric Ollivier1-1/+1
It mainly avoids mixing input and output. Criteria is now an input set in functest/ci/testcases.yaml and then must be passed as __init__() args (which will be proposed in an additional change). Then it also renames the related TestCase method to check_result(). Change-Id: Ifc3c8e3ea6cde7e3edf7174bed4bf2bf0894e8e3 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-04-11Modify TestCase constructor attributesCédric Ollivier1-1/+2
Every feature/testcase now allows receiving the data defined in testcases.yaml as args (name is renamed to case_name). From the time being, only project and case names are handled. Next pending patches will add criteria, cmd and repo to this list. It keeps the default values for case names except for features which will be aggregated into Feature. Change-Id: Id742d100b8183d7f10894c24ae6879d1b2b60ac9 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-04-04Add case_name as constructor argCédric Ollivier1-1/+1
It allows managing multiple TestCase names with only one TestCase module. It is mainly required by odl which implements: - odl, - odl_netvirt, - fds. It also renames case to case_name in Features to conform with TestCases. JIRA: FUNCTEST-762 Change-Id: Ie254f754a0ea3077a8afda1c470528d38c79478f Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-03-27Rename testcase_base to testcaseCédric Ollivier1-3/+3
It simply removes the useless suffix. Change-Id: Ib9a682b57d8f33518359d817a593bb3786577390 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>