aboutsummaryrefslogtreecommitdiffstats
path: root/functest/core/testcase.py
AgeCommit message (Collapse)AuthorFilesLines
2017-05-30Define create_snapshot() and clean() in TestCaseCédric Ollivier1-0/+50
They replace the former calls in run_tests which are not suitable for all test cases. Now any test case can define how to clean its resources. If the snapshot cannot be created, the test case is considered as failed. Only a message is printed if any failure during cleaning. It also defines a new class called OSGCTestCase useful for test cases which don't clean their OpenStack resources. All test cases located in opnfv_tests/openstack inherit from it to keep the global behavior unchanged. It also deletes exit instructions in openstack_clean and openstack_snapshot, removes clean flags in testcases.yaml and updates the related utils. All Docs are modified as well. JIRA: FUNCTEST-438 Change-Id: I8938e6255708012380389763a24059ace4ce45d8 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-17Fix an unorderable types error (if python 3)Cédric Ollivier1-0/+1
Change-Id: I07c5a548f10a1779faead4d9c596a1976aff8866 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-16Switch from generate_report to PrettyTableCédric Ollivier1-1/+1
run_tests.py now relies on PrettyTable as most of the openstack clients. generate_report.py and its related unit tests are simply removed. It sets padding_width=5 in testcase.py too to conform with run_tests.py. Now report is printed in every case. Change-Id: Id9ce93f984503f25d6a2150482f397853fa3dd64 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-11Modify TestCase.__str__() to use PrettyTableCédric Ollivier1-6/+10
It adds PrettyTable as requirement even if it's already defined in OpenStack client dependencies. If the TestCase object is considered as invalid, it simply returns the default str. Change-Id: Iee788aef2a13694d9482560977cbbf21c7f2c967 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-10Merge "Conform ODL with last framework updates"Morgan Richomme1-4/+5
2017-05-09Conform ODL with last framework updatesCédric Ollivier1-4/+5
Now ODL TestCase calculates the ratio between critical tests passed and failed and saves it in result. Non-critical test cases can fail as result doesn't take them into account [1]. It also updates default attribute values in TestCase and allows result to be float. [1] http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#setting-criticality Change-Id: Id4a89271b5b1a90cd3c1e2b08591ff26ffaffee0 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-06Implement TestCase __str__()Cédric Ollivier1-1/+16
It's designed to be called by run_tests.py when printing the results of test cases and when generating the global report. Change-Id: If4a6f023ef2344bbc4f940d07dde4b776dce5d68 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-05Rename TestCase check_result() to is_successful()Cédric Ollivier1-2/+2
Change-Id: I41cea33b4c1dae79e75231e7c7e069fc89736f34 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-05Print the real testcase durationCédric Ollivier1-0/+18
TestCase offers a new public method to calculate and print the duration. It also adds the related unit tests and adapts run_tests.py. Change-Id: Ib6a7e637dafacb9027146199aeb033c2dcb986c6 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-04Define loggers as class-private membersCédric Ollivier1-7/+8
This mangling ensures that all info messages printed from core packages are shown in console. It also avoids sphinx to print them. Change-Id: I07db9f33060c195bce3b48b06a6640eb6c56c2eb Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-05-04Delete functest.utils.functest_loggerCédric Ollivier1-2/+2
It deletes functest.utils.functest_logger and the related unit tests. Then it modifies all functest modules to get all loggers via logging.getLogger(). __name__ is mainly used as getLogger arg as proposed by logging [1]. All loggers and handlers are now defined via functest/ci/logging.ini instead of a dict loaded by an external json file. Now only warn messages and info messages from ci and core packages are printed in console. [1] https://docs.python.org/2/library/logging.html Change-Id: Ic192855e0f9bf94825d8f7ec73549a0f3b8d44c5 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-04-26Manage criteria in TestCaseCédric Ollivier1-5/+17
It converts all criteria values to the corresponding percent in functest/ci/testcases.yaml. Result is expected to be equal or greater than criteria. If both are 0, result is considered as false. It is compatible with the old behavior but warns to update. It will allow a safer remove. It also fixes a bug in test_tempest to allow merging [1] and tier_handler.py which required that type criteria was str. [1] https://gerrit.opnfv.org/gerrit/#/c/27949/ Change-Id: Ib6edcfa3103b7d51b0bdc83119f1cea2a8be9fbc Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-04-26Switch TestCase attribute criteria to resultCédric Ollivier1-12/+12
It mainly avoids mixing input and output. Criteria is now an input set in functest/ci/testcases.yaml and then must be passed as __init__() args (which will be proposed in an additional change). Then it also renames the related TestCase method to check_result(). Change-Id: Ifc3c8e3ea6cde7e3edf7174bed4bf2bf0894e8e3 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-04-21Switch testcase to test case in docstringsCédric Ollivier1-4/+4
Change-Id: Iea64673788a3d5ab5eb3028522f659de183440ed Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-04-20Minor changes in docstringsCédric Ollivier1-11/+12
It removes case_name from the list of attributes that run() must set because it's now managed by run_tests.py (see __init__()) Change-Id: Id6228880257d0e9fb27483c56aa9985197feb04e Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-04-18Add docstings in feature.pyCédric Ollivier1-1/+1
It also modifies the testcase module docstring. Now features.py is rated 10/10 by pylint. Change-Id: I83f2ac385b2a713d116c1ae4b49ba9cc9b26a83c Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-04-18Fix Sphinx warnings/errorsCédric Ollivier1-1/+3
It takes into account the following warnings/errors: - Inline strong start-string without end-string - Unexpected indentation Change-Id: Idb05c7b04ba98ef8f3ef65e6ec7c9bb5b7571cad Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-04-14Add class attribute docstrings in testcase.pyCédric Ollivier1-0/+7
Change-Id: I91a5c6fb14f7c76a348bb4329d023bab53a3804c Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-04-11Modify TestCase constructor attributesCédric Ollivier1-3/+3
Every feature/testcase now allows receiving the data defined in testcases.yaml as args (name is renamed to case_name). From the time being, only project and case names are handled. Next pending patches will add criteria, cmd and repo to this list. It keeps the default values for case names except for features which will be aggregated into Feature. Change-Id: Id742d100b8183d7f10894c24ae6879d1b2b60ac9 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-04-04Add case_name as constructor argCédric Ollivier1-2/+3
It allows managing multiple TestCase names with only one TestCase module. It is mainly required by odl which implements: - odl, - odl_netvirt, - fds. It also renames case to case_name in Features to conform with TestCases. JIRA: FUNCTEST-762 Change-Id: Ie254f754a0ea3077a8afda1c470528d38c79478f Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-03-27Add docstrings in testcase.pyCédric Ollivier1-0/+51
testcase.py is now rated at 10.00/10 by pylint. Change-Id: Ia90e53db7a25aac3827cd29bb9fd6baaa46cd487 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2017-03-27Rename testcase_base to testcaseCédric Ollivier1-0/+64
It simply removes the useless suffix. Change-Id: Ib9a682b57d8f33518359d817a593bb3786577390 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>