diff options
| -rw-r--r-- | docs/testing/developer/devguide/index.rst | 36 | ||||
| -rw-r--r-- | docs/testing/user/configguide/configguide.rst | 6 | ||||
| -rw-r--r-- | docs/testing/user/userguide/index.rst | 8 | ||||
| -rw-r--r-- | functest/ci/config_functest.yaml | 2 | ||||
| -rw-r--r-- | functest/energy/energy.py | 96 | ||||
| -rw-r--r-- | functest/tests/unit/energy/test_functest_energy.py | 48 |
6 files changed, 158 insertions, 38 deletions
diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst index 43f0804d7..551edec6f 100644 --- a/docs/testing/developer/devguide/index.rst +++ b/docs/testing/developer/devguide/index.rst @@ -81,7 +81,7 @@ The internal test cases in Danube are: By internal, we mean that this particular test cases have been developped and/or integrated by functest contributors and the associated code is hosted in the Functest repository. -An internal case can be fully developped or a simple integration of +An internal case can be fully developed or a simple integration of upstream suites (e.g. Tempest/Rally developped in OpenStack are just integrated in Functest). The structure of this repository is detailed in `[1]`_. @@ -123,7 +123,7 @@ The external test cases are: The code to run these test cases may be directly in the repository of the project. We have also a **features** sub directory under opnfv_tests -directory that may be used (it can be usefull if you want to reuse +directory that may be used (it can be useful if you want to reuse Functest library). Functest framework @@ -131,12 +131,12 @@ Functest framework Functest can be considered as a framework. Functest is release as a docker file, including tools, scripts and a CLI -to prepare the environement and run tests. +to prepare the environment and run tests. It simplifies the integration of external test suites in CI pipeline and provide commodity tools to collect and display results. Since Colorado, test categories also known as tiers have been created to -group similar tests, provide consistant sub-lists and at the end optimize +group similar tests, provide consistent sub-lists and at the end optimize test duration for CI (see How To section). The definition of the tiers has been agreed by the testing working group. @@ -212,7 +212,7 @@ functest/utils/ `-- openstack_utils.py Note that for Openstack, keystone v3 is now deployed by default by compass, -fuel and joid in Danube. All installers still support keysone v2 (deprecated in +fuel and joid in Danube. All installers still support keystone v2 (deprecated in next version). Test collection framework @@ -323,7 +323,7 @@ Please note that currently token authorization is implemented but is not yet ena =================== An automatic reporting page has been created in order to provide a - consistant view of the scenarios. + consistent view of the scenarios. In this page, each scenario is evaluated according to test criteria. The code for the automatic reporting is available at `[8]`_. @@ -368,7 +368,7 @@ Please note that currently token authorization is implemented but is not yet ena os-odl_l2-nofeature scenarios. If no result is available or if all the results are failed, the test case get 0 point. - If it was succesfull at least once but not anymore during the 4 runs, + If it was successful at least once but not anymore during the 4 runs, the case get 1 point (it worked once). If at least 3 of the last 4 runs were successful, the case get 2 points. If the last 4 runs of the test are successful, the test get 3 points. @@ -400,7 +400,7 @@ Please note that currently token authorization is implemented but is not yet ena Dashboard ========= -Dashboard is used to provide a consistant view of the results collected +Dashboard is used to provide a consistent view of the results collected in CI. The results showed on the dashboard are post processed from the Database, which only contains raw results. @@ -473,10 +473,10 @@ are identified but not covered yet by an existing testing project (e.g security_scan before the creation of the security repository) -How test constraints are defined? +How are test constraints defined? ================================= -Test constraints are defined according to 2 paramaters: +Test constraints are defined according to 2 parameters: * The scenario (DEPLOY_SCENARIO env variable) * The installer (INSTALLER_TYPE env variable) @@ -518,7 +518,7 @@ bgpvpn scenarios:: scenario: '(ocl)|(nosdn)|^(os-odl)((?!bgpvpn).)*$' -How to write and check constaint regex? +How to write and check constraint regex? ======================================= Regex are standard regex. You can have a look at `[11]`_ @@ -534,7 +534,7 @@ How to know which test I can run? You can use the API `[13]`_. The static declaration is in git `[5]`_ If you are in a Functest docker container (assuming that the -environement has been prepared): just use the CLI. +environment has been prepared): just use the CLI. You can get the list per Test cases or by Tier:: @@ -692,7 +692,7 @@ e.g.:: This command will run all the test cases of the first 2 tiers, i.e. healthcheck, connection_check, api_check, vping_ssh, vping_userdata, -snaps_somke, tempest_smoke_serial and rally_sanity. +snaps_smoke, tempest_smoke_serial and rally_sanity. How to push your results into the Test Database @@ -769,7 +769,7 @@ It can be described as follows:: Please note that each exclusion must be justified. the goal is not to exclude test cases because they do not pass. Several scenarios reached the 100% criteria. -So it is expected in the patch submited to exclude the cases to indicate the +So it is expected in the patch submitted to exclude the cases to indicate the reasons of the exclusion. @@ -790,7 +790,7 @@ I have tests, to which category should I declare them? CATEGORIES/TIERS description: +----------------+-------------------------------------------------------------+ -| healthcheck | Simple OpenStack healtcheck tests case that validates the | +| healthcheck | Simple OpenStack healthcheck tests case that validates the | | | basic operations in OpenStack | +----------------+-------------------------------------------------------------+ | Smoke | Set of smoke test cases/suites to validate the most common | @@ -800,7 +800,7 @@ CATEGORIES/TIERS description: | | Those come from Feature projects and need a bit of support | | | for integration | +----------------+-------------------------------------------------------------+ -| Components | Advanced Openstack tests: Full Tempest, Full Rally | +| Components | Advanced OpenStack tests: Full Tempest, Full Rally | +----------------+-------------------------------------------------------------+ | Performance | Out of Functest Scope | +----------------+-------------------------------------------------------------+ @@ -816,7 +816,7 @@ We recommend to declare your test in the feature category. VNF category is really dedicated to test including: * creation of resources - * deployement of an orchestrator/VNFM + * deployment of an orchestrator/VNFM * deployment of the VNF * test of the VNFM * free resources @@ -939,7 +939,7 @@ You can precise some configuration parameters in config_functest.yaml Create your own VnfOnboarding file -you must create your entry point through a python clase as referenced in the +you must create your entry point through a python class as referenced in the configuration file e.g. aaa => creation of the file <Functest repo>/functest/opnfv_tests/vnf/aaa/aaa.py diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst index 8a444e739..5c89cf0e6 100644 --- a/docs/testing/user/configguide/configguide.rst +++ b/docs/testing/user/configguide/configguide.rst @@ -283,7 +283,7 @@ to attach the 'Up' status Functest container and start bash mode:: docker exec -it <Functest_Container_Name> bash -4, Functest environemnt preparation and check +4, Functest environment preparation and check To see the Section below `Preparing the Functest environment`_. @@ -415,7 +415,7 @@ We may distinguish several directories, the first level has 4 directories: profile or any other test inputs that could be reused by any test project. * **docker**: This directory includes the needed files and tools to - build the Funtest Docker image. + build the Functest Docker image. * **docs**: This directory includes documentation: Release Notes, User Guide, Configuration Guide and Developer Guide. * **functest**: This directory contains all the code needed to run @@ -511,7 +511,7 @@ This script will make sure that the requirements to run the tests are met and will install the needed libraries and tools by all Functest test cases. It should be run only once every time the Functest docker container is started from scratch. If you try to run this command, on -an already prepared enviroment, you will be prompted whether you really +an already prepared environment, you will be prompted whether you really want to continue or not:: functest env prepare diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst index 0041d09b3..c877be7b5 100644 --- a/docs/testing/user/userguide/index.rst +++ b/docs/testing/user/userguide/index.rst @@ -198,7 +198,7 @@ updates the appropriate parameters into the configuration file. When the Tempest suite is executed, each test duration is measured and the full console output is stored to a *log* file for further analysis. -The Tempest testcases are distributed accross two +The Tempest testcases are distributed across two Tiers: * Smoke Tier - Test Case 'tempest_smoke_serial' @@ -239,7 +239,7 @@ The OPNFV Rally scenarios are based on the collection of the actual Rally scenar A basic SLA (stop test on errors) has been implemented. -The Rally testcases are distributed accross two Tiers: +The Rally testcases are distributed across two Tiers: * Smoke Tier - Test Case 'rally_sanity' * Components Tier - Test case 'rally_full' @@ -414,11 +414,11 @@ The list of tests can be described as follows: * Delete operations * Delete the port previously created via OpenStack - * Check that the port has been also succesfully deleted in OpenDaylight + * Check that the port has been also successfully deleted in OpenDaylight * Delete previously subnet created via OpenStack * Check that the subnet has also been successfully deleted in OpenDaylight * Delete the network created via OpenStack - * Check that the network has also been succesfully deleted in OpenDaylight + * Check that the network has also been successfully deleted in OpenDaylight Note: the checks in OpenDaylight are based on the returned HTTP status code returned by OpenDaylight. diff --git a/functest/ci/config_functest.yaml b/functest/ci/config_functest.yaml index b93730cdf..1807082d9 100644 --- a/functest/ci/config_functest.yaml +++ b/functest/ci/config_functest.yaml @@ -200,6 +200,6 @@ results: test_db_url: http://testresults.opnfv.org/test/api/v1/results energy_recorder: - api_url: http://161.105.253.100:8888/resources + api_url: http://opnfv.fr:8888/resources api_user: "" api_password: "" diff --git a/functest/energy/energy.py b/functest/energy/energy.py index a20c79926..71b712393 100644 --- a/functest/energy/energy.py +++ b/functest/energy/energy.py @@ -20,6 +20,8 @@ import functest.utils.functest_utils as ft_utils def enable_recording(method): """ + Record energy during method execution. + Decorator to record energy during "method" exection. param method: Method to suround with start and stop @@ -29,10 +31,21 @@ def enable_recording(method): attribute """ def wrapper(*args): - """Wrapper for decorator to handle method arguments.""" + """ + Record energy during method execution (implementation). + + Wrapper for decorator to handle method arguments. + """ + current_scenario = EnergyRecorder.get_current_scenario() EnergyRecorder.start(args[0].case_name) return_value = method(*args) - EnergyRecorder.stop() + if current_scenario is None: + EnergyRecorder.stop() + else: + EnergyRecorder.submit_scenario( + current_scenario["scenario"], + current_scenario["step"] + ) return return_value return wrapper @@ -47,7 +60,7 @@ class EnergyRecorder(object): energy_recorder_api = None # Default initial step - INITIAL_STEP = "starting" + INITIAL_STEP = "running" @staticmethod def load_config(): @@ -92,22 +105,24 @@ class EnergyRecorder(object): } @staticmethod - def start(scenario): + def submit_scenario(scenario, step): """ - Start a recording session for scenario. + Submit a complet scenario definition to Energy recorder API. - param scenario: Starting scenario + param scenario: Scenario name :type scenario: string + param step: Step name + :type step: string """ return_status = True try: - EnergyRecorder.logger.debug("Starting recording") + EnergyRecorder.logger.debug("Submitting scenario") # Ensure that connectyvity settings are loaded EnergyRecorder.load_config() # Create API payload payload = { - "step": EnergyRecorder.INITIAL_STEP, + "step": step, "scenario": scenario } # Call API to start energy recording @@ -120,10 +135,34 @@ class EnergyRecorder(object): } ) if response.status_code != 200: - log_msg = "Error while starting energy recording session\n{}" + log_msg = "Error while submitting scenario\n{}" log_msg = log_msg.format(response.text) EnergyRecorder.logger.info(log_msg) return_status = False + except Exception: # pylint: disable=broad-except + # Default exception handler to ensure that method + # is safe for caller + EnergyRecorder.logger.exception( + "Error while submitting scenarion to energy recorder API" + ) + return_status = False + return return_status + + @staticmethod + def start(scenario): + """ + Start a recording session for scenario. + + param scenario: Starting scenario + :type scenario: string + """ + return_status = True + try: + EnergyRecorder.logger.debug("Starting recording") + return_status = EnergyRecorder.submit_scenario( + scenario, + EnergyRecorder.INITIAL_STEP + ) except Exception: # pylint: disable=broad-except # Default exception handler to ensure that method @@ -201,3 +240,42 @@ class EnergyRecorder(object): ) return_status = False return return_status + + @staticmethod + def get_current_scenario(): + """Get current running scenario (if any, None else).""" + EnergyRecorder.logger.debug("Getting current scenario") + return_value = None + print "In get current" + try: + # Ensure that connectyvity settings are loaded + EnergyRecorder.load_config() + + # Call API get running scenario + response = requests.get( + EnergyRecorder.energy_recorder_api["uri"], + auth=EnergyRecorder.energy_recorder_api["auth"] + ) + if response.status_code == 200: + return_value = json.loads(response.text) + elif response.status_code == 404: + log_msg = "No current running scenario at {}" + log_msg = log_msg.format( + EnergyRecorder.energy_recorder_api["uri"]) + EnergyRecorder.logger.error(log_msg) + print log_msg + return_value = None + else: + log_msg = "Error while getting current scenario\n{}" + log_msg = log_msg.format(response.text) + EnergyRecorder.logger.error(log_msg) + print log_msg + return_value = None + except Exception: # pylint: disable=broad-except + # Default exception handler to ensure that method + # is safe for caller + EnergyRecorder.logger.exception( + "Error while getting current scenario from energy recorder API" + ) + return_value = None + return return_value diff --git a/functest/tests/unit/energy/test_functest_energy.py b/functest/tests/unit/energy/test_functest_energy.py index 6387b97ba..177788bc4 100644 --- a/functest/tests/unit/energy/test_functest_energy.py +++ b/functest/tests/unit/energy/test_functest_energy.py @@ -1,5 +1,8 @@ #!/usr/bin/env python +# -*- coding: UTF-8 -*- +# Copyright (c) 2017 Orange and others. +# # All rights reserved. This program and the accompanying materials # are made available under the terms of the Apache License, Version 2.0 # which accompanies this distribution, and is available at @@ -16,8 +19,11 @@ from functest.energy.energy import EnergyRecorder import functest.energy.energy as energy -CASE_NAME = "UNIT_test_CASE" -STEP_NAME = "UNIT_test_STEP" +CASE_NAME = "UNIT_TEST_CASE" +STEP_NAME = "UNIT_TEST_STEP" + +PREVIOUS_SCENARIO = "previous_scenario" +PREVIOUS_STEP = "previous_step" class MockHttpResponse(object): # pylint: disable=too-few-public-methods @@ -197,6 +203,8 @@ class EnergyRecorderTest(unittest.TestCase): """Call with to energy recorder decorators.""" raise Exception(self.exception_message_to_preserve) + @mock.patch("functest.energy.energy.EnergyRecorder.get_current_scenario", + return_value=None) @mock.patch("functest.energy.energy.EnergyRecorder") @mock.patch("functest.utils.functest_utils.get_pod_name", return_value="MOCK_POD") @@ -205,13 +213,34 @@ class EnergyRecorderTest(unittest.TestCase): def test_decorators(self, loader_mock=None, pod_mock=None, - recorder_mock=None): + recorder_mock=None, + cur_scenario_mock=None): """Test energy module decorators.""" self.__decorated_method() calls = [mock.call.start(self.case_name), mock.call.stop()] recorder_mock.assert_has_calls(calls) + @mock.patch("functest.energy.energy.EnergyRecorder.get_current_scenario", + return_value={"scenario": PREVIOUS_SCENARIO, + "step": PREVIOUS_STEP}) + @mock.patch("functest.energy.energy.EnergyRecorder") + @mock.patch("functest.utils.functest_utils.get_pod_name", + return_value="MOCK_POD") + @mock.patch("functest.utils.functest_utils.get_functest_config", + side_effect=config_loader_mock) + def test_decorators_with_previous(self, + loader_mock=None, + pod_mock=None, + recorder_mock=None, + cur_scenario_mock=None): + """Test energy module decorators.""" + self.__decorated_method() + calls = [mock.call.start(self.case_name), + mock.call.submit_scenario(PREVIOUS_SCENARIO, + PREVIOUS_STEP)] + recorder_mock.assert_has_calls(calls) + def test_decorator_preserve_return(self): """Test that decorator preserve method returned value.""" self.test_load_config() @@ -270,6 +299,19 @@ class EnergyRecorderTest(unittest.TestCase): EnergyRecorder.load_config() self.assertEquals(EnergyRecorder.energy_recorder_api, None) + @mock.patch("functest.utils.functest_utils.get_functest_config", + return_value=None) + @mock.patch("functest.utils.functest_utils.get_pod_name", + return_value="MOCK_POD") + @mock.patch('functest.energy.energy.requests.get', + return_value=RECORDER_OK) + def test_get_current_scenario(self, loader_mock=None, + pod_mock=None, get_mock=None): + """Test get_current_scenario.""" + self.test_load_config() + scenario = EnergyRecorder.get_current_scenario() + self.assertTrue(scenario is not None) + if __name__ == "__main__": logging.disable(logging.CRITICAL) |