diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/com/pres/framework/framework.html | 52 | ||||
| -rw-r--r-- | docs/com/pres/framework/framework.md | 266 | ||||
| -rw-r--r-- | docs/testing/user/configguide/configguide.rst | 1 | ||||
| -rw-r--r-- | docs/testing/user/userguide/index.rst | 120 |
4 files changed, 437 insertions, 2 deletions
diff --git a/docs/com/pres/framework/framework.html b/docs/com/pres/framework/framework.html new file mode 100644 index 000000000..950c2bebf --- /dev/null +++ b/docs/com/pres/framework/framework.html @@ -0,0 +1,52 @@ +<html> +<head> +<title>OPNFV Functest Framework</title> +<meta name="author" content="Cédric Ollivier"> +<meta name="viewport" + content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"> +<link rel="stylesheet" href="../reveal.js/css/reveal.css"> +<link rel="stylesheet" href="../reveal.js/css/theme/white.css"> +<link rel="stylesheet" href="../reveal.js/lib/css/zenburn.css"> +<script> +var link = document.createElement( 'link' ); +link.rel = 'stylesheet'; +link.type = 'text/css'; +link.href = window.location.search.match( /print-pdf/gi ) ? '../reveal.js/css/print/pdf.css' : '../reveal.js/css/print/paper.css'; +document.getElementsByTagName( 'head' )[0].appendChild( link ); +</script> +</head> +<body> + <div class="reveal"> + <div class="slides"> + <section data-markdown="framework.md" data-separator="^\n\n\n" + data-separator-vertical="^\n\n" data-separator-notes="^Note:"></section> + </div> + </div> + <script src="../reveal.js/lib/js/head.min.js"></script> + <script src="../reveal.js/js/reveal.js"></script> + <script> + Reveal.initialize({ + dependencies : [ { + src : '../reveal.js/plugin/markdown/marked.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src : '../reveal.js/plugin/markdown/markdown.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src: '../reveal.js/plugin/highlight/highlight.js', + async: true, + callback: function() { + hljs.initHighlightingOnLoad(); + } + }, { + src: '../reveal.js/plugin/notes/notes.js', + async: true + } ] + }); + </script> +</body> +</html> diff --git a/docs/com/pres/framework/framework.md b/docs/com/pres/framework/framework.md new file mode 100644 index 000000000..b80ad3dd7 --- /dev/null +++ b/docs/com/pres/framework/framework.md @@ -0,0 +1,266 @@ +# Functest Framework + +created by [Cédric Ollivier](mailto:cedric.ollivier@orange.com) + +2017/04/24 + +Note: + +- Functest integrates lots of heteregeounous testcases: + - python vs bash + - internal vs external +- it aims to benefit from object programming + - to define common operations + - to avoid conditional instructions regarding the testcases + - to avoid duplicating code + - to ease the integration of third-party testcases (written in Bash or Python) + + + +## Quick overview + + +### Functest function calls + +- **CI** calls *run_tests.py* (please see [jenkins jobs](https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=tree;f=jjb/functest)) +- *run_tests.py* parses *functest/ci/testcases.yaml* to: + - check which testcase(s) must be run + - execute the common operations on every testcase (run, push its results to db...) +<!-- .element: class="fragment highlight-red"--> + - return the right status code to **CI** + + +### Our target + +- limit run_tests.py instructions by defining: + - the basic testcase attritutes + - all common operations + - the status codes expected +- avoid duplicating codes between testcases +- ease the developpement of third-party testcases (aka features) + + + +## class TestCase + +base model for single test case + + +### instance attributes + +- project_name (default: 'functest') +- case_name +- criteria +- start_time +- stop_time +- details + + +### methods + +| Method | Purpose | +|-------------------|--------------------------------------------| +| run(**kwargs) | run the test case | +| check_criteria() | interpret the results of the test case | +| push_to_db() | push the results of the test case to the DB| + + +### run(**kwargs) + +- the subclasses must override the default implementation which is false on purpose +- the new implementation must set the following attributes to push the results to DB: + - criteria + - start_time + - stop_time + + +### class attributes + +| Status code | Returned when | +|--------------------|---------------------| +| EX_OK | everything is OK | +| EX_RUN_ERROR | run() failed | +| EX_TESTCASE_FAILED | results are false | +| EX_PUSH_TO_DB_ERROR| push_to_db() failed | + + +### run_tests.py + +```python +module = importlib.import_module(run_dict['module']) +cls = getattr(module, run_dict['class']) +test_dict = ft_utils.get_dict_by_test(test_name) +test_case = cls(**test_dict) +try: + kwargs = run_dict['args'] + result = test_case.run(**kwargs) +except KeyError: + result = test_case.run() +if result == testcase.TestCase.EX_OK: + if GlobalVariables.REPORT_FLAG: + test_case.push_to_db() + result = test_case.check_criteria() +``` + + + +## Your first test case + + +### first.py + +```python +#!/usr/bin/env python + +import time + +from functest.core import testcase + +class Test(testcase.TestCase): + + def run(self, **kwargs): + self.start_time = time.time() + print "Hello World" + self.criteria = 'PASS' + self.stop_time = time.time() + return testcase.TestCase.EX_OK +``` + + +### functest/ci/testcases.yaml + +```yaml +case_name: first +project_name: functest +criteria: 'status == "PASS"' +blocking: true +clean_flag: false +description: '' +dependencies: + installer: '' + scenario: '' +run: + module: 'first' + class: 'Test' +``` + + + +## class Feature +bases: TestCase + +base model for single feature + + +### methods + +| Method | Purpose | +|-------------------|---------------------------| +| run(**kwargs) | run the feature | +| execute(**kwargs) | execute the Python method | + + +### run(**kwargs) + +- allows executing any Python method by calling execute() +- sets the following attributes required to push the results to DB: + - criteria + - start_time + - stop_time +- doesn't fulfill details when pushing the results to the DB. + + +### execute(**kwargs) + +- the subclasses must override the default implementation which is false on purpose +- the new implementation must return 0 if success or anything else if failure. + + + +## Your second test case + + +### second.py + +```python +#!/usr/bin/env python + +from functest.core import feature + +class Test(feature.Feature): + + def execute(self, **kwargs): + print "Hello World" + return 0 +``` + + +### functest/ci/testcases.yaml + +```yaml +case_name: second +project_name: functest +criteria: 'status == "PASS"' +blocking: true +clean_flag: false +description: '' +dependencies: + installer: '' + scenario: '' +run: + module: 'second' + class: 'Test' +``` + + + +## class BashFeature +bases: Feature + +class designed to run any bash command + + +### execute(**kwargs) + +execute the cmd passed as arg. + + + +## Your third test case + + +### functest/ci/testcases.yaml + +``` +case_name: third +project_name: functest +criteria: 'status == "PASS"' +blocking: true +clean_flag: false +description: '' +dependencies: + installer: '' + scenario: '' +run: + module: 'functest.core.feature' + class: 'BashFeature' + args: + cmd: 'echo Hello World; exit 0' +``` + + + +## Euphrates + + +### Next actions + +- __to finish VNF abstraction (coverage + pylint)__ +- to publish doc API +- to manage criteria as written in testcases.yaml + +Please see [Functest Euphrates page](https://wiki.opnfv.org/display/functest/Functest+Euphrates+page) for more details + + + +## Thank You! diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst index e3485be40..9a1749789 100644 --- a/docs/testing/user/configguide/configguide.rst +++ b/docs/testing/user/configguide/configguide.rst @@ -336,6 +336,7 @@ should now be in place:: |-- parser |-- promise |-- rally + |-- refstack-client |-- releng |-- sdnvpn |-- securityscanning diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst index e050cf14b..c1faecdaa 100644 --- a/docs/testing/user/userguide/index.rst +++ b/docs/testing/user/userguide/index.rst @@ -247,13 +247,125 @@ The Rally testcases are distributed accross two Tiers: NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test cases. Test case 'rally_full' executes the full defined set of Rally tests. + +Refstack-client to run Defcore testcases +----------------------------------------- + +Refstack-client `[8]`_ is a command line utility that allows you to +execute Tempest test runs based on configurations you specify. +It is the official tool to run Defcore `[9]`_ testcases, +which focuses on testing interoperability between OpenStack clouds. + +Refstack-client is integrated in Functest, consumed by Dovetail, which +intends to define and provide a set of OPNFV related validation criteria +that will provide input for the evaluation of the use of OPNFV trademarks. +This progress is under the guideline of Compliance Verification Program(CVP). + +Defcore testcases +^^^^^^^^^^^^^^^^^^ + +*Danube Release* + +Set of DefCore tempest test cases not flagged and required. +According to `[10]`_, some tests are still flagged due to outstanding bugs +in the Tempest library, particularly tests that require SSH. Refstack developers +are working on correcting these bugs upstream. Please note that although some tests +are flagged because of bugs, there is still an expectation that the capabilities +covered by the tests are available. It only contains Openstack core compute +(no object storage). The approved guidelines (2016.08) are valid for Kilo, +Liberty, Mitaka and Newton releases of OpenStack. +The list can be generated using the Rest API from RefStack project: +https://refstack.openstack.org/api/v1/guidelines/2016.08/tests?target=compute&type=required&alias=true&flag=false + +Running methods +^^^^^^^^^^^^^^^ + +Two running methods are provided after refstack-client integrated into +Functest, Functest command line and manually, respectively. + +By default, for Defcore test cases run by Functest command line, +are run followed with automatically generated +configuration file, i.e., refstack_tempest.conf. In some circumstances, +the automatic configuration file may not quite satisfied with the SUT, +Functest also inherits the refstack-client command line and provides a way +for users to set its configuration file according to its own SUT manually. + +*command line* + +Inside the Functest container, first to prepare Functest environment: + +:: + + cd /home/opnfv/repos/functest + pip install -e . + functest env prepare + +then to run default defcore testcases by using refstack-client: + +:: + + functest testcase run refstack_defcore + +In OPNFV Continuous Integration(CI) system, the command line method is used. + +*manually* + +Inside the Functest container, first to prepare the refstack virtualenv: + +:: + + cd /home/opnfv/repos/refstack-client + source .venv/bin/activate + +then prepare the tempest configuration file and the testcases want to run with the SUT, +run the testcases with: + +:: + + ./refstack-client test -c <Path of the tempest configuration file to use> -v --test-list <Path or URL of test list> + +using help for more information: + +:: + + ./refstack-client --help + ./refstack-client test --help + +Reference tempest configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +*command line method* + +When command line method is used, the default tempest configuration file +is generated by Rally. + +*manually* + +When running manually is used, recommended way to generate tempest configuration +file is: + +:: + + cd /home/opnfv/repos/functest/functest/opnfv_tests/openstack/refstack_client + python tempest_conf.py + +a file called tempest.conf is stored in the current path by default, users can do +some adjustment according to the SUT: + +:: + + vim refstack_tempest.conf + +a reference article can be used `[15]`_. + + snaps_smoke ------------ This test case contains tests that setup and destroy environments with VMs with and without Floating IPs with a newly created user and project. Set the config value snaps.use_floating_ips (True|False) to toggle this functionality. When -the config value of snaps.use_keystone is True, functest must have access +the config value of snaps.use_keystone is True, Functest must have access the cloud's private network. This suite consists in 38 tests (test duration < 10 minutes) @@ -362,7 +474,7 @@ The test cases are described as follows: Features -------- -In Danube, functest supports the integration of: +In Danube, Functest supports the integration of: * barometer * bgpvpn @@ -520,10 +632,14 @@ References .. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml .. _`[6]`: https://scap.nist.gov/ .. _`[7]`: https://github.com/OpenSCAP/openscap +.. _`[8]`: https://github.com/openstack/refstack-client +.. _`[9]`: https://github.com/openstack/defcore +.. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst .. _`[11]`: http://robotframework.org/ .. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html .. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter +.. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/ `OPNFV main site`_ |