diff options
Diffstat (limited to 'docs/testing/developer/genericframework')
| -rw-r--r-- | docs/testing/developer/genericframework/index.rst | 382 |
1 files changed, 382 insertions, 0 deletions
diff --git a/docs/testing/developer/genericframework/index.rst b/docs/testing/developer/genericframework/index.rst new file mode 100644 index 00000000..9bafb3e4 --- /dev/null +++ b/docs/testing/developer/genericframework/index.rst @@ -0,0 +1,382 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Huawei Technologies Co.,Ltd, and others + +==================================== +Dovetail as a Generic Test Framework +==================================== + +.. toctree:: + :maxdepth: 2 + + +Overview +======== + +Dovetail is responsible for the technical realization of the OPNFV Verified +Program (OVP) and other compliance verification projects within the scope of +the Linux Foundation Networking (LFN) umbrella projects. +Dovetail provides a generic framework for executing a specific set of test cases +which define the scope of a given compliance verification program, such as OVP. + +This document aims at introducing what Dovetail generic framework looks like and +how to develop within this framework. + + +Introduction of Dovetail Framework +================================== + +The following diagram illustrates Dovetail generic framework. + +.. image:: ../../../images/dovetail_generic_framework.png + :align: center + :scale: 50% + +In this diagram, there are 5 main parts, `TestcaseFactory`, `TestRunnerFactory`, +`CrawlerFactory`, `CheckerFactory` and test case groups. + +- **TestcaseFactory**: For each project, there needs to create its own + testcase class such as `FunctestTestcase` and `OnapVtpTestcase`. All these + classes are based on class `Testcase`. There are already many functions in this + base class which are mainly used to parse test case configuration files. If no + other special requirements exist, it only needs to initialize these classes with + different types. Otherwise, it needs to overload or add some functions. + +- **TestRunnerFactory**: Similar to `TestcaseFactory`, each project has its own + test runner class. Dovetail supports 2 kinds of test runners, `DockerRunner` + and `ShellRunner`. For projects based on Docker, it needs to create + their own test runner classes such as `FunctestRunner` which inherit from class + `DockerRunner`. For other projects that are based on Shell, it can use `ShellRunner` + directly. Test case runners provide many functions to support test cases runs + such as preparing test tool of each project, run all the commands defined by + each test case and clean the environment. + +- **Test case groups**: Each group is composed of one project configuration file + and a set of test cases belonging to this project. These groups are used as the + input of test runners to provide information of projects and test cases. For + `ShellRunner`, it only needs test case configuratons as the input. + +- **CrawlerFactory**: This is used to parse the results of test cases and record + them with unified format. The original result data report by each project is + different. So it needs to create different crawler classes for different projects + to parse their results. + +- **CheckerFactory**: This is used to check the result data generated by crawler. + Each project should have its own checker class due to the different requirements + of different projects. + + +Development with Dovetail Framework +=================================== + +Everyone who is interested in developing Dovetail framework to integrate new upstream +test cases will face one of the two following scenarios: + +- **Adding test cases that belong to integrated projects**: There are already some + projects integrated in Dovetail. These projects are coming from OPNFV (Open Platform + for NFV) and ONAP (Open Network Automation Platform) communities. It will be + much easier to add new test cases that belong to these projects. + +- **Adding test cases that not belong to integrated projects**: The test cases + may belong to other projects that haven't been integrated into Dovetail yet. + These projects could be in OPNFV, ONAP or other communities. This scenario is a + little more complicated. + + +Test cases belonging to integrated projects +------------------------------------------- + +Dovetail framework already includes a large amount of test cases. All these test +cases are implemented by upstream projects in OPNFV and ONAP. The upstream +projects already integrated in Dovetail are Functest, Yardstick and Bottlenecks +from OPNFV and VNF SDK and VVP from ONAP. + +In order to add a test case belonging to one of these projects, there +only need to add one test case configuration file which is in yaml format. +Following is the introduction about how to use the file to add one new test case. +Please refer to `Dovetail test case github +<https://github.com/opnfv/dovetail/tree/master/etc/testcase>`_ +for all configuration files of all test cases. + +.. code-block:: bash + + --- + Test case name in Dovetail: + name: Test case name in Dovetail + objective: Test case description + validate: + type: 'shell' or name of the project already integrated in Dovetail + testcase: The test case name called in this project + image_name: Name of the Docker image used to run this test + pre_condition: + - 'Commands needed to be executed before running this test' + - 'e.g. cp src_file dest_file' + cmds: + - 'Commands used to run this test case' + post_condition: + - 'Commands needed to be executed after running this test case' + report: + source_archive_files: + - test.log + dest_archive_files: + - path/to/archive/test.log + check_results_file: results.json + sub_testcase_list: + - sub_test_1 + - sub_test_2 + - sub_test_3 + +This is the complete format of test case configuration file. Here are some +detailed description for each of the configuration options. + +- **Test case name in Dovetail**: All test cases should be named as 'xxx.yyy.zzz'. + This is the name in Dovetail and has no relationship with its name in its own + project. The first part is used to identify the project where this test case + come from (e.g. functest, onap-vtp). The second part is used to classify this + test case according to test area (e.g. healthcheck, ha). Dovetail supports to + run whole test cases in one test suite with the same test area. Also the area + is used to group all test cases and generate the summary report at the end of + the test. The last part is special for this test case itself (e.g. image, + haproxy, csar). It's better to keep the file name the same as the test case + name to make it easier to find the config file according to this test case + name in Dovetail. + +- **validate**: This is the main section to define how to run this test case. + + - **type**: This is the type of this test case. It can be `shell` which means + running this test case with Linux bash commands within Dovetail container. Also it + can be one of the projects already integrated in Dovetail (functest, yardstick, + bottlenecks, onap-vtp and onap-vvp). Then this type is used to map to its project + configuration yaml file. For example, in order to add a test case + in OPNFV project Functest to Dovetail framework, the type here should be + `functest`, and will map to `functest_config.yml` for more configurations + in project level. Please refer to `Dovetail project config github + <https://github.com/opnfv/dovetail/tree/master/etc/conf>`_ for more details. + + - **testcase**: This is the name defined in its own project. One test case can + be uniquely identified by `type` and `testcase`. Take the test case + `functest.vping.ssh` as an example. Its `type` is 'functest' and `testcase` + is 'vping_ssh'. With these 2 properties, it can be uniquely identified. Users only + need to know that there is a test case named `functest.vping.ssh` in OVP + compliance test scope. Dovetail Framework will run `vping_ssh` within Functest + Docker container. + + - **image_name**: [optional] If the type is `shell`, there is no need to give + this. For other types, there are default docker images defined in their project + configuration files. If this test case uses a different docker image, it needs + to overwrite it by adding `image_name` here. The `image_name` here should only + be the docker image name without tag. The tag is defined in project's configuration + file for all test cases belonging to this project. + + - **pre_condition**: [optional] A list of all preparations needed by this + test case. If the list is the same as the default one in its project configuration + file, then there is no need to repeat it here. Otherwise, it's necessary to + overwrite it. If its type is `shell`, then all commands in `pre_condition`, + `cmds` and `post_condition` should be executable within Dovetail Ubuntu 14.04 + Docker container. If its type is one of the Docker runner projects, then all + commands should be executable within their own containers. For Functest, it's + alpine 3.8. For Yardstick and Bottlenecks it's Ubuntu 16.04. For VNF SDK it's + Ubuntu 14.04. Also all these commands should not require network connection + because some commercial platforms may be offline environments in private labs. + + - **cmds**: [optional] A list of all commands used to run this test case. + + - **post_condition**: [optional] A list of all commands needed after executing + this test case such as some clean up operations. + +- **report**: This is the section for this test case to archive some log files and + provide the result file for reporting PASS or FAIL. + + - **source_archive_files**: [optional] If there is no need to archive any files, + this section can be removed. Otherwise, this is a list of all source files + needed to be archived. All files generated by all integrated projects will be + put under `$DOVETAIL_HOME/results`. In order to classify and avoid overwriting + them, it needs to rename some important files or move them to new directories. + Navigating directory `$DOVETAIL_HOME/results` to find out all files + needed to be archived. The paths here should be relative ones according to + `$DOVETAIL_HOME/results`. + + - **dest_archive_files**: [optional] This should be a list corresponding to the + list of `source_archive_files`. Also all paths here should be relative ones + according to `$DOVETAIL_HOME/results`. + + - **check_results_file**: This should be the name and relative path of the result + file generated by this test case. Dovetail will parse this file to get the + result (PASS or FAIL). + + - **sub_testcase_list**: [optional] This section is almost only for Tempest tests + in Functest. Take `functest.tempest.osinterop` as an example. The `sub_testcase_list` + here is an check list for this kind of tempest tests. Only when all sub test + cases list here are passed, this test case can be taken as PASS. The other kind + of tempest tests is `tempest_custom` such as `functest.tempest.image`. Besides + taking the `sub_testcase_list` as the check list, it's also used to generate an + input file of Functest to define the list of sub test cases to be tested. + + +Test cases not belonging to integrated projects +----------------------------------------------- + +If test cases waiting to be added into Dovetail do not belong to any project +that is already integrated into Dovetail framework, then besides adding the test +case configuration files introduced before, there are some other files needed to +be added or modified. + + +Step 1: Add a project configuration file +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For a new test case that belongs to a new project, it needs to create a project +configuration file to define this new project in Dovetail first. Now Dovetail +only supports integration of projects by using their Docker images. If this test +case should be run with shell runner, then can only add test case configuration +files with `type` 'shell' as describing before and skip the following steps. Following is +the introduction of how to use project configuration file to add one new project +into Dovetail. Please refer to `Dovetail projects configuration github +<https://github.com/opnfv/dovetail/tree/master/etc/conf>`_ for all configuration +files of all integrated projects. + +.. code-block:: bash + + --- + + {% set validate_testcase = validate_testcase or '' %} + {% set testcase = testcase or '' %} + {% set dovetail_home = dovetail_home or '' %} + {% set debug = debug or 'false' %} + {% set build_tag = build_tag or '' %} + {% set userconfig_dir = '/tmp/userconfig' %} + {% set patches_dir = '/tmp/patches' %} + {% set result_dir = '/tmp/results' %} + + project name: + image_name: name of the docker image + docker_tag: tag of the docker image + opts: options needed such as '-itd' + envs: envs used to create containers such as '-e DEBUG={{debug}}' + volumes: + - '-v {{dovetail_home}}/pre_config:/home/opnfv/pre_config' + - '-v {{dovetail_home}}/userconfig:{{userconfig_dir}}' + - '-v {{dovetail_home}}/patches:{{patches_dir}}' + - '-v {{dovetail_home}}/results:{{result_dir}}' + patches_dir: {{patches_dir}} + pre_condition: + - 'Commands needed to be executed before running this test' + cmds: + - 'Commands used to run this test case' + post_condition: + - 'Commands needed to be executed after running this test case' + openrc: absolute path of openstack credential files + extra_container: + - container1_name + - container2_name + +This is the complete format of project configuration file. Here are some +detailed description for each of the configuration options. + +- **Jinja Template**: At the begining of this yaml file, it uses Jinja template + to define some parameters that will be used somewhere in this file (e.g. result_dir). + Also there are some parameters providing by Dovetail framework as input of this + file, and other parameters can be defined by using these ones (e.g. testcase and + dovetail_home). The whole input parameters are list below. + + - **validate_testcase**: This is the name of the test case instance which calls this + project configuration file. The name is provided by the configuration file + of this test case (validate -> testcase). + + - **testcase**: This is the name of the test case which calls this project + configuration file. Different from `validate_testcase`, this is the name + defined in Dovetail not its own project. + + - **os_insecure**: This is only for test cases aiming at OpenStack. This is + `True` or `False` according to `env_config.sh` file. + + - **cacert**: This is also only for OpenStack test cases. It is the absolute + path of the OpenStack certificate provided in `env_config.sh` file. + + - **deploy_scenario**: This is the input when running Dovetail with option + `--deploy-scenario`. + + - **ram_num**: This is the input when running Dovetail with option + `--ram-num`. + + - **dovetail_home**: This is the `DOVETAIL_HOME` getting from the ENV. + + - **debug**: This is `True` or `False` according to the command running test + cases with or without option `--debug`. + + - **build_tag**: This is a string includes the UUID generated by Dovetail. + + - **host_url**: This is only for ONAP VNF SDK to get the HOST_URL provided + in `env_config.sh` file. + + - **csar_file**: This is also only for ONAP VNF SDK to get the CSAR_FILE + provided in `env_config.sh` file. + +- **project name**: This is the project name defined in Dovetail. For example + OPNFV Functest project is named as 'functest' here in Dovetail. This project + name will be used by test case configuration files as well as somewhere in + Dovetail source code. + +- **image_name**: This is the name of the default Docker image for all test cases + within this project. Each test case can overwrite it with its own configuration. + +- **docker_tag**: This is the tag of all Docker images for all test cases within + this project. For each release, it should use one Docker image with a stable + and official release version. + +- **opts**: Here are all options used to run Docker containers except envs and + volume mappings (e.g. '-it --privileged=true'). + +- **envs**: Here are all envs used to run Docker containers (e.g. '-e ONE=one + -e TWO=two'). + +- **volumes**: A volume mapping list used to run Docker containers. Every project + should at least map the `$DOVETAIL_HOME/pre_config` and `$DOVETAIL_HOME/results` + in the test host to containers to get config files and collect all result files. + +- **patches_dir**: This is an absolute path of the patches applied to the containers. + +- **pre_condition**: A list of all default preparations needed by this project. + It can be overwritten by configurations of test cases. + +- **cmds**: A list of all default commands used to run all test cases within + this project. Also it can be overwritten by configurations of test cases. + +- **post_condition**: A list of all default cleaning commands needed by this + project. + +- **openrc**: [optional] If the system under test is OpenStack, then it needs to + provide the absolute path here to copy the credential file in the Test Host to + containers. + + +Step 2: Add related classes +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +After adding the project and test case configuration files, there also need to +add some related classes into the source code. + +- **Test Case class**: Each project should have its own test case class in + `testcase.py` for `TestcaseFactory`. + +- **Test Runner class**: Each project should have its own test runner class in + `test_runner.py` for `TestRunnerFactory`. + +- **Crawler class**: Each project should have its own test results crawler class + in `report.py` for `CrawlerFactory`. + +- **Checker class**: Each project should have its own test results checker class + in `report.py` for `CheckerFactory`. + + +Step 3: Create related logs +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If the classes added in step2 have function `create_log`, then need to call +these functions in `run.py` to initial the log instances at the very begining. + + +Step 4: Update unit tests +^^^^^^^^^^^^^^^^^^^^^^^^^ + +A patch is not going to be verified without 100% coverage when applying acceptance check. |