diff options
| -rw-r--r-- | docs/testing/developer/genericframework/index.rst | 135 |
1 files changed, 83 insertions, 52 deletions
diff --git a/docs/testing/developer/genericframework/index.rst b/docs/testing/developer/genericframework/index.rst index 9bafb3e4..75721469 100644 --- a/docs/testing/developer/genericframework/index.rst +++ b/docs/testing/developer/genericframework/index.rst @@ -13,7 +13,7 @@ Dovetail as a Generic Test Framework Overview ======== -Dovetail is responsible for the technical realization of the OPNFV Verified +Dovetail is responsible for the technical realization of the OPNFV Verification Program (OVP) and other compliance verification projects within the scope of the Linux Foundation Networking (LFN) umbrella projects. Dovetail provides a generic framework for executing a specific set of test cases @@ -33,7 +33,7 @@ The following diagram illustrates Dovetail generic framework. :scale: 50% In this diagram, there are 5 main parts, `TestcaseFactory`, `TestRunnerFactory`, -`CrawlerFactory`, `CheckerFactory` and test case groups. +`CrawlerFactory`, `CheckerFactory` and `test case groups`. - **TestcaseFactory**: For each project, there needs to create its own testcase class such as `FunctestTestcase` and `OnapVtpTestcase`. All these @@ -106,7 +106,7 @@ for all configuration files of all test cases. objective: Test case description validate: type: 'shell' or name of the project already integrated in Dovetail - testcase: The test case name called in this project + testcase: The original test case name called in the project that it is developed image_name: Name of the Docker image used to run this test pre_condition: - 'Commands needed to be executed before running this test' @@ -120,7 +120,9 @@ for all configuration files of all test cases. - test.log dest_archive_files: - path/to/archive/test.log - check_results_file: results.json + check_results_files: + - results.json + portal_key_file: path/to/key/logs/xxx.log sub_testcase_list: - sub_test_1 - sub_test_2 @@ -130,7 +132,7 @@ This is the complete format of test case configuration file. Here are some detailed description for each of the configuration options. - **Test case name in Dovetail**: All test cases should be named as 'xxx.yyy.zzz'. - This is the name in Dovetail and has no relationship with its name in its own + This is the alias in Dovetail and has no relationship with its name in its own project. The first part is used to identify the project where this test case come from (e.g. functest, onap-vtp). The second part is used to classify this test case according to test area (e.g. healthcheck, ha). Dovetail supports to @@ -139,7 +141,7 @@ detailed description for each of the configuration options. the test. The last part is special for this test case itself (e.g. image, haproxy, csar). It's better to keep the file name the same as the test case name to make it easier to find the config file according to this test case - name in Dovetail. + alias in Dovetail. - **validate**: This is the main section to define how to run this test case. @@ -156,7 +158,7 @@ detailed description for each of the configuration options. - **testcase**: This is the name defined in its own project. One test case can be uniquely identified by `type` and `testcase`. Take the test case `functest.vping.ssh` as an example. Its `type` is 'functest' and `testcase` - is 'vping_ssh'. With these 2 properties, it can be uniquely identified. Users only + is 'vping_ssh'. With these 2 properties, it can be uniquely identified. End users only need to know that there is a test case named `functest.vping.ssh` in OVP compliance test scope. Dovetail Framework will run `vping_ssh` within Functest Docker container. @@ -200,9 +202,12 @@ detailed description for each of the configuration options. list of `source_archive_files`. Also all paths here should be relative ones according to `$DOVETAIL_HOME/results`. - - **check_results_file**: This should be the name and relative path of the result - file generated by this test case. Dovetail will parse this file to get the - result (PASS or FAIL). + - **check_results_files**: This should be a list of relative paths of + the result files generated by this test case. Dovetail will parse these files + to get the result (PASS or FAIL). + + - **portal_key_file**: This should be the key log file of this test case which will + be used by the OVP portal for review. - **sub_testcase_list**: [optional] This section is almost only for Tempest tests in Functest. Take `functest.tempest.osinterop` as an example. The `sub_testcase_list` @@ -247,17 +252,29 @@ files of all integrated projects. {% set userconfig_dir = '/tmp/userconfig' %} {% set patches_dir = '/tmp/patches' %} {% set result_dir = '/tmp/results' %} + {% set openrc_file = '/home/conf/env_file' %} project name: image_name: name of the docker image docker_tag: tag of the docker image - opts: options needed such as '-itd' - envs: envs used to create containers such as '-e DEBUG={{debug}}' + opts: + detach: true + stdin_open: true + privileged: true + shell: '/bin/bash' + envs: + - 'CI_DEBUG={{debug}}' + - 'DEPLOY_SCENARIO={{deploy_scenario}}' + - 'ENV_NAME=env_value' volumes: - - '-v {{dovetail_home}}/pre_config:/home/opnfv/pre_config' - - '-v {{dovetail_home}}/userconfig:{{userconfig_dir}}' - - '-v {{dovetail_home}}/patches:{{patches_dir}}' - - '-v {{dovetail_home}}/results:{{result_dir}}' + - '{{dovetail_home}}/userconfig:{{userconfig_dir}}' + - '{{dovetail_home}}/results:{{result_dir}}' + - '/path/on/host:/path/in/container' + - '/path/of/host/file:/file/path/in/container' + mounts: + - 'source={{dovetail_home}}/pre_config/env_config.sh,target={{openrc_file}} + - 'source={{dovetail_home}}/pre_config,target=/home/opnfv/pre_config' + - 'source=/file/or/derectory/on/host,target=/file/or/derectory/in/container' patches_dir: {{patches_dir}} pre_condition: - 'Commands needed to be executed before running this test' @@ -274,67 +291,77 @@ This is the complete format of project configuration file. Here are some detailed description for each of the configuration options. - **Jinja Template**: At the begining of this yaml file, it uses Jinja template - to define some parameters that will be used somewhere in this file (e.g. result_dir). - Also there are some parameters providing by Dovetail framework as input of this - file, and other parameters can be defined by using these ones (e.g. testcase and - dovetail_home). The whole input parameters are list below. - - - **validate_testcase**: This is the name of the test case instance which calls this - project configuration file. The name is provided by the configuration file - of this test case (validate -> testcase). + to define some parameters that will be used somewhere in this file (e.g. result_dir + and openrc_file). Besides those, there are some other parameters providing by Dovetail + framework as input of this file, and other parameters can be defined by using these + ones (e.g. testcase and dovetail_home). The whole input parameters which can be used + are list below. - - **testcase**: This is the name of the test case which calls this project - configuration file. Different from `validate_testcase`, this is the name - defined in Dovetail not its own project. + - **attack_host**: This is the attack host name of the test case which calls this + project configuration file. It's only for HA test cases and can be given in HA + configuration file `pod.yaml`. - - **os_insecure**: This is only for test cases aiming at OpenStack. This is - `True` or `False` according to `env_config.sh` file. + - **attack_process**: This is the attack process name of the test case which calls + this project configuration file. It's only for HA test cases and can be given in HA + configuration file `pod.yaml`. - - **cacert**: This is also only for OpenStack test cases. It is the absolute - path of the OpenStack certificate provided in `env_config.sh` file. + - **build_tag**: This is a string includes the UUID generated by Dovetail. - - **deploy_scenario**: This is the input when running Dovetail with option - `--deploy-scenario`. + - **cacert**: This is also only for OpenStack test cases. It is the absolute + path of the OpenStack certificate provided in `env_config.sh` file. - - **ram_num**: This is the input when running Dovetail with option - `--ram-num`. + - **deploy_scenario**: This is the input when running Dovetail with option + `--deploy-scenario`. - - **dovetail_home**: This is the `DOVETAIL_HOME` getting from the ENV. + - **debug**: This is `True` or `False` according to the command running test + cases with or without option `--debug`. - - **debug**: This is `True` or `False` according to the command running test - cases with or without option `--debug`. + - **dovetail_home**: This is the `DOVETAIL_HOME` getting from the ENV. - - **build_tag**: This is a string includes the UUID generated by Dovetail. + - **os_insecure**: This is only for test cases aiming at OpenStack. This is + `True` or `False` according to `env_config.sh` file. - - **host_url**: This is only for ONAP VNF SDK to get the HOST_URL provided - in `env_config.sh` file. + - **testcase**: This is the name of the test case which calls this project + configuration file. Different from `validate_testcase`, this is the alias + defined in Dovetail not in its own project. - - **csar_file**: This is also only for ONAP VNF SDK to get the CSAR_FILE - provided in `env_config.sh` file. + - **validate_testcase**: This is the name of the test case instance which calls this + project configuration file. The name is provided by the configuration file + of this test case (validate -> testcase). - **project name**: This is the project name defined in Dovetail. For example OPNFV Functest project is named as 'functest' here in Dovetail. This project name will be used by test case configuration files as well as somewhere in Dovetail source code. -- **image_name**: This is the name of the default Docker image for all test cases +- **image_name**: This is the name of the default Docker image for most test cases within this project. Each test case can overwrite it with its own configuration. - **docker_tag**: This is the tag of all Docker images for all test cases within this project. For each release, it should use one Docker image with a stable and official release version. -- **opts**: Here are all options used to run Docker containers except envs and - volume mappings (e.g. '-it --privileged=true'). +- **opts**: Here are all options used to run Docker containers except 'image', + 'command', 'environment', 'volumes', 'mounts' and 'extra_hosts'. For example, + the options include 'detach', 'privileged' and 'tty'. The full list of all + options can be found in `Docker python SDK docs <https://docker-py.readthedocs.io/en/stable/containers.html>`_. + +- **shell**: This is the command used to run in the container. -- **envs**: Here are all envs used to run Docker containers (e.g. '-e ONE=one - -e TWO=two'). +- **envs**: This is a list of all envs used to run Docker containers. -- **volumes**: A volume mapping list used to run Docker containers. Every project - should at least map the `$DOVETAIL_HOME/pre_config` and `$DOVETAIL_HOME/results` - in the test host to containers to get config files and collect all result files. +- **volumes**: A volume mapping list used to run Docker containers. The source volumes + list here are allowed to be nonexistent and Docker will create new directories for them + on the host. Every project should at least map the `$DOVETAIL_HOME/results` + in the test host to containers to collect all result files. -- **patches_dir**: This is an absolute path of the patches applied to the containers. +- **mounts**: A mount mapping list used to run Docker containers. More powerful alternative + to **volumes**. The source volumes list here are not allowed to be nonexistent. + Every project should at least mount the `$DOVETAIL_HOME/pre_config` in the test host to + containers to get config files. + +- **patches_dir**: [optional] This is an absolute path of the patches applied to + the containers. - **pre_condition**: A list of all default preparations needed by this project. It can be overwritten by configurations of test cases. @@ -349,6 +376,10 @@ detailed description for each of the configuration options. provide the absolute path here to copy the credential file in the Test Host to containers. +- **extra_container**: [optional] The extra containers needed to be removed at the + end of the test. These containers are created by the test cases themselves at + runtime rather than created by Dovetail. + Step 2: Add related classes ^^^^^^^^^^^^^^^^^^^^^^^^^^^ |