path: root/docs/testing/developer/genericframework/index.rst

.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
.. (c) Huawei Technologies Co.,Ltd, and others

====================================
Dovetail as a Generic Test Framework
====================================

.. toctree::
   :maxdepth: 2


Overview
========

Dovetail is responsible for the technical realization of the OPNFV Verified
Program (OVP) and other compliance verification projects within the scope of
the Linux Foundation Networking (LFN) umbrella projects.
Dovetail provides a generic framework for executing a specific set of test cases
which define the scope of a given compliance verification program, such as OVP.

This document aims at introducing what Dovetail generic framework looks like and
how to develop within this framework.


Introduction of Dovetail Framework
==================================

The following diagram illustrates Dovetail generic framework.

.. image:: ../../../images/dovetail_generic_framework.png
    :align: center
    :scale: 50%

In this diagram, there are 5 main parts, `TestcaseFactory`, `TestRunnerFactory`,
`CrawlerFactory`, `CheckerFactory` and test case groups.

- **TestcaseFactory**: For each project, there needs to create its own
  testcase class such as `FunctestTestcase` and `OnapVtpTestcase`. All these
  classes are based on class `Testcase`. There are already many functions in this
  base class which are mainly used to parse test case configuration files. If no
  other special requirements exist, it only needs to initialize these classes with
  different types. Otherwise, it needs to overload or add some functions.

- **TestRunnerFactory**: Similar to `TestcaseFactory`, each project has its own
  test runner class. Dovetail supports 2 kinds of test runners, `DockerRunner`
  and `ShellRunner`. For projects based on Docker, it needs to create
  their own test runner classes such as `FunctestRunner` which inherit from class
  `DockerRunner`. For other projects that are based on Shell, it can use `ShellRunner`
  directly. Test case runners provide many functions to support test cases runs
  such as preparing test tool of each project, run all the commands defined by
  each test case and clean the environment.

- **Test case groups**: Each group is composed of one project configuration file
  and a set of test cases belonging to this project. These groups are used as the
  input of test runners to provide information of projects and test cases. For
  `ShellRunner`, it only needs test case configuratons as the input.

- **CrawlerFactory**: This is used to parse the results of test cases and record
  them with unified format. The original result data report by each project is
  different. So it needs to create different crawler classes for different projects
  to parse their results.

- **CheckerFactory**: This is used to check the result data generated by crawler.
  Each project should have its own checker class due to the different requirements
  of different projects.


Development with Dovetail Framework
===================================

Everyone who is interested in developing Dovetail framework to integrate new upstream
test cases will face one of the two following scenarios:

- **Adding test cases that belong to integrated projects**: There are already some
  projects integrated in Dovetail. These projects are coming from OPNFV (Open Platform
  for NFV) and ONAP (Open Network Automation Platform) communities. It will be
  much easier to add new test cases that belong to these projects.

- **Adding test cases that not belong to integrated projects**: The test cases
  may belong to other projects that haven't been integrated into Dovetail yet.
  These projects could be in OPNFV, ONAP or other communities. This scenario is a
  little more complicated.


Test cases belonging to integrated projects
-------------------------------------------

Dovetail framework already includes a large amount of test cases. All these test
cases are implemented by upstream projects in OPNFV and ONAP. The upstream
projects already integrated in Dovetail are Functest, Yardstick and Bottlenecks
from OPNFV and VNF SDK and VVP from ONAP.

In order to add a test case belonging to one of these projects, there
only need to add one test case configuration file which is in yaml format.
Following is the introduction about how to use the file to add one new test case.
Please refer to `Dovetail test case github
<https://github.com/opnfv/dovetail/tree/master/etc/testcase>`_
for all configuration files of all test cases.

.. code-block:: bash

   ---
   Test case name in Dovetail:
     name: Test case name in Dovetail
     objective: Test case description
     validate:
       type: 'shell' or name of the project already integrated in Dovetail
       testcase: The test case name called in this project
       image_name: Name of the Docker image used to run this test
       pre_condition:
         - 'Commands needed to be executed before running this test'
         - 'e.g. cp src_file dest_file'
       cmds:
         - 'Commands used to run this test case'
       post_condition:
         - 'Commands needed to be executed after running this test case'
     report:
       source_archive_files:
         - test.log
       dest_archive_files:
         - path/to/archive/test.log
       check_results_file: results.json
       sub_testcase_list:
         - sub_test_1
         - sub_test_2
         - sub_test_3

This is the complete format of test case configuration file. Here are some
detailed description for each of the configuration options.

- **Test case name in Dovetail**: All test cases should be named as 'xxx.yyy.zzz'.
  This is the name in Dovetail and has no relationship with its name in its own
  project. The first part is used to identify the project where this test case
  come from (e.g. functest, onap-vtp). The second part is used to classify this
  test case according to test area (e.g. healthcheck, ha). Dovetail supports to
  run whole test cases in one test suite with the same test area. Also the area
  is used to group all test cases and generate the summary report at the end of
  the test. The last part is special for this test case itself (e.g. image,
  haproxy, csar). It's better to keep the file name the same as the test case
  name to make it easier to find the config file according to this test case
  name in Dovetail.

- **validate**: This is the main section to define how to run this test case.

  - **type**: This is the type of this test case. It can be `shell` which means
    running this test case with Linux bash commands within Dovetail container. Also it
    can be one of the projects already integrated in Dovetail (functest, yardstick,
    bottlenecks, onap-vtp and onap-vvp). Then this type is used to map to its project
    configuration yaml file. For example, in order to add a test case
    in OPNFV project Functest to Dovetail framework, the type here should be
    `functest`, and will map to `functest_config.yml` for more configurations
    in project level. Please refer to `Dovetail project config github
    <https://github.com/opnfv/dovetail/tree/master/etc/conf>`_ for more details.

  - **testcase**: This is the name defined in its own project. One test case can
    be uniquely identified by `type` and `testcase`. Take the test case
    `functest.vping.ssh` as an example. Its `type` is 'functest' and `testcase`
    is 'vping_ssh'. With these 2 properties, it can be uniquely identified. Users only
    need to know that there is a test case named `functest.vping.ssh` in OVP
    compliance test scope. Dovetail Framework will run `vping_ssh` within Functest
    Docker container.

  - **image_name**: [optional] If the type is `shell`, there is no need to give
    this. For other types, there are default docker images defined in their project
    configuration files. If this test case uses a different docker image, it needs
    to overwrite it by adding `image_name` here. The `image_name` here should only
    be the docker image name without tag. The tag is defined in project's configuration
    file for all test cases belonging to this project.

  - **pre_condition**: [optional] A list of all preparations needed by this
    test case. If the list is the same as the default one in its project configuration
    file, then there is no need to repeat it here. Otherwise, it's necessary to
    overwrite it. If its type is `shell`, then all commands in `pre_condition`,
    `cmds` and `post_condition` should be executable within Dovetail Ubuntu 14.04
    Docker container. If its type is one of the Docker runner projects, then all
    commands should be executable within their own containers. For Functest, it's
    alpine 3.8. For Yardstick and Bottlenecks it's Ubuntu 16.04. For VNF SDK it's
    Ubuntu 14.04. Also all these commands should not require network connection
    because some commercial platforms may be offline environments in private labs.

  - **cmds**: [optional] A list of all commands used to run this test case.

  - **post_condition**: [optional] A list of all commands needed after executing
    this test case such as some clean up operations.

- **report**: This is the section for this test case to archive some log files and
  provide the result file for reporting PASS or FAIL.

  - **source_archive_files**: [optional] If there is no need to archive any files,
    this section can be removed. Otherwise, this is a list of all source files
    needed to be archived. All files generated by all integrated projects will be
    put under `$DOVETAIL_HOME/results`. In order to classify and avoid overwriting
    them, it needs to rename some important files or move them to new directories.
    Navigating directory `$DOVETAIL_HOME/results` to find out all files
    needed to be archived. The paths here should be relative ones according to
    `$DOVETAIL_HOME/results`.

  - **dest_archive_files**: [optional] This should be a list corresponding to the
    list of `source_archive_files`. Also all paths here should be relative ones
    according to `$DOVETAIL_HOME/results`.

  - **check_results_file**: This should be the name and relative path of the result
    file generated by this test case. Dovetail will parse this file to get the
    result (PASS or FAIL).

  - **sub_testcase_list**: [optional] This section is almost only for Tempest tests
    in Functest. Take `functest.tempest.osinterop` as an example. The `sub_testcase_list`
    here is an check list for this kind of tempest tests. Only when all sub test
    cases list here are passed, this test case can be taken as PASS. The other kind
    of tempest tests is `tempest_custom` such as `functest.tempest.image`. Besides
    taking the `sub_testcase_list` as the check list, it's also used to generate an
    input file of Functest to define the list of sub test cases to be tested.


Test cases not belonging to integrated projects
-----------------------------------------------

If test cases waiting to be added into Dovetail do not belong to any project
that is already integrated into Dovetail framework, then besides adding the test
case configuration files introduced before, there are some other files needed to
be added or modified.


Step 1: Add a project configuration file
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For a new test case that belongs to a new project, it needs to create a project
configuration file to define this new project in Dovetail first. Now Dovetail
only supports integration of projects by using their Docker images. If this test
case should be run with shell runner, then can only add test case configuration
files with `type` 'shell' as describing before and skip the following steps. Following is
the introduction of how to use project configuration file to add one new project
into Dovetail. Please refer to `Dovetail projects configuration github
<https://github.com/opnfv/dovetail/tree/master/etc/conf>`_ for all configuration
files of all integrated projects.

.. code-block:: bash

   ---

   {% set validate_testcase = validate_testcase or '' %}
   {% set testcase = testcase or '' %}
   {% set dovetail_home = dovetail_home or '' %}
   {% set debug = debug or 'false' %}
   {% set build_tag = build_tag or '' %}
   {% set userconfig_dir = '/tmp/userconfig' %}
   {% set patches_dir = '/tmp/patches' %}
   {% set result_dir = '/tmp/results' %}

   project name:
     image_name: name of the docker image
     docker_tag: tag of the docker image
     opts: options needed such as '-itd'
     envs: envs used to create containers such as '-e DEBUG={{debug}}'
     volumes:
       - '-v {{dovetail_home}}/pre_config:/home/opnfv/pre_config'
       - '-v {{dovetail_home}}/userconfig:{{userconfig_dir}}'
       - '-v {{dovetail_home}}/patches:{{patches_dir}}'
       - '-v {{dovetail_home}}/results:{{result_dir}}'
     patches_dir: {{patches_dir}}
     pre_condition:
       - 'Commands needed to be executed before running this test'
     cmds:
       - 'Commands used to run this test case'
     post_condition:
       - 'Commands needed to be executed after running this test case'
     openrc: absolute path of openstack credential files
     extra_container:
       - container1_name
       - container2_name

This is the complete format of project configuration file. Here are some
detailed description for each of the configuration options.

- **Jinja Template**: At the begining of this yaml file, it uses Jinja template
  to define some parameters that will be used somewhere in this file (e.g. result_dir).
  Also there are some parameters providing by Dovetail framework as input of this
  file, and other parameters can be defined by using these ones (e.g. testcase and
  dovetail_home). The whole input parameters are list below.

    - **validate_testcase**: This is the name of the test case instance which calls this
      project configuration file. The name is provided by the configuration file
      of this test case (validate -> testcase).

    - **testcase**: This is the name of the test case which calls this project
      configuration file. Different from `validate_testcase`, this is the name
      defined in Dovetail not its own project.

    - **os_insecure**: This is only for test cases aiming at OpenStack. This is
      `True` or `False` according to `env_config.sh` file.

    - **cacert**: This is also only for OpenStack test cases. It is the absolute
      path of the OpenStack certificate provided in `env_config.sh` file.

    - **deploy_scenario**: This is the input when running Dovetail with option
      `--deploy-scenario`.

    - **ram_num**: This is the input when running Dovetail with option
      `--ram-num`.

    - **dovetail_home**: This is the `DOVETAIL_HOME` getting from the ENV.

    - **debug**: This is `True` or `False` according to the command running test
      cases with or without option `--debug`.

    - **build_tag**: This is a string includes the UUID generated by Dovetail.

    - **host_url**: This is only for ONAP VNF SDK to get the HOST_URL provided
      in `env_config.sh` file.

    - **csar_file**: This is also only for ONAP VNF SDK to get the CSAR_FILE
      provided in `env_config.sh` file.

- **project name**: This is the project name defined in Dovetail. For example
  OPNFV Functest project is named as 'functest' here in Dovetail. This project
  name will be used by test case configuration files as well as somewhere in
  Dovetail source code.

- **image_name**: This is the name of the default Docker image for all test cases
  within this project. Each test case can overwrite it with its own configuration.

- **docker_tag**: This is the tag of all Docker images for all test cases within
  this project. For each release, it should use one Docker image with a stable
  and official release version.

- **opts**: Here are all options used to run Docker containers except envs and
  volume mappings (e.g. '-it --privileged=true').

- **envs**: Here are all envs used to run Docker containers (e.g. '-e ONE=one
  -e TWO=two').

- **volumes**: A volume mapping list used to run Docker containers. Every project
  should at least map the `$DOVETAIL_HOME/pre_config` and `$DOVETAIL_HOME/results`
  in the test host to containers to get config files and collect all result files.

- **patches_dir**: This is an absolute path of the patches applied to the containers.

- **pre_condition**: A list of all default preparations needed by this project.
  It can be overwritten by configurations of test cases.

- **cmds**: A list of all default commands used to run all test cases within
  this project. Also it can be overwritten by configurations of test cases.

- **post_condition**: A list of all default cleaning commands needed by this
  project.

- **openrc**: [optional] If the system under test is OpenStack, then it needs to
  provide the absolute path here to copy the credential file in the Test Host to
  containers.


Step 2: Add related classes
^^^^^^^^^^^^^^^^^^^^^^^^^^^

After adding the project and test case configuration files, there also need to
add some related classes into the source code.

- **Test Case class**: Each project should have its own test case class in
  `testcase.py` for `TestcaseFactory`.

- **Test Runner class**: Each project should have its own test runner class in
  `test_runner.py` for `TestRunnerFactory`.

- **Crawler class**: Each project should have its own test results crawler class
  in `report.py` for `CrawlerFactory`.

- **Checker class**: Each project should have its own test results checker class
  in `report.py` for `CheckerFactory`.


Step 3: Create related logs
^^^^^^^^^^^^^^^^^^^^^^^^^^^

If the classes added in step2 have function `create_log`, then need to call
these functions in `run.py` to initial the log instances at the very begining.


Step 4: Update unit tests
^^^^^^^^^^^^^^^^^^^^^^^^^

A patch is not going to be verified without 100% coverage when applying acceptance check.