Update Dovetail generic framework guide

JIRA: DOVETAIL-797

Change-Id: I9be48de191591deeb356e4f2ad0e8508720d8e54
Signed-off-by: xudan <xudan16@huawei.com>

1 files changed, 83 insertions, 52 deletions

diff --git a/docs/testing/developer/genericframework/index.rst b/docs/testing/developer/genericframework/index.rst
index 9bafb3e4..75721469 100644
--- a/docs/testing/developer/genericframework/index.rst
+++ b/docs/testing/developer/genericframework/index.rst
@@ -13,7 +13,7 @@ Dovetail as a Generic Test Framework
 Overview
 ========
 
-Dovetail is responsible for the technical realization of the OPNFV Verified
+Dovetail is responsible for the technical realization of the OPNFV Verification
 Program (OVP) and other compliance verification projects within the scope of
 the Linux Foundation Networking (LFN) umbrella projects.
 Dovetail provides a generic framework for executing a specific set of test cases
@@ -33,7 +33,7 @@ The following diagram illustrates Dovetail generic framework.
     :scale: 50%
 
 In this diagram, there are 5 main parts, `TestcaseFactory`, `TestRunnerFactory`,
-`CrawlerFactory`, `CheckerFactory` and test case groups.
+`CrawlerFactory`, `CheckerFactory` and `test case groups`.
 
 - **TestcaseFactory**: For each project, there needs to create its own
   testcase class such as `FunctestTestcase` and `OnapVtpTestcase`. All these
@@ -106,7 +106,7 @@ for all configuration files of all test cases.
      objective: Test case description
      validate:
        type: 'shell' or name of the project already integrated in Dovetail
-       testcase: The test case name called in this project
+       testcase: The original test case name called in the project that it is developed
        image_name: Name of the Docker image used to run this test
        pre_condition:
          - 'Commands needed to be executed before running this test'
@@ -120,7 +120,9 @@ for all configuration files of all test cases.
          - test.log
        dest_archive_files:
          - path/to/archive/test.log
-       check_results_file: results.json
+       check_results_files:
+         - results.json
+       portal_key_file: path/to/key/logs/xxx.log
        sub_testcase_list:
          - sub_test_1
          - sub_test_2
@@ -130,7 +132,7 @@ This is the complete format of test case configuration file. Here are some
 detailed description for each of the configuration options.
 
 - **Test case name in Dovetail**: All test cases should be named as 'xxx.yyy.zzz'.
-  This is the name in Dovetail and has no relationship with its name in its own
+  This is the alias in Dovetail and has no relationship with its name in its own
   project. The first part is used to identify the project where this test case
   come from (e.g. functest, onap-vtp). The second part is used to classify this
   test case according to test area (e.g. healthcheck, ha). Dovetail supports to
@@ -139,7 +141,7 @@ detailed description for each of the configuration options.
   the test. The last part is special for this test case itself (e.g. image,
   haproxy, csar). It's better to keep the file name the same as the test case
   name to make it easier to find the config file according to this test case
-  name in Dovetail.
+  alias in Dovetail.
 
 - **validate**: This is the main section to define how to run this test case.
 
@@ -156,7 +158,7 @@ detailed description for each of the configuration options.
   - **testcase**: This is the name defined in its own project. One test case can
     be uniquely identified by `type` and `testcase`. Take the test case
     `functest.vping.ssh` as an example. Its `type` is 'functest' and `testcase`
-    is 'vping_ssh'. With these 2 properties, it can be uniquely identified. Users only
+    is 'vping_ssh'. With these 2 properties, it can be uniquely identified. End users only
     need to know that there is a test case named `functest.vping.ssh` in OVP
     compliance test scope. Dovetail Framework will run `vping_ssh` within Functest
     Docker container.
@@ -200,9 +202,12 @@ detailed description for each of the configuration options.
     list of `source_archive_files`. Also all paths here should be relative ones
     according to `$DOVETAIL_HOME/results`.
 
-  - **check_results_file**: This should be the name and relative path of the result
-    file generated by this test case. Dovetail will parse this file to get the
-    result (PASS or FAIL).
+  - **check_results_files**: This should be a list of relative paths of
+    the result files generated by this test case. Dovetail will parse these files
+    to get the result (PASS or FAIL).
+
+  - **portal_key_file**: This should be the key log file of this test case which will
+    be used by the OVP portal for review.
 
   - **sub_testcase_list**: [optional] This section is almost only for Tempest tests
     in Functest. Take `functest.tempest.osinterop` as an example. The `sub_testcase_list`
@@ -247,17 +252,29 @@ files of all integrated projects.
    {% set userconfig_dir = '/tmp/userconfig' %}
    {% set patches_dir = '/tmp/patches' %}
    {% set result_dir = '/tmp/results' %}
+   {% set openrc_file = '/home/conf/env_file' %}
 
    project name:
      image_name: name of the docker image
      docker_tag: tag of the docker image
-     opts: options needed such as '-itd'
-     envs: envs used to create containers such as '-e DEBUG={{debug}}'
+     opts:
+       detach: true
+       stdin_open: true
+       privileged: true
+     shell: '/bin/bash'
+     envs:
+       - 'CI_DEBUG={{debug}}'
+       - 'DEPLOY_SCENARIO={{deploy_scenario}}'
+       - 'ENV_NAME=env_value'
      volumes:
-       - '-v {{dovetail_home}}/pre_config:/home/opnfv/pre_config'
-       - '-v {{dovetail_home}}/userconfig:{{userconfig_dir}}'
-       - '-v {{dovetail_home}}/patches:{{patches_dir}}'
-       - '-v {{dovetail_home}}/results:{{result_dir}}'
+       - '{{dovetail_home}}/userconfig:{{userconfig_dir}}'
+       - '{{dovetail_home}}/results:{{result_dir}}'
+       - '/path/on/host:/path/in/container'
+       - '/path/of/host/file:/file/path/in/container'
+     mounts:
+       - 'source={{dovetail_home}}/pre_config/env_config.sh,target={{openrc_file}}
+       - 'source={{dovetail_home}}/pre_config,target=/home/opnfv/pre_config'
+       - 'source=/file/or/derectory/on/host,target=/file/or/derectory/in/container'
      patches_dir: {{patches_dir}}
      pre_condition:
        - 'Commands needed to be executed before running this test'
@@ -274,67 +291,77 @@ This is the complete format of project configuration file. Here are some
 detailed description for each of the configuration options.
 
 - **Jinja Template**: At the begining of this yaml file, it uses Jinja template
-  to define some parameters that will be used somewhere in this file (e.g. result_dir).
-  Also there are some parameters providing by Dovetail framework as input of this
-  file, and other parameters can be defined by using these ones (e.g. testcase and
-  dovetail_home). The whole input parameters are list below.
-
-    - **validate_testcase**: This is the name of the test case instance which calls this
-      project configuration file. The name is provided by the configuration file
-      of this test case (validate -> testcase).
+  to define some parameters that will be used somewhere in this file (e.g. result_dir
+  and openrc_file). Besides those, there are some other parameters providing by Dovetail
+  framework as input of this file, and other parameters can be defined by using these
+  ones (e.g. testcase and dovetail_home). The whole input parameters which can be used
+  are list below.
 
-    - **testcase**: This is the name of the test case which calls this project
-      configuration file. Different from `validate_testcase`, this is the name
-      defined in Dovetail not its own project.
+  - **attack_host**: This is the attack host name of the test case which calls this
+    project configuration file. It's only for HA test cases and can be given in HA
+    configuration file `pod.yaml`.
 
-    - **os_insecure**: This is only for test cases aiming at OpenStack. This is
-      `True` or `False` according to `env_config.sh` file.
+  - **attack_process**: This is the attack process name of the test case which calls
+    this project configuration file. It's only for HA test cases and can be given in HA
+    configuration file `pod.yaml`.
 
-    - **cacert**: This is also only for OpenStack test cases. It is the absolute
-      path of the OpenStack certificate provided in `env_config.sh` file.
+  - **build_tag**: This is a string includes the UUID generated by Dovetail.
 
-    - **deploy_scenario**: This is the input when running Dovetail with option
-      `--deploy-scenario`.
+  - **cacert**: This is also only for OpenStack test cases. It is the absolute
+    path of the OpenStack certificate provided in `env_config.sh` file.
 
-    - **ram_num**: This is the input when running Dovetail with option
-      `--ram-num`.
+  - **deploy_scenario**: This is the input when running Dovetail with option
+    `--deploy-scenario`.
 
-    - **dovetail_home**: This is the `DOVETAIL_HOME` getting from the ENV.
+  - **debug**: This is `True` or `False` according to the command running test
+    cases with or without option `--debug`.
 
-    - **debug**: This is `True` or `False` according to the command running test
-      cases with or without option `--debug`.
+  - **dovetail_home**: This is the `DOVETAIL_HOME` getting from the ENV.
 
-    - **build_tag**: This is a string includes the UUID generated by Dovetail.
+  - **os_insecure**: This is only for test cases aiming at OpenStack. This is
+    `True` or `False` according to `env_config.sh` file.
 
-    - **host_url**: This is only for ONAP VNF SDK to get the HOST_URL provided
-      in `env_config.sh` file.
+  - **testcase**: This is the name of the test case which calls this project
+    configuration file. Different from `validate_testcase`, this is the alias
+    defined in Dovetail not in its own project.
 
-    - **csar_file**: This is also only for ONAP VNF SDK to get the CSAR_FILE
-      provided in `env_config.sh` file.
+  - **validate_testcase**: This is the name of the test case instance which calls this
+    project configuration file. The name is provided by the configuration file
+    of this test case (validate -> testcase).
 
 - **project name**: This is the project name defined in Dovetail. For example
   OPNFV Functest project is named as 'functest' here in Dovetail. This project
   name will be used by test case configuration files as well as somewhere in
   Dovetail source code.
 
-- **image_name**: This is the name of the default Docker image for all test cases
+- **image_name**: This is the name of the default Docker image for most test cases
   within this project. Each test case can overwrite it with its own configuration.
 
 - **docker_tag**: This is the tag of all Docker images for all test cases within
   this project. For each release, it should use one Docker image with a stable
   and official release version.
 
-- **opts**: Here are all options used to run Docker containers except envs and
-  volume mappings (e.g. '-it --privileged=true').
+- **opts**: Here are all options used to run Docker containers except 'image',
+  'command', 'environment', 'volumes', 'mounts' and 'extra_hosts'. For example,
+  the options include 'detach', 'privileged' and 'tty'. The full list of all
+  options can be found in `Docker python SDK docs <https://docker-py.readthedocs.io/en/stable/containers.html>`_.
+
+- **shell**: This is the command used to run in the container.
 
-- **envs**: Here are all envs used to run Docker containers (e.g. '-e ONE=one
-  -e TWO=two').
+- **envs**: This is a list of all envs used to run Docker containers.
 
-- **volumes**: A volume mapping list used to run Docker containers. Every project
-  should at least map the `$DOVETAIL_HOME/pre_config` and `$DOVETAIL_HOME/results`
-  in the test host to containers to get config files and collect all result files.
+- **volumes**: A volume mapping list used to run Docker containers. The source volumes
+  list here are allowed to be nonexistent and Docker will create new directories for them
+  on the host. Every project should at least map the `$DOVETAIL_HOME/results`
+  in the test host to containers to collect all result files.
 
-- **patches_dir**: This is an absolute path of the patches applied to the containers.
+- **mounts**: A mount mapping list used to run Docker containers. More powerful alternative
+  to **volumes**. The source volumes list here are not allowed to be nonexistent.
+  Every project should at least mount the `$DOVETAIL_HOME/pre_config` in the test host to
+  containers to get config files.
+
+- **patches_dir**: [optional] This is an absolute path of the patches applied to
+  the containers.
 
 - **pre_condition**: A list of all default preparations needed by this project.
   It can be overwritten by configurations of test cases.
@@ -349,6 +376,10 @@ detailed description for each of the configuration options.
   provide the absolute path here to copy the credential file in the Test Host to
   containers.
 
+- **extra_container**: [optional] The extra containers needed to be removed at the
+  end of the test. These containers are created by the test cases themselves at
+  runtime rather than created by Dovetail.
+
 
 Step 2: Add related classes
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^