aboutsummaryrefslogtreecommitdiffstats
path: root/docs/testing/developer
diff options
context:
space:
mode:
Diffstat (limited to 'docs/testing/developer')
-rw-r--r--docs/testing/developer/devguide/index.rst503
1 files changed, 259 insertions, 244 deletions
diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst
index ce5dc77be..d5295903c 100644
--- a/docs/testing/developer/devguide/index.rst
+++ b/docs/testing/developer/devguide/index.rst
@@ -1,3 +1,6 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. SPDX-License-Identifier: CC-BY-4.0
+
******************************
OPNFV FUNCTEST developer guide
******************************
@@ -62,18 +65,18 @@ Functest internal test cases
============================
The internal test cases in Danube are:
- * healthcheck
- * connection_check
+
* api_check
+ * cloudify_ims
+ * connection_check
* vping_ssh
* vping_userdata
* odl
- * snaps_smoke
- * tempest_smoke_serial
+ * rally_full
* rally_sanity
+ * snaps_health_check
* tempest_full_parallel
- * rally_full
- * cloudify_ims
+ * tempest_smoke_serial
By internal, we mean that this particular test cases have been
developped and/or integrated by functest contributors and the associated
@@ -86,7 +89,7 @@ The main internal test cases are in the opnfv_tests subfolder of the
repository, the internal test cases are:
* sdn: odl, onos
- * openstack: healthcheck, vping_ssh, vping_userdata, tempest_*, rally_*, connection_check, api_check, snaps_smoke
+ * openstack: api_check, connection_check, snaps_health_check, vping_ssh, vping_userdata, tempest_*, rally_*, snaps_smoke
* vnf: cloudify_ims
If you want to create a new test case you will have to create a new
@@ -99,19 +102,23 @@ especially the feature projects.
The external test cases are:
- * promise
- * doctor
- * onos
+ * barometer
* bgpvpn
- * copper
- * security_scan
- * sfc-odl
- * sfc-onos
- * parser
+ * doctor
* domino
+ * odl-netvirt
+ * onos
+ * fds
* multisite
- * opera_ims
+ * netready
* orchestra_ims
+ * parser
+ * promise
+ * refstack_defcore
+ * security_scan
+ * snaps_smoke
+ * sfc-odl
+ * vyos_vrouter
The code to run these test cases may be directly in the repository of
@@ -155,7 +162,7 @@ introduced in Danube:
The goal is to unify the way to run test from Functest.
-feature_base and vnf_base inherit from testcase_base.
+feature_base and vnf_base inherit from testcase_base::
+-----------------------------------------+
| |
@@ -174,9 +181,9 @@ feature_base and vnf_base inherit from testcase_base.
| feature_base | | vnf_base |
| | | |
| - prepare() | | - prepare() |
- | - post() | | - deploy_orchestrator() |
- | - parse_results() | | - deploy_vnf() |
- | | | - test_vnf() |
+ | - execute() | | - deploy_orchestrator() |
+ | - post() | | - deploy_vnf() |
+ | - parse_results() | | - test_vnf() |
| | | - clean() |
| | | - execute() |
| | | |
@@ -270,232 +277,13 @@ the API will return an error message.
An additional method dashboard has been added to post-process
the raw results in release Brahmaputra (deprecated in Colorado).
-The data model is very basic, 4 objects are created:
+The data model is very basic, 5 objects are created:
* Pods
* Projects
* Testcases
* Results
-
-Pods::
-
- {
- "id": <ID>,
- "details": <URL description of the POD>,
- "creation_date": "YYYY-MM-DD HH:MM:SS",
- "name": <The POD Name>,
- "mode": <metal or virtual>,
- "role": <ci-pod or community-pod or single-node>
- },
-
-Projects::
-
- {
- "id": <ID>,
- "name": <Name of the Project>,
- "creation_date": "YYYY-MM-DD HH:MM:SS",
- "description": <Short description>
- },
-
-Testcases::
-
- {
- "id": <ID>,
- "name":<Name of the test case>,
- "project_name":<Name of belonged project>,
- "creation_date": "YYYY-MM-DD HH:MM:SS",
- "description": <short description>,
- "url":<URL for longer description>
- },
-
-Results::
-
- {
- "_id": <ID>,
- "case_name": <Reference to the test case>,
- "project_name": <Reference to project>,
- "pod_name": <Reference to POD where the test was executed>,
- "installer": <Installer Apex or Compass or Fuel or Joid>,
- "version": <master or Colorado or Brahmaputra>,
- "start_date": "YYYY-MM-DD HH:MM:SS",
- "stop_date": "YYYY-MM-DD HH:MM:SS",
- "build_tag": <such as "jenkins-functest-fuel-baremetal-daily-master-108">,
- "scenario": <Scenario on which the test was executed>,
- "criteria": <PASS or FAILED>,
- "trust_indicator": {
- "current": 0,
- "histories": []
- }
- }
-
-The API can described as follows. For detailed information, please go to
-
- http://testresults.opnfv.org/test/swagger/spec.html
-
- Authentication: opnfv/api@opnfv
-
-Version:
-
- +--------+--------------------------+-----------------------------------------+
- | Method | Path | Description |
- +========+==========================+=========================================+
- | GET | /versions | Get all supported API versions |
- +--------+--------------------------+-----------------------------------------+
-
-
-Pods:
-
- +--------+----------------------------+-----------------------------------------+
- | Method | Path | Description |
- +========+============================+=========================================+
- | GET | /api/v1/pods | Get the list of declared Labs (PODs) |
- +--------+----------------------------+-----------------------------------------+
- | POST | /api/v1/pods | Declare a new POD |
- | | | Content-Type: application/json |
- | | | { |
- | | | "name": "pod_foo", |
- | | | "mode": "metal", |
- | | | "role": "ci-pod", |
- | | | "details": "it is a ci pod" |
- | | | } |
- +--------+----------------------------+-----------------------------------------+
- | GET | /api/v1/pods/{pod_name} | Get a declared POD |
- +--------+----------------------------+-----------------------------------------+
-
-Projects:
-
- +--------+----------------------------+-----------------------------------------+
- | Method | Path | Description |
- +========+============================+=========================================+
- | GET | /api/v1/projects | Get the list of declared projects |
- +--------+----------------------------+-----------------------------------------+
- | POST | /api/v1/projects | Declare a new test project |
- | | | Content-Type: application/json |
- | | | { |
- | | | "name": "project_foo", |
- | | | "description": "whatever you want" |
- | | | } |
- +--------+----------------------------+-----------------------------------------+
- | DELETE | /api/v1/projects/{project} | Delete a test project |
- +--------+----------------------------+-----------------------------------------+
- | GET | /api/v1/projects/{project} | Get details on a {project} |
- | | | |
- +--------+----------------------------+-----------------------------------------+
- | PUT | /api/v1/projects/{project} | Update a test project |
- | | | |
- | | | Content-Type: application/json |
- | | | { |
- | | | <the field(s) you want to modify> |
- | | | } |
- +--------+----------------------------+-----------------------------------------+
-
-
-Testcases:
-
- +--------+----------------------------+-----------------------------------------+
- | Method | Path | Description |
- +========+============================+=========================================+
- | GET | /api/v1/projects/{project}/| Get the list of testcases of {project} |
- | | cases | |
- +--------+----------------------------+-----------------------------------------+
- | POST | /api/v1/projects/{project}/| Add a new test case to {project} |
- | | cases | Content-Type: application/json |
- | | | { |
- | | | "name": "case_foo", |
- | | | "description": "whatever you want" |
- | | | "url": "whatever you want" |
- | | | } |
- +--------+----------------------------+-----------------------------------------+
- | DELETE | /api/v1/projects/{project}/| Delete a test case |
- | | cases/{case} | |
- +--------+----------------------------+-----------------------------------------+
- | GET | /api/v1/projects/{project}/| Get a declared test case |
- | | cases/{case} | |
- +--------+----------------------------+-----------------------------------------+
- | PUT | /api/v1/projects/{project}?| Modify a test case of {project} |
- | | cases/{case} | |
- | | | Content-Type: application/json |
- | | | { |
- | | | <the field(s) you want to modify> |
- | | | } |
- +--------+----------------------------+-----------------------------------------+
-
-Results:
-
- +--------+----------------------------+------------------------------------------+
- | Method | Path | Description |
- +========+============================+==========================================+
- | GET | /api/v1/results | Get all the test results |
- +--------+----------------------------+------------------------------------------+
- | POST | /api/v1/results | Add a new test results |
- | | | Content-Type: application/json |
- | | | { |
- | | | "project_name": "project_foo", |
- | | | "scenario": "odl-l2", |
- | | | "stop_date": "2016-05-28T14:42:58.384Z", |
- | | | "trust_indicator": 0.5, |
- | | | "case_name": "vPing", |
- | | | "build_tag": "", |
- | | | "version": "Colorado", |
- | | | "pod_name": "pod_foo", |
- | | | "criteria": "PASS", |
- | | | "installer": "fuel", |
- | | | "start_date": "2016-05-28T14:41:58.384Z",|
- | | | "details": <your results> |
- | | | } |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of {case} |
- | | case={case} | |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of build_tag |
- | | build_tag={tag} | {tag}. |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get last {N} records of test results |
- | | last={N} | |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of scenario |
- | | scenario={scenario} | {scenario}. |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of trust_indicator |
- | | trust_indicator={ind} | {ind}. |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of last days |
- | | period={period} | {period}. |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of {project} |
- | | project={project} | |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of version |
- | | version={version} | {version}. |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of criteria |
- | | criteria={criteria} | {criteria}. |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | get the results on pod {pod} |
- | | pod={pod} | |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of installer {inst} |
- | | installer={inst} | |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the results according to combined |
- | | <query conditions> | query conditions supported above |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results/{result_id}| Get the test result by result_id |
- +--------+----------------------------+------------------------------------------+
-
-Scenarios:
-
- +--------+----------------------------+-----------------------------------------+
- | Method | Path | Description |
- +========+============================+=========================================+
- | GET | /api/v1/scenarios | Get the list of declared scenarios |
- +--------+----------------------------+-----------------------------------------+
- | POST | /api/v1/scenario | Declare a new scenario |
- +--------+----------------------------+-----------------------------------------+
- | GET | /api/v1/scenario? | Get a declared scenario |
- | | <query conditions> | |
- +--------+----------------------------+-----------------------------------------+
-
+ * Scenarios
The code of the API is hosted in the releng repository `[6]`_.
The static documentation of the API can be found at `[17]`_.
@@ -574,6 +362,7 @@ Please note that currently token authorization is implemented but is not yet ena
+---------------------+---------+---------+---------+---------+
| copper | X | | | X |
+---------------------+---------+---------+---------+---------+
+ src: colorado (see release note for the last matrix version)
All the testcases listed in the table are runnable on
os-odl_l2-nofeature scenarios.
@@ -736,8 +525,7 @@ Regex are standard regex. You can have a look at `[11]`_
You can also easily test your regex via an online regex checker such as `[12]`_.
Put your scenario in the TEST STRING window (e.g. os-odl_l3-ovs-ha), put
-your regex in the REGULAR EXPRESSION window, then you can test your rule
-.
+your regex in the REGULAR EXPRESSION window, then you can test your rule.
How to know which test I can run?
@@ -941,6 +729,233 @@ You can also reuse a python function defined in functest_utils.py::
return False
+Where can I find the documentation on the test API?
+===================================================
+
+http://artifacts.opnfv.org/releng/docs/testapi.html
+
+
+How to exclude Tempest case from default Tempest smoke suite?
+=============================================================
+
+Tempest default smoke suite deals with 165 test cases.
+Since Colorado the success criteria is 100%, i.e. if 1 test is failed the
+success criteria is not matched for the scenario.
+
+It is necessary to exclude some test cases that are expected to fail due to
+known upstream bugs (see release notes).
+
+A file has been created for such operation: https://git.opnfv.org/cgit/functest/tree/functest/opnfv_tests/openstack/tempest/custom_tests/blacklist.txt.
+
+It can be described as follows::
+
+ -
+ scenarios:
+ - os-odl_l2-bgpvpn-ha
+ - os-odl_l2-bgpvpn-noha
+ installers:
+ - fuel
+ - apex
+ tests:
+ - tempest.api.compute.servers.test_create_server.ServersTestJSON.test_list_servers
+ - tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_server_details
+ - tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_list_servers
+ - tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_server_details
+ - tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_reboot_server_hard
+ - tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops
+ - tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops
+ - tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern
+ - tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern
+
+Please note that each exclusion must be justified. the goal is not to exclude
+test cases because they do not pass. Several scenarios reached the 100% criteria.
+So it is expected in the patch submited to exclude the cases to indicate the
+reasons of the exclusion.
+
+
+How do I know the Functest status of a scenario?
+================================================
+
+A Functest automatic reporting page is generated daily.
+This page is dynamically created through a cron job and is based on the results
+stored in the Test DB.
+You can access this reporting page: http://testresults.opnfv.org/reporting
+
+See https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617 for details.
+
+
+I have tests, to which category should I declare them?
+======================================================
+
+CATEGORIES/TIERS description:
+
++----------------+-------------------------------------------------------------+
+| healthcheck | Simple OpenStack healtcheck tests case that validates the |
+| | basic operations in OpenStack |
++----------------+-------------------------------------------------------------+
+| Smoke | Set of smoke test cases/suites to validate the most common |
+| | OpenStack and SDN Controller operations |
++----------------+-------------------------------------------------------------+
+| Features | Test cases that validate a specific feature on top of OPNFV.|
+| | Those come from Feature projects and need a bit of support |
+| | for integration |
++----------------+-------------------------------------------------------------+
+| Components | Advanced Openstack tests: Full Tempest, Full Rally |
++----------------+-------------------------------------------------------------+
+| Performance | Out of Functest Scope |
++----------------+-------------------------------------------------------------+
+| VNF | Test cases related to deploy an open source VNF including |
+| | an orchestrator |
++----------------+-------------------------------------------------------------+
+
+The main ambiguity could be between features and VNF.
+In fact sometimes you have to spawn VMs to demonstrate the capabilities of the
+feature you introduced.
+We recommend to declare your test in the feature category.
+
+VNF category is really dedicated to test including:
+
+ * creation of resources
+ * deployement of an orchestrator/VNFM
+ * deployment of the VNF
+ * test of the VNFM
+ * free resources
+
+The goal is not to study a particular feature on the infrastructure but to have
+a whole end to end test of a VNF automatically deployed in CI.
+Moreover VNF are run in weekly jobs (one a week), feature tests are in daily
+jobs and use to get a scenario score.
+
+Where are the logs?
+===================
+
+Functest deals with internal and external testcases. Each testcase can generate
+logs.
+
+Since Colorado we introduce the possibility to push the logs to the artifact.
+A new script (https://git.opnfv.org/releng/tree/utils/push-test-logs.sh) has
+been created for CI.
+
+When called, and assuming that the POD is authorized to push the logs to
+artifacts, the script will push all the results or logs locally stored under
+/home/opnfv/functest/results/.
+
+If the POD is not connected to CI, logs are not pushed.
+But in both cases, logs are stored in /home/opnfv/functest/results in the
+container.
+Projects are encouraged to push their logs here.
+
+Since Colorado it is also easy for feature project to integrate this feature by
+adding the log file as output_file parameter when calling execute_command from
+functest_utils library
+
+ ret_val = functest_utils.execute_command(cmd, output_file=log_file)
+
+
+How does Functest deal with VNF onboarding?
+===========================================
+
+VNF onboarding has been introduced in Brahmaputra through the automation of a
+clearwater vIMS deployed thanks to cloudify orchestrator.
+
+This automation has been described at OpenStack summit Barcelona:
+https://youtu.be/Jr4nG74glmY
+
+The goal of Functest consists in testing OPNFV from a functional perspective:
+the NFVI and/or the features developed in OPNFV. Feature test suites are
+provided by the feature project. Functest just simplifies the integration of
+the suite into the CI and gives a consolidated view of the tests per scenario.
+
+Functest does not develop VNFs.
+
+Functest does not test any MANO stack.
+
+OPNFV projects dealing with VNF onboarding
+------------------------------------------
+
+Testing VNF is not the main goal however it gives interesting and realistic
+feedback on OPNFV as a Telco cloud.
+
+Onboarding VNF also allows to test a full stack: orchestrator + VNF.
+
+Functest is VNF and MANO stack agnostic.
+
+An internship has been initiated to reference the Open Source VNF: Intern
+Project Open Source VNF catalog
+
+New projects dealing with orchestrators or VNFs are candidate for Danube.
+
+The 2 projects dealing with orchestration are:
+
+ * orchestra (Openbaton)
+ * opera (Open-O)
+
+The Models project address various goals for promoting availability and
+convergence of information and/or data models related to NFV service/VNF
+management, as being defined in standards (SDOs) and as developed in open
+source projects.
+
+Functest VNF onboarding
+-----------------------
+
+In order to simplify VNF onboarding a new abstraction class has been developed
+in Functest.
+
+This class is based on vnf_base and can be described as follow:
+
+ +------------+ +--------------+
+ | test_base |------------>| vnf_base |
+ +------------+ +--------------+
+ |_ prepare
+ |_ deploy_orchestrator (optional)
+ |_ deploy_vnf
+ |_ test_vnf
+ |_ clean
+
+
+Several methods are declared in vnf_base:
+
+ * prepare
+ * deploy_orchestrator
+ * deploy_vnf
+ * test_vnf
+ * clean
+
+deploy_vnf and test_vnf are mandatory.
+
+prepare will create a user and a project.
+
+How to declare your orchestrator/VNF?
+-------------------------------------
+1) test declaration
+
+You must declare your testcase in the file <Functest repo>/functest/ci/testcases.yaml
+
+2) configuration
+
+You can precise some configuration parameters in config_functest.yaml
+
+3) implement your test
+
+Create your own VnfOnboarding file
+
+you must create your entry point through a python clase as referenced in the
+configuration file
+
+e.g. aaa => creation of the file <Functest repo>/functest/opnfv_tests/vnf/aaa/aaa.py
+
+the class shall inherit vnf_base.
+You must implement the methods deploy_vnf() and test_vnf() and may implement
+deploy_orchestrator()
+
+you can call the code from your repo (but need to add the repo in Functest if
+it is not the case)
+
+4) success criteria
+
+So far we considered the test as PASS if vnf_deploy and test_vnf is PASS
+(see example in aaa).
+
==========
References
==========
@@ -989,4 +1004,4 @@ _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_varia
_`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
-_`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml
+_`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/functest/ci/config_functest.yaml