diff options
Diffstat (limited to 'docs')
35 files changed, 2385 insertions, 806 deletions
diff --git a/docs/release/release-notes/apex.rst b/docs/release/release-notes/apex.rst new file mode 100644 index 000000000..e8b90bee5 --- /dev/null +++ b/docs/release/release-notes/apex.rst @@ -0,0 +1,44 @@ +apex +==== + +os-nosdn-nofeature-ha +--------------------- + +==================== =================== ========== ======== ========= +testcase date pod_name result jira +==================== =================== ========== ======== ========= +connection_check 2018-04-07 04:52:18 lf-pod1 PASS +api_check 2018-04-07 05:05:30 lf-pod1 PASS +snaps_health_check 2018-04-07 05:06:12 lf-pod1 PASS +vping_ssh 2018-04-07 05:07:49 lf-pod1 PASS +vping_userdata 2018-04-07 05:08:55 lf-pod1 PASS +tempest_smoke_serial 2018-04-07 05:24:59 lf-pod1 PASS +rally_sanity 2018-04-07 05:46:43 lf-pod1 FAIL APEX-564 +refstack_defcore 2018-04-07 05:49:54 lf-pod1 PASS +patrole 2018-04-07 05:53:10 lf-pod1 PASS +snaps_smoke 2018-04-07 06:50:35 lf-pod1 FAIL SNAPS-283 +neutron_trunk 2018-04-07 06:53:13 lf-pod1 PASS +cloudify_ims 2018-04-07 07:19:50 lf-pod1 PASS +vyos_vrouter 2018-04-07 07:40:53 lf-pod1 PASS +juju_epc 2018-04-07 08:06:00 lf-pod1 FAIL APEX-570 +==================== =================== ========== ======== ========= + +os-odl-nofeature-ha +------------------- + +==================== =================== ========== ======== ========= +testcase date pod_name result jira +==================== =================== ========== ======== ========= +connection_check 2018-04-07 13:18:17 lf-pod1 PASS +api_check 2018-04-07 13:30:11 lf-pod1 PASS +snaps_health_check 2018-04-07 13:31:46 lf-pod1 PASS +vping_ssh 2018-04-07 13:32:46 lf-pod1 PASS +vping_userdata 2018-04-07 13:33:40 lf-pod1 PASS +tempest_smoke_serial 2018-04-07 13:53:58 lf-pod1 FAIL +rally_sanity 2018-04-07 14:12:42 lf-pod1 FAIL +refstack_defcore 2018-04-07 14:21:20 lf-pod1 FAIL +patrole 2018-04-07 14:24:48 lf-pod1 FAIL +snaps_smoke 2018-04-07 15:05:15 lf-pod1 FAIL SNAPS-283 +odl 2018-04-07 14:25:10 lf-pod1 PASS +neutron_trunk 2018-04-07 15:07:53 lf-pod1 FAIL +==================== =================== ========== ======== ========= diff --git a/docs/release/release-notes/compass.rst b/docs/release/release-notes/compass.rst new file mode 100644 index 000000000..a1481fa37 --- /dev/null +++ b/docs/release/release-notes/compass.rst @@ -0,0 +1,54 @@ +compass +======= + +os-nosdn-nofeature-ha +--------------------- + +==================== =================== =========== ======== =========== +testcase date pod_name result jira +==================== =================== =========== ======== =========== +connection_check 2018-04-07 11:59:03 huawei-pod1 PASS +api_check 2018-04-07 12:09:28 huawei-pod1 PASS +snaps_health_check 2018-04-07 12:11:01 huawei-pod1 PASS +vping_ssh 2018-04-07 12:12:45 huawei-pod1 PASS +vping_userdata 2018-04-07 12:14:32 huawei-pod1 PASS +tempest_smoke_serial 2018-04-07 12:31:35 huawei-pod1 FAIL COMPASS-588 +rally_sanity 2018-04-07 13:00:49 huawei-pod1 PASS +refstack_defcore 2018-04-07 13:06:36 huawei-pod1 PASS +patrole 2018-04-07 13:10:05 huawei-pod1 PASS +snaps_smoke 2018-04-07 14:02:28 huawei-pod1 FAIL SNAPS-283 +neutron_trunk 2018-04-07 14:04:43 huawei-pod1 FAIL +cloudify_ims 2018-04-07 14:32:36 huawei-pod1 PASS +vyos_vrouter 2018-04-07 14:56:46 huawei-pod1 PASS +juju_epc 2018-04-07 15:52:35 huawei-pod1 PASS +==================== =================== =========== ======== =========== + +os-odl_l3-nofeature-ha +---------------------- + +==================== =================== =========== ======== ========= +testcase date pod_name result jira +==================== =================== =========== ======== ========= +connection_check 2018-04-04 08:53:18 huawei-pod1 PASS +api_check 2018-04-04 09:03:23 huawei-pod1 PASS +snaps_health_check 2018-04-04 09:03:59 huawei-pod1 PASS +vping_ssh 2016-07-23 20:21:55 huawei-pod1 PASS +vping_userdata 2018-04-04 09:04:43 huawei-pod1 PASS +tempest_smoke_serial 2018-04-04 09:21:20 huawei-pod1 FAIL +rally_sanity 2018-04-04 09:50:09 huawei-pod1 PASS +refstack_defcore 2018-04-04 09:52:32 huawei-pod1 PASS +patrole 2018-04-04 09:56:08 huawei-pod1 PASS +snaps_smoke 2018-04-04 10:48:07 huawei-pod1 FAIL SNAPS-283 +odl 2018-04-04 09:56:30 huawei-pod1 PASS +neutron_trunk 2018-04-04 10:51:10 huawei-pod1 FAIL +==================== =================== =========== ======== ========= + +k8-nosdn-nofeature-ha +--------------------- + +=============== =================== =========== ======== ====== +testcase date pod_name result jira +=============== =================== =========== ======== ====== +k8s_smoke 2018-04-07 05:46:58 huawei-pod1 PASS +k8s_conformance 2018-04-07 06:38:26 huawei-pod1 PASS +=============== =================== =========== ======== ====== diff --git a/docs/release/release-notes/conf.py b/docs/release/release-notes/conf.py new file mode 100644 index 000000000..77a71a37d --- /dev/null +++ b/docs/release/release-notes/conf.py @@ -0,0 +1,184 @@ +# -*- coding: utf-8 -*- +# +# Functest Release Notes documentation build configuration file, created by +# sphinx-quickstart on Tue Apr 3 03:40:39 2018. +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) +import sphinx_opnfv_theme + + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +# +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'Functest Release Notes' +copyright = u'2018, Cédric Ollivier <cedric.ollivier@orange.com>' +author = u'Cédric Ollivier <cedric.ollivier@orange.com>' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = u'master' +# The full version, including alpha/beta/rc tags. +release = u'master' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This patterns also effect to html_static_path and html_extra_path +exclude_patterns = [] + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = False + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'opnfv' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} +html_theme_options = { + 'bootswatch_theme': 'journal', + 'navbar_sidebarrel': False, + 'navbar_title': '', +} + +# Add any paths that contain custom themes here, relative to this directory. +# html_theme_path = [] +html_theme_path = sphinx_opnfv_theme.get_html_theme_path() + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +# html_static_path = [] + +# Custom sidebar templates, must be a dictionary that maps document names +# to template names. +# +# This is required for the alabaster theme +# refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars +html_sidebars = { + '**': [ + 'relations.html', # needs 'show_related': True theme option to display + 'searchbox.html', + ] +} + + +# -- Options for HTMLHelp output ------------------------------------------ + +# Output file base name for HTML help builder. +htmlhelp_basename = 'FunctestReleaseNotesdoc' + + +# -- Options for LaTeX output --------------------------------------------- + +latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # + # 'preamble': '', + + # Latex figure (float) alignment + # + # 'figure_align': 'htbp', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, 'FunctestReleaseNotes.tex', + u'Functest Release Notes Documentation', + u'Cédric Ollivier \\textless{}cedric.ollivier@orange.com\\textgreater{}', + 'manual'), +] + + +# -- Options for manual page output --------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, + 'functestreleasenotes', + u'Functest Release Notes Documentation', + [author], + 1) +] + + +# -- Options for Texinfo output ------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, + 'FunctestReleaseNotes', + u'Functest Release Notes Documentation', + author, + 'FunctestReleaseNotes', + 'One line description of project.', + 'Miscellaneous'), +] diff --git a/docs/release/release-notes/daisy.rst b/docs/release/release-notes/daisy.rst new file mode 100644 index 000000000..0039579f5 --- /dev/null +++ b/docs/release/release-notes/daisy.rst @@ -0,0 +1,44 @@ +daisy +===== + +os-nosdn-nofeature-ha +--------------------- + +==================== =================== ========== ======== ========= +testcase date pod_name result jira +==================== =================== ========== ======== ========= +connection_check 2018-04-03 13:41:18 zte-pod2 PASS +api_check 2018-04-03 13:52:18 zte-pod2 PASS +snaps_health_check 2018-04-03 13:53:02 zte-pod2 PASS +vping_ssh 2018-04-03 13:55:21 zte-pod2 PASS +vping_userdata 2018-04-03 13:56:22 zte-pod2 PASS +tempest_smoke_serial 2018-04-03 14:12:18 zte-pod2 PASS +rally_sanity 2018-04-03 14:39:43 zte-pod2 PASS +refstack_defcore 2018-04-03 14:44:51 zte-pod2 PASS +patrole 2018-04-03 14:49:10 zte-pod2 PASS +snaps_smoke 2018-04-03 15:34:51 zte-pod2 FAIL SNAPS-283 +neutron_trunk zte-pod2 +cloudify_ims 2018-04-03 15:47:44 zte-pod2 FAIL +vyos_vrouter 2018-04-03 15:58:55 zte-pod2 FAIL +juju_epc 2018-04-03 16:49:31 zte-pod2 PASS +==================== =================== ========== ======== ========= + +os-odl-nofeature-ha +------------------- + +==================== =================== ========== ======== ========= +testcase date pod_name result jira +==================== =================== ========== ======== ========= +connection_check 2018-04-05 19:46:11 zte-pod2 PASS +api_check 2018-04-05 19:57:00 zte-pod2 PASS +snaps_health_check 2018-04-05 19:57:39 zte-pod2 PASS +vping_ssh 2018-04-05 20:00:05 zte-pod2 PASS +vping_userdata 2018-04-05 20:01:04 zte-pod2 PASS +tempest_smoke_serial 2018-04-05 20:17:48 zte-pod2 PASS +rally_sanity 2018-04-05 20:46:11 zte-pod2 PASS +refstack_defcore 2018-04-05 20:51:36 zte-pod2 PASS +patrole 2018-04-05 20:57:27 zte-pod2 PASS +snaps_smoke 2018-04-05 21:51:34 zte-pod2 PASS SNAPS-283 +odl 2018-04-05 20:57:48 zte-pod2 PASS +neutron_trunk zte-pod2 +==================== =================== ========== ======== ========= diff --git a/docs/release/release-notes/fuel_amd64.rst b/docs/release/release-notes/fuel_amd64.rst new file mode 100644 index 000000000..39f281c42 --- /dev/null +++ b/docs/release/release-notes/fuel_amd64.rst @@ -0,0 +1,44 @@ +fuel(amd64) +=========== + +os-nosdn-nofeature-ha +--------------------- + +==================== =================== ========== ======== ======== +testcase date pod_name result jira +==================== =================== ========== ======== ======== +connection_check 2018-04-08 22:25:51 lf-pod2 PASS +api_check 2018-04-08 22:37:50 lf-pod2 PASS +snaps_health_check 2018-04-08 22:38:39 lf-pod2 PASS +vping_ssh 2018-04-08 22:39:46 lf-pod2 PASS +vping_userdata 2018-04-08 22:41:03 lf-pod2 PASS +tempest_smoke_serial 2018-04-08 23:00:26 lf-pod2 PASS +rally_sanity 2018-04-08 23:31:15 lf-pod2 PASS +refstack_defcore 2018-04-08 23:35:19 lf-pod2 PASS +patrole 2018-04-08 23:40:12 lf-pod2 PASS +snaps_smoke 2018-04-09 00:46:28 lf-pod2 FAIL FUEL-356 +neutron_trunk 2018-04-09 00:49:24 lf-pod2 PASS +cloudify_ims 2018-04-09 01:07:03 lf-pod2 FAIL +vyos_vrouter 2018-04-09 01:24:45 lf-pod2 FAIL +juju_epc 2018-04-09 01:33:37 lf-pod2 FAIL +==================== =================== ========== ======== ======== + +os-odl-nofeature-ha +------------------- + +==================== =================== ========== ======== ========= +testcase date pod_name result jira +==================== =================== ========== ======== ========= +connection_check 2018-04-06 23:17:26 lf-pod2 PASS +api_check 2018-04-06 23:28:58 lf-pod2 PASS +snaps_health_check 2018-04-06 23:29:48 lf-pod2 PASS +vping_ssh 2018-04-06 23:31:06 lf-pod2 PASS +vping_userdata 2018-04-06 23:32:13 lf-pod2 PASS +tempest_smoke_serial 2018-04-06 23:51:25 lf-pod2 FAIL +rally_sanity 2018-04-07 00:24:03 lf-pod2 FAIL +refstack_defcore 2018-04-07 00:30:34 lf-pod2 PASS +patrole 2018-04-07 00:34:31 lf-pod2 PASS +snaps_smoke 2018-04-07 01:44:38 lf-pod2 FAIL SNAPS-283 +odl 2018-04-07 00:34:42 lf-pod2 FAIL +neutron_trunk 2018-04-07 01:47:59 lf-pod2 FAIL +==================== =================== ========== ======== ========= diff --git a/docs/release/release-notes/fuel_arm64.rst b/docs/release/release-notes/fuel_arm64.rst new file mode 100644 index 000000000..f948dddbc --- /dev/null +++ b/docs/release/release-notes/fuel_arm64.rst @@ -0,0 +1,21 @@ +fuel(arm64) +=========== + +os-nosdn-nofeature-ha +--------------------- + +==================== =================== ========== ======== ========= +testcase date pod_name result jira +==================== =================== ========== ======== ========= +connection_check 2018-04-05 19:46:24 arm-pod6 PASS +api_check 2018-04-05 20:23:40 arm-pod6 PASS +snaps_health_check 2018-04-05 20:25:54 arm-pod6 PASS +vping_ssh 2018-04-05 20:31:11 arm-pod6 PASS +vping_userdata 2018-04-05 20:35:28 arm-pod6 PASS +tempest_smoke_serial 2018-04-05 21:47:34 arm-pod6 PASS +rally_sanity 2018-04-05 23:07:52 arm-pod6 PASS +refstack_defcore 2018-04-05 23:32:04 arm-pod6 FAIL +patrole 2018-04-05 23:47:59 arm-pod6 PASS +snaps_smoke 2018-04-06 02:56:45 arm-pod6 FAIL SNAPS-283 +neutron_trunk 2018-04-06 03:10:36 arm-pod6 PASS +==================== =================== ========== ======== ========= diff --git a/docs/release/release-notes/functest-gating.rst b/docs/release/release-notes/functest-gating.rst new file mode 100644 index 000000000..0aac41b33 --- /dev/null +++ b/docs/release/release-notes/functest-gating.rst @@ -0,0 +1,27 @@ +.. SPDX-License-Identifier: CC-BY-4.0 + +Release Gating +============== + +Thanks to the analysis of the offical OPNFV results and local tests (see +`Orange ONAP Openlab`_), Functest is trustable for verifying all OPNFV Fraser +installers and more generally classical OpenStack Pike and Kubernetes +deployments. + +It should be noted that: + + * any failed result highlights side effects for end users + * OpenStack scenarios which don't pass tempest_smoke_serial break the + upstream rules (and then decrease the overall quality) as all patches are + tested vs tempest in OpenStack gating + +.. _`Orange ONAP Openlab`: https://wiki.opnfv.org/pages/viewpage.action?pageId=13211751 + +.. toctree:: + + apex.rst + compass.rst + daisy.rst + fuel_amd64.rst + fuel_arm64.rst + joid.rst diff --git a/docs/release/release-notes/functest-release.rst b/docs/release/release-notes/functest-release.rst index 78ec8f227..66ba1916c 100644 --- a/docs/release/release-notes/functest-release.rst +++ b/docs/release/release-notes/functest-release.rst @@ -1,33 +1,25 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 -======= -License -======= - -OPNFV Euphrates release note for Functest Docs -are licensed under a Creative Commons Attribution 4.0 International License. -You should have received a copy of the license along with this. -If not, see <http://creativecommons.org/licenses/by/4.0/>. - -============================================= -OPNFV Euphrates 5.1 release note for Functest -============================================= +====================================== +OPNFV master release note for Functest +====================================== Abstract ======== This document contains the release notes of the Functest project. - -OPNFV Euphrates Release -====================== +OPNFV master Release +==================== Functest deals with functional testing of the OPNFV solution. It includes test cases developed within the project, test cases developed in -other OPNFV projects and it also intgrates test cases from other upstream +other OPNFV projects and it also integrates test cases from other upstream communities. +OpenStack +--------- + The internal test cases are: * connection_check @@ -36,48 +28,48 @@ The internal test cases are: * vping_ssh * vping_userdata * tempest_smoke_serial - * refstack_defcore - * snaps_smoke * rally_sanity + * refstack_defcore + * patrole * odl + * odl-netvirt + * snaps_smoke + * neutron_trunk * tempest_full_parallel * rally_full * cloudify_ims * vyos_vrouter + * juju_epc The OPNFV projects integrated into Functest framework for automation are: - * barometer - * bgpvpn * doctor - * domino - * fds + * bgpvpn * odl-sfc - * odl-netvirt + * barometer + * fds * parser - * promise - * orchestra_openims - * orchestra_clearwaterims +Kubernetes +---------- + +The internal test cases are: + + * k8s_smoke + * k8s_conformance + +The OPNFV projects integrated into Functest framework for automation are: + + * stor4nfv + * clover Release Data ============ +--------------------------------------+--------------------------------------+ | **Project** | functest | -| | | -+--------------------------------------+--------------------------------------+ -| **Repo/tag** | opnfv-5.1.0 | -| | | -+--------------------------------------+--------------------------------------+ -| **Release designation** | Euphrates 5.1 release | -| | | -+--------------------------------------+--------------------------------------+ -| **Release date** | December 15th 2017 | -| | | +--------------------------------------+--------------------------------------+ -| **Purpose of the delivery** | Euphrates second release | -| | | +| **Repository branch** | master | +--------------------------------------+--------------------------------------+ Deliverables @@ -86,7 +78,7 @@ Deliverables Software -------- - Functest Docker images: + Functest Docker images (OpenStack): * https://hub.docker.com/r/opnfv/functest-healthcheck * https://hub.docker.com/r/opnfv/functest-smoke @@ -94,156 +86,80 @@ Software * https://hub.docker.com/r/opnfv/functest-components * https://hub.docker.com/r/opnfv/functest-vnf * https://hub.docker.com/r/opnfv/functest-parser - * https://hub.docker.com/r/opnfv/functest-restapi - TestAPI Docker image: + Functest Docker images (Kubernetes): - * https://hub.docker.com/r/opnfv/testapi + * https://hub.docker.com/r/opnfv/functest-kubernetes-healthcheck + * https://hub.docker.com/r/opnfv/functest-kubernetes-smoke + * https://hub.docker.com/r/opnfv/functest-kubernetes-features -Docker tag for Euphrates 5.1 release: opnfv-5.1.0 -Docker tag for Euphrates with latest bugfixes: euphrates +Docker tag for master: latest Documents --------- - - Installation/configuration guide: http://docs.opnfv.org/en/stable-euphrates/submodules/functest/docs/testing/user/configguide/index.html - - - User Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/functest/docs/testing/user/userguide/index.html - - - Developer Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/functest/docs/testing/developer/devguide/index.html - - - API Docs: http://artifacts.opnfv.org/functest/docs/index.html - - - Functest Framework presentation: http://testresults.opnfv.org/functest/framework/index.html - + * Config Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html + * User Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html + * Developer Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/developer/devguide/index.html + * API Docs: http://functest.readthedocs.io/en/latest/ Version change ============== -Functest now delivers light-weigth Docker images based on Alpine 3.7. The test cases are grouped into several categories -or tiers and must be run from the corresponding container. For example, to run the test case healthcheck, the image -opnfv/functest-healthcheck shall be used. The tiers and the tests within them are explained in detail in the User Guide. - -The former Ubuntu image is not longer maintained. - -The Parser test case has its own dedicated Docker image since it requires libraries released for OpenStack Pike and -Euphrates is based on Ocata. - -The Docker images do not contain OS images (Cirros, Ubuntu, Centos, ..) anymore. A script has been created under the -ci directory (download_images.sh) which contains all the needed images for all the tests. This file can be modified by -the user since not all the images might be used. It must be executed before starting Functest and attach the needed -images as a Docker volume. See Configuration Guide for more information. - -The requirements have been split into 3 files: - * requirements.txt : lists all abstract dependencies of the OPNFV packages - * test-requirements.txt : lists all abstract dependencies required by Functest Unit Tests - * upper-constraints.txt : lists all concrete upstream dependencies required by Functest Docker container - -OPNFV (test-)requirements.txt have been updated according to stable/ocata global-requirements.txt. -Functest uses (and completes) stable/ocata upper-constraints.txt in Dockerfiles and tox configuration. -The project relies on pbr, which injects requirements into the install_requires, tests_require and/or dependency_links -arguments to setup. It also supports conditional dependencies which can be added to the requirements (e.g. dnspython>=1.14.0;python_version=='2.7') - -The way to manage logging has been centralized to a configuration file (logging.ini) which might be modified by the user. -By default, the output of executing the test cases is redirected to log files and is not displayed on the console, only result -messages and summary tables are displayed. - -The framework has been refactored and all the test cases inherit from a core class TestCase. For Feature projects who develop -test cases, 2 sub-classes have been created: - - Feature: it implements all the needed functions and the developer must only overwrite the method "execute" (e.g. Barometer) - - BashFeature: it is used if the third party test case is a shell script. This way, the execution command must be specified in - testcases.yaml as the argument (e.g. Domino, Doctor) - -An internal REST API has been introduced in Euphrates. The goal is to trigger Functest operations through an API in addition of the CLI. -This could be considered as a first step towards a pseudo micro services approach where the different test projects could expose and -consume APIs to the other test projects. - -Euphrates 5.1 improvements -========================== - -* Alpine images are now supported for ARM (arm64). -* Added Vyos_router test case. -* Updated of Rally 0.9.1 and fixed some bugs in cinder scenarios. -* Patch to allow building containers from a gerrit change. -* Selection of a subset of SNAPS test cases. -* Reorder VNF test cases and adjust timeouts in VNFs. - - - -Euphrates 5.1 known restrictions/issues -======================================= -+--------------+-----------+----------------------------------------------+ -| Installer | Scenario | Issue | -+==============+===========+==============================================+ -| fuel@aarch64 | any | VNF tier not supported yet. | -+--------------+-----------+----------------------------------------------+ -| | | The test cases belonging to the VNF tier | -| any | any | have been only tested on os-nosdn-nofeature | -| | | scenarios and baremetal deployments. | -+--------------+-----------+----------------------------------------------+ -| Joid | k8 | Functest does not offer test suites for | -| Compass | | Kubernetes scenarios yet. | -+--------------+-----------+----------------------------------------------+ - - -Test and installer/scenario dependencies -======================================== - -It is not always possible to run all the test cases on all the scenarios. -The scenario dependencies (installer or scenario) are detailed -in the different testcases.yaml for each tier: - - * https://git.opnfv.org/functest/tree/docker/healthcheck/testcases.yaml?h=stable/euphrates - * https://git.opnfv.org/functest/tree/docker/smoke/testcases.yaml?h=stable/euphrates - * https://git.opnfv.org/functest/tree/docker/features/testcases.yaml?h=stable/euphrates - * https://git.opnfv.org/functest/tree/docker/components/testcases.yaml?h=stable/euphrates - * https://git.opnfv.org/functest/tree/docker/vnf/testcases.yaml?h=stable/euphrates - * https://git.opnfv.org/functest/tree/docker/parser/testcases.yaml?h=stable/euphrates - - -Test results -============ - -The Functest scenario status on December 15, 2017 can be seen on -http://testresults.opnfv.org/functest/euphrates/ - -Test logs are available in: - - - test results logs from CI: http://artifacts.opnfv.org (within different directories 'logs_functest_X') - - - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/ - - - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/ - - - -Open JIRA tickets -================= - -+------------------+-----------------------------------------------+ -| JIRA | Description | -+==================+===============================================+ -| | | -| | | -+------------------+-----------------------------------------------+ - -All the tickets that are not blocking have been fixed or postponed -the next release. - +New test cases +-------------- + + * neutron_trunk + * patrole + * juju_epc + * k8s_smoke + * k8s_conformance + * stor4nfv + * clover + +Key changes +----------- + + * update test cases and containers to `OpenStack Pike`_ + * move the framework into a separate project: Xtesting_ + * ease running all containers thanks to default values + * clean interfaces with OPNFV Installers and Features + * rewrite all vnfs to allow multiple tests in parallel + * fully support non-default region names and Keystone v3 domains + * refactor all tempest-based test cases (e.g. refstack_defcore) + * remove obsolete OpenStack and Functest utils + * verify all changes via doc8 and yamllint too + * generate reports for all tempest-based test cases + +.. _`OpenStack Pike`: https://raw.githubusercontent.com/openstack/requirements/stable/pike/upper-constraints.txt +.. _Xtesting: http://xtesting.readthedocs.io/en/latest/ + +Key benefits +------------ + + * the enduser can run all tests by setting only one input (EXTERNAL_NETWORK) + * the developer can only work on the test suites without diving into CI/CD + integration + * both OpenStack and Kubernetes deployments can be verified + * Functest test cases are trustable as they meet the best coding rules (unit + tests, coverage, linters, etc.) + * Functest can be reused in other projects (e.g. + `ONS: Re-using OPNFV framework tests for LFN projects`_) + +.. _`ONS: Re-using OPNFV framework tests for LFN projects`: https://wiki.lfnetworking.org/display/LN/LFN+Developer+Forum+Schedule?preview=/328197/328329/ONS-OPNFV%20framework%20tests%20for%20LFN%20projects.pdf + +Code quality +------------ + + * pylint: ~9.5/10 + * code coverage: ~70% Useful links ============ - - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing - - - wiki Functest Euphrates page: https://wiki.opnfv.org/display/functest/5.+Euphrates - - - Functest repo: https://git.opnfv.org/cgit/functest - - - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/ - - - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611 - - - Functest IRC chan: #opnfv-functest - - - Reporting page: http://testresults.opnfv.org/reporting/euphrates.html + * wiki project page: https://wiki.opnfv.org/opnfv_functional_testing + * Functest git repository: https://git.opnfv.org/cgit/functest + * Functest CI dashboard: https://build.opnfv.org/ci/view/functest/ + * JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611 + * Functest IRC channel: #opnfv-functest + * Reporting page: http://testresults.opnfv.org/reporting/master/functest/functest.html diff --git a/docs/release/release-notes/index.rst b/docs/release/release-notes/index.rst index 411e09bdf..5ded938da 100644 --- a/docs/release/release-notes/index.rst +++ b/docs/release/release-notes/index.rst @@ -1,4 +1,3 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 .. _functest-releasenotes: @@ -12,9 +11,6 @@ Functest Release Notes :maxdepth: 4 functest-release.rst - -Revision: _sha1_ - -:Author: Jose Lausuch (jalausuch@suse.com) + functest-gating.rst Build date: |today| diff --git a/docs/release/release-notes/joid.rst b/docs/release/release-notes/joid.rst new file mode 100644 index 000000000..02c574187 --- /dev/null +++ b/docs/release/release-notes/joid.rst @@ -0,0 +1,24 @@ +joid +==== + +os-nosdn-nofeature-ha +--------------------- + +==================== =================== ============ ======== ========= +testcase date pod_name result jira +==================== =================== ============ ======== ========= +connection_check 2018-03-27 21:43:25 huawei-pod12 PASS +api_check 2018-03-27 21:51:36 huawei-pod12 PASS +snaps_health_check 2018-03-27 21:52:05 huawei-pod12 PASS +vping_ssh 2018-03-27 21:53:37 huawei-pod12 PASS +vping_userdata 2018-03-27 21:55:08 huawei-pod12 PASS +tempest_smoke_serial 2018-03-27 22:10:05 huawei-pod12 FAIL +rally_sanity 2018-03-27 22:13:28 huawei-pod12 FAIL +refstack_defcore 2018-03-27 22:17:14 huawei-pod12 FAIL +patrole 2018-03-27 22:20:18 huawei-pod12 FAIL +snaps_smoke 2018-03-27 23:03:54 huawei-pod12 FAIL SNAPS-283 +neutron_trunk huawei-pod12 +cloudify_ims 2018-03-27 23:06:48 huawei-pod12 FAIL +vyos_vrouter 2018-03-27 23:15:54 huawei-pod12 FAIL +juju_epc 2018-03-27 23:16:20 huawei-pod12 FAIL +==================== =================== ============ ======== ========= diff --git a/docs/testing/developer/devguide/conf.py b/docs/testing/developer/devguide/conf.py new file mode 100644 index 000000000..1b56ee8e8 --- /dev/null +++ b/docs/testing/developer/devguide/conf.py @@ -0,0 +1,185 @@ +# -*- coding: utf-8 -*- +# +# Functest Developer Guide documentation build configuration file, created by +# sphinx-quickstart on Wed Apr 4 05:36:49 2018. +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) +import sphinx_opnfv_theme + + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +# +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'Functest Developer Guide' +copyright = u'2018, Cédric Ollivier <cedric.ollivier@orange.com>' +author = u'Cédric Ollivier <cedric.ollivier@orange.com>' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = u'master' +# The full version, including alpha/beta/rc tags. +release = u'master' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This patterns also effect to html_static_path and html_extra_path +exclude_patterns = [] + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = False + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'opnfv' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} +html_theme_options = { + 'bootswatch_theme': 'journal', + 'navbar_sidebarrel': False, + 'navbar_title': '', +} + +# Add any paths that contain custom themes here, relative to this directory. +# html_theme_path = [] +html_theme_path = sphinx_opnfv_theme.get_html_theme_path() + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +# html_static_path = [] + +# Custom sidebar templates, must be a dictionary that maps document names +# to template names. +# +# This is required for the alabaster theme +# refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars +html_sidebars = { + '**': [ + 'relations.html', # needs 'show_related': True theme option to display + 'searchbox.html', + ] +} + + +# -- Options for HTMLHelp output ------------------------------------------ + +# Output file base name for HTML help builder. +htmlhelp_basename = 'FunctestDeveloperGuidedoc' + + +# -- Options for LaTeX output --------------------------------------------- + +latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # + # 'preamble': '', + + # Latex figure (float) alignment + # + # 'figure_align': 'htbp', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, + 'FunctestDeveloperGuide.tex', + u'Functest Developer Guide Documentation', + u'Cédric Ollivier \\textless{}cedric.ollivier@orange.com\\textgreater{}', + 'manual'), +] + + +# -- Options for manual page output --------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, + 'functestdeveloperguide', + u'Functest Developer Guide Documentation', + [author], + 1) +] + + +# -- Options for Texinfo output ------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, + 'FunctestDeveloperGuide', + u'Functest Developer Guide Documentation', + author, + 'FunctestDeveloperGuide', + 'One line description of project.', + 'Miscellaneous'), +] diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst index 19a1bdcb8..1812a4cb5 100644 --- a/docs/testing/developer/devguide/index.rst +++ b/docs/testing/developer/devguide/index.rst @@ -1,4 +1,3 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 ************************ @@ -109,11 +108,12 @@ The main internal test cases are in the opnfv_tests subfolder of the repository, the internal test cases can be grouped by domain: * sdn: odl, odl_netvirt, odl_fds - * openstack: api_check, connection_check, snaps_health_check, vping_ssh, vping_userdata, tempest_*, rally_* + * openstack: api_check, connection_check, snaps_health_check, vping_ssh, + vping_userdata, tempest_*, rally_* * vnf: cloudify_ims -If you want to create a new test case you will have to create a new folder under -the testcases directory (See next section for details). +If you want to create a new test case you will have to create a new folder +under the testcases directory (See next section for details). Functest external test cases ============================ @@ -157,10 +157,10 @@ Functest framework Functest is a framework. -Historically Functest is released as a docker file, including tools, scripts and -a CLI to prepare the environment and run tests. -It simplifies the integration of external test suites in CI pipeline and provide -commodity tools to collect and display results. +Historically Functest is released as a docker file, including tools, scripts +and a CLI to prepare the environment and run tests. +It simplifies the integration of external test suites in CI pipeline and +provide commodity tools to collect and display results. Since Colorado, test categories also known as **tiers** have been created to group similar tests, provide consistent sub-lists and at the end optimize @@ -250,8 +250,8 @@ In order to simplify the creation of test cases, Functest develops also some functions that are used by internal test cases. Several features are supported such as logger, configuration management and Openstack capabilities (tacker,..). -These functions can be found under <repo>/functest/utils and can be described as -follows:: +These functions can be found under <repo>/functest/utils and can be described +as follows:: functest/utils/ |-- config.py @@ -262,8 +262,8 @@ follows:: |-- openstack_tacker.py `-- openstack_utils.py -It is recommended to use the SNAPS-OO library for deploying OpenStack instances. -SNAPS `[4]`_ is an OPNFV project providing OpenStack utils. +It is recommended to use the SNAPS-OO library for deploying OpenStack +instances. SNAPS `[4]`_ is an OPNFV project providing OpenStack utils. TestAPI diff --git a/docs/testing/developer/internship/security_group/conf.py b/docs/testing/developer/internship/security_group/conf.py new file mode 100644 index 000000000..f883ab3d0 --- /dev/null +++ b/docs/testing/developer/internship/security_group/conf.py @@ -0,0 +1,185 @@ +# -*- coding: utf-8 -*- +# +# Functest Security group test cases documentation build configuration file, +# created by sphinx-quickstart on Wed Apr 4 05:56:19 2018. +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) +import sphinx_opnfv_theme + + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +# +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'Functest Security group test cases' +copyright = u'2018, Cédric Ollivier <cedric.ollivier@orange.com>' +author = u'Cédric Ollivier <cedric.ollivier@orange.com>' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = u'master' +# The full version, including alpha/beta/rc tags. +release = u'master' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This patterns also effect to html_static_path and html_extra_path +exclude_patterns = [] + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = False + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'opnfv' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} +html_theme_options = { + 'bootswatch_theme': 'journal', + 'navbar_sidebarrel': False, + 'navbar_title': '', +} + +# Add any paths that contain custom themes here, relative to this directory. +# html_theme_path = [] +html_theme_path = sphinx_opnfv_theme.get_html_theme_path() + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +# html_static_path = [] + +# Custom sidebar templates, must be a dictionary that maps document names +# to template names. +# +# This is required for the alabaster theme +# refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars +html_sidebars = { + '**': [ + 'relations.html', # needs 'show_related': True theme option to display + 'searchbox.html', + ] +} + + +# -- Options for HTMLHelp output ------------------------------------------ + +# Output file base name for HTML help builder. +htmlhelp_basename = 'FunctestSecuritygrouptestcasesdoc' + + +# -- Options for LaTeX output --------------------------------------------- + +latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # + # 'preamble': '', + + # Latex figure (float) alignment + # + # 'figure_align': 'htbp', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, + 'FunctestSecuritygrouptestcases.tex', + u'Functest Security group test cases Documentation', + u'Cédric Ollivier \\textless{}cedric.ollivier@orange.com\\textgreater{}', + 'manual'), +] + + +# -- Options for manual page output --------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, + 'functestsecuritygrouptestcases', + u'Functest Security group test cases Documentation', + [author], + 1) +] + + +# -- Options for Texinfo output ------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, + 'FunctestSecuritygrouptestcases', + u'Functest Security group test cases Documentation', + author, + 'FunctestSecuritygrouptestcases', + 'One line description of project.', + 'Miscellaneous'), +] diff --git a/docs/testing/developer/internship/security_group/index.rst b/docs/testing/developer/internship/security_group/index.rst index d1cdbdd8f..87bcd90e5 100644 --- a/docs/testing/developer/internship/security_group/index.rst +++ b/docs/testing/developer/internship/security_group/index.rst @@ -1,11 +1,4 @@ -======= -License -======= - -Functest Docs are licensed under a Creative Commons Attribution 4.0 -International License. -You should have received a copy of the license along with this. -If not, see <http://creativecommons.org/licenses/by/4.0/>. +.. SPDX-License-Identifier: CC-BY-4.0 ================================== Functest Security group test cases @@ -57,9 +50,9 @@ Schedule: | **Date** | **Comment** | | | | +--------------------------+------------------------------------------+ -| December - January | ........ | +| December - January | '........' | +--------------------------+------------------------------------------+ -| January - february | ........ | +| January - february | '........' | +--------------------------+------------------------------------------+ @@ -67,4 +60,3 @@ References: =========== .. _`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Security+groups+test+case+in+Functest - diff --git a/docs/testing/developer/internship/testapi_evolution/conf.py b/docs/testing/developer/internship/testapi_evolution/conf.py new file mode 100644 index 000000000..cb7224155 --- /dev/null +++ b/docs/testing/developer/internship/testapi_evolution/conf.py @@ -0,0 +1,186 @@ +# -*- coding: utf-8 -*- +# +# TestAPI evolution documentation build configuration file, created by +# sphinx-quickstart on Wed Apr 4 06:00:42 2018. +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) +import sphinx_opnfv_theme + + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +# +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'TestAPI evolution' +copyright = u'2018, Cédric Ollivier <cedric.ollivier@orange.com>' +author = u'Cédric Ollivier <cedric.ollivier@orange.com>' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = u'master' +# The full version, including alpha/beta/rc tags. +release = u'master' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This patterns also effect to html_static_path and html_extra_path +exclude_patterns = [] + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = False + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'opnfv' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} +html_theme_options = { + 'bootswatch_theme': 'journal', + 'navbar_sidebarrel': False, + 'navbar_title': '', +} + +# Add any paths that contain custom themes here, relative to this directory. +# html_theme_path = [] +html_theme_path = sphinx_opnfv_theme.get_html_theme_path() + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +# html_static_path = [] + + +# Custom sidebar templates, must be a dictionary that maps document names +# to template names. +# +# This is required for the alabaster theme +# refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars +html_sidebars = { + '**': [ + 'relations.html', # needs 'show_related': True theme option to display + 'searchbox.html', + ] +} + + +# -- Options for HTMLHelp output ------------------------------------------ + +# Output file base name for HTML help builder. +htmlhelp_basename = 'TestAPIevolutiondoc' + + +# -- Options for LaTeX output --------------------------------------------- + +latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # + # 'preamble': '', + + # Latex figure (float) alignment + # + # 'figure_align': 'htbp', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, + 'TestAPIevolution.tex', + u'TestAPI evolution Documentation', + u'Cédric Ollivier \\textless{}cedric.ollivier@orange.com\\textgreater{}', + 'manual'), +] + + +# -- Options for manual page output --------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, + 'testapievolution', + u'TestAPI evolution Documentation', + [author], + 1) +] + + +# -- Options for Texinfo output ------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, + 'TestAPIevolution', + u'TestAPI evolution Documentation', + author, + 'TestAPIevolution', + 'One line description of project.', + 'Miscellaneous'), +] diff --git a/docs/testing/developer/internship/testapi_evolution/index.rst b/docs/testing/developer/internship/testapi_evolution/index.rst index 3038d0ac6..d2704de53 100644 --- a/docs/testing/developer/internship/testapi_evolution/index.rst +++ b/docs/testing/developer/internship/testapi_evolution/index.rst @@ -1,11 +1,4 @@ -======= -License -======= - -Functest Docs are licensed under a Creative Commons Attribution 4.0 -International License. -You should have received a copy of the license along with this. -If not, see <http://creativecommons.org/licenses/by/4.0/>. +.. SPDX-License-Identifier: CC-BY-4.0 ================= TestAPI evolution diff --git a/docs/testing/developer/internship/unit_tests/conf.py b/docs/testing/developer/internship/unit_tests/conf.py new file mode 100644 index 000000000..7bb31d406 --- /dev/null +++ b/docs/testing/developer/internship/unit_tests/conf.py @@ -0,0 +1,185 @@ +# -*- coding: utf-8 -*- +# +# Functest Unit tests documentation build configuration file, created by +# sphinx-quickstart on Wed Apr 4 06:04:23 2018. +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) +import sphinx_opnfv_theme + + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +# +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'Functest Unit tests' +copyright = u'2018, Cédric Ollivier <cedric.ollivier@orange.com>' +author = u'Cédric Ollivier <cedric.ollivier@orange.com>' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = u'master' +# The full version, including alpha/beta/rc tags. +release = u'master' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This patterns also effect to html_static_path and html_extra_path +exclude_patterns = [] + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = False + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'opnfv' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} +html_theme_options = { + 'bootswatch_theme': 'journal', + 'navbar_sidebarrel': False, + 'navbar_title': '', +} + +# Add any paths that contain custom themes here, relative to this directory. +# html_theme_path = [] +html_theme_path = sphinx_opnfv_theme.get_html_theme_path() + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +# html_static_path = [] + +# Custom sidebar templates, must be a dictionary that maps document names +# to template names. +# +# This is required for the alabaster theme +# refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars +html_sidebars = { + '**': [ + 'relations.html', # needs 'show_related': True theme option to display + 'searchbox.html', + ] +} + + +# -- Options for HTMLHelp output ------------------------------------------ + +# Output file base name for HTML help builder. +htmlhelp_basename = 'FunctestUnittestsdoc' + + +# -- Options for LaTeX output --------------------------------------------- + +latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # + # 'preamble': '', + + # Latex figure (float) alignment + # + # 'figure_align': 'htbp', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, + 'FunctestUnittests.tex', + u'Functest Unit tests Documentation', + u'Cédric Ollivier \\textless{}cedric.ollivier@orange.com\\textgreater{}', + 'manual'), +] + + +# -- Options for manual page output --------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, + 'functestunittests', + u'Functest Unit tests Documentation', + [author], + 1) +] + + +# -- Options for Texinfo output ------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, + 'FunctestUnittests', + u'Functest Unit tests Documentation', + author, + 'FunctestUnittests', + 'One line description of project.', + 'Miscellaneous'), +] diff --git a/docs/testing/developer/internship/unit_tests/index.rst b/docs/testing/developer/internship/unit_tests/index.rst index a117c8609..f79e9e296 100644 --- a/docs/testing/developer/internship/unit_tests/index.rst +++ b/docs/testing/developer/internship/unit_tests/index.rst @@ -1,11 +1,4 @@ -======= -License -======= - -Functest Docs are licensed under a Creative Commons Attribution 4.0 -International License. -You should have received a copy of the license along with this. -If not, see <http://creativecommons.org/licenses/by/4.0/>. +.. SPDX-License-Identifier: CC-BY-4.0 =================== Functest Unit tests @@ -36,39 +29,43 @@ Version history Overview: ========= Functest project is developing and integrating functional test cases for OPNFV -and it is part of OPNFV since the beginning. Functest develops its own testcases -and framework. This framework includes several utility libraries. The Project is -growing rapidly with more features, tests added as per requirement. It becomes -the responsibility of every developer to maintain the integrity of code i.e. new -patch should not break the previous functionality of the project. To automate this -process of software development, we should write unit tests and add them to CI so -that when a new patch is ready to merge, we shouldn't allow those which are breaking -previous unit tests or decreasing the coverage. +and it is part of OPNFV since the beginning. Functest develops its own +testcases and framework. This framework includes several utility libraries. The +Project is growing rapidly with more features, tests added as per requirement. +It becomes the responsibility of every developer to maintain the integrity of +code i.e. new patch should not break the previous functionality of the project. +To automate this process of software development, we should write unit tests +and add them to CI so that when a new patch is ready to merge, we shouldn't +allow those which are breaking previous unit tests or decreasing the coverage. Problem Statement: ------------------ -The goal of the intership consists in creating unit test suites on Functest code -with good code coverage (>80%) and integrate it in continuous integration in order -to consolidate existing code. +The goal of the intership consists in creating unit test suites on Functest +code with good code coverage (>80%) and integrate it in continuous integration +in order to consolidate existing code. Curation Phase -------------- -The curation phase was the first 3 to 4 weeks of the internship. This phase was to get -familiar with the functest code and functionality and explore the solutions for unit -testing in other projects and come up with the strategy for writing unit tests in functest. +The curation phase was the first 3 to 4 weeks of the internship. This phase was +to get familiar with the functest code and functionality and explore the +solutions for unit testing in other projects and come up with the strategy for +writing unit tests in functest. In this phase we decided, -- Coverage should be 80%. There are some functions like __init__, getter, setter and other - private methods for which writing unit test is a tedious job, so we are leaving these methods - for now. -- Do method wise testing for every module. -- Use mock for external or third party services, system calls and other external library calls - which could impact the behaviour of system during the run of unit test. -- Add it in jenkins as passing criteria for patches. -- Write tests in modular way so that it can help to serve as a form of documentation. + + - Coverage should be 80%. There are some functions like __init__, getter, + setter and other private methods for which writing unit test is a tedious + job, so we are leaving these methods for now. + - Do method wise testing for every module. + - Use mock for external or third party services, system calls and other + external library calls which could impact the behaviour of system during the + run of unit test. + - Add it in jenkins as passing criteria for patches. + - Write tests in modular way so that it can help to serve as a form of + documentation. diff --git a/docs/testing/developer/internship/vnf_catalog/conf.py b/docs/testing/developer/internship/vnf_catalog/conf.py new file mode 100644 index 000000000..4bc609810 --- /dev/null +++ b/docs/testing/developer/internship/vnf_catalog/conf.py @@ -0,0 +1,185 @@ +# -*- coding: utf-8 -*- +# +# Open Source VNF Catalog documentation build configuration file, created by +# sphinx-quickstart on Wed Apr 4 06:07:56 2018. +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) +import sphinx_opnfv_theme + + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +# +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'Open Source VNF Catalog' +copyright = u'2018, Cédric Ollivier <cedric.ollivier@orange.com>' +author = u'Cédric Ollivier <cedric.ollivier@orange.com>' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = u'master' +# The full version, including alpha/beta/rc tags. +release = u'master' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This patterns also effect to html_static_path and html_extra_path +exclude_patterns = [] + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = False + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'opnfv' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} +html_theme_options = { + 'bootswatch_theme': 'journal', + 'navbar_sidebarrel': False, + 'navbar_title': '', +} + +# Add any paths that contain custom themes here, relative to this directory. +# html_theme_path = [] +html_theme_path = sphinx_opnfv_theme.get_html_theme_path() + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +# html_static_path = [] + +# Custom sidebar templates, must be a dictionary that maps document names +# to template names. +# +# This is required for the alabaster theme +# refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars +html_sidebars = { + '**': [ + 'relations.html', # needs 'show_related': True theme option to display + 'searchbox.html', + ] +} + + +# -- Options for HTMLHelp output ------------------------------------------ + +# Output file base name for HTML help builder. +htmlhelp_basename = 'OpenSourceVNFCatalogdoc' + + +# -- Options for LaTeX output --------------------------------------------- + +latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # + # 'preamble': '', + + # Latex figure (float) alignment + # + # 'figure_align': 'htbp', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, + 'OpenSourceVNFCatalog.tex', + u'Open Source VNF Catalog Documentation', + u'Cédric Ollivier \\textless{}cedric.ollivier@orange.com\\textgreater{}', + 'manual'), +] + + +# -- Options for manual page output --------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, + 'opensourcevnfcatalog', + u'Open Source VNF Catalog Documentation', + [author], + 1) +] + + +# -- Options for Texinfo output ------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, + 'OpenSourceVNFCatalog', + u'Open Source VNF Catalog Documentation', + author, + 'OpenSourceVNFCatalog', + 'One line description of project.', + 'Miscellaneous'), +] diff --git a/docs/testing/developer/internship/vnf_catalog/index.rst b/docs/testing/developer/internship/vnf_catalog/index.rst index df7633391..634b728bd 100644 --- a/docs/testing/developer/internship/vnf_catalog/index.rst +++ b/docs/testing/developer/internship/vnf_catalog/index.rst @@ -1,11 +1,4 @@ -======= -License -======= - -Functest Docs are licensed under a Creative Commons Attribution 4.0 -International License. -You should have received a copy of the license along with this. -If not, see <http://creativecommons.org/licenses/by/4.0/>. +.. SPDX-License-Identifier: CC-BY-4.0 ======================= Open Source VNF Catalog @@ -68,57 +61,54 @@ Curation Phase This phase pertains to studying various Open Source VNFs available and classification of them based on certain parameters. The parameters that I currently have in mind are: + * Developer Metrics: These pertain to repo characteristics of VNF under - study - * Usage Statistics - Activity, Number of Commits, stars - * Maturity Statistics - For instance if an NFV community decides code - coverage is important for them, it shows the NFV community is serious - about taking the project forward + study + * Usage Statistics - Activity, Number of Commits, stars + * Maturity Statistics - For instance if an NFV community decides code + coverage is important for them, it shows the NFV community is serious + about taking the project forward * Technical Tagging: These are the tags that pertain to technical - characteristics of a VNF - * Broad Use Cases - Whether the VNF fits strictly in IaaS, PaaS or - SaaS layer or is an hybrid of two/all. - * Generic Use Cases - This in my opinion is the broadest - classification category. For instance a VNF could be built with a - broad idea of powering IOT devices at home or from usage perspective - of Telco Operators (vFW, vEPC, vIMS, vCDN, vAAA, vCPE,...).`[2]`_ - * Fields of Application - * Library Status - Whether APIs are standardized, support RESTful - services. - * Dependency Forwarding Graph - This is pretty complex tagging - mechanism. It essentially tries to establish a graph relationship - between the VNFs (elementary VNFs are used in Service Function - Chaining chains such as Firewall, DPI, content enrichment,..). In my - opinion this is useful immensely. This will allow users to go to - platform and ask a question like - “I have this X tech stack to - support, Y and Z are my use cases, which NFVs should I use to support - this. - * Visitor Score - Based on `[1]`_ I plan to evolve a visitor score for - the platform. This will allow users to score an NFV on certain - parameters, may be post comments. + characteristics of a VNF + * Broad Use Cases - Whether the VNF fits strictly in IaaS, PaaS or + SaaS layer or is an hybrid of two/all. + * Generic Use Cases - This in my opinion is the broadest + classification category. For instance a VNF could be built with a + broad idea of powering IOT devices at home or from usage perspective + of Telco Operators (vFW, vEPC, vIMS, vCDN, vAAA, vCPE,...).`[2]`_ + * Fields of Application + * Library Status - Whether APIs are standardized, support RESTful + services. + * Dependency Forwarding Graph - This is pretty complex tagging + mechanism. It essentially tries to establish a graph relationship + between the VNFs (elementary VNFs are used in Service Function + Chaining chains such as Firewall, DPI, content enrichment,..). In my + opinion this is useful immensely. This will allow users to go to + platform and ask a question like - “I have this X tech stack to + support, Y and Z are my use cases, which NFVs should I use to support + this. + * Visitor Score - Based on `[1]`_ I plan to evolve a visitor score for + the platform. This will allow users to score an NFV on certain + parameters, may be post comments. **I plan to use the above three scores and evolve cumulative score which will be displayed next to each of the NFV on the platform.** * Platform building phase - This will involve erecting a Web Platform - which will be similar to this `[1]`_. I am decently familiar with - Django and hence I will write the platform in Django. There are two - action plans that I have in mind right now. Either I can start writing - the platform simultaneously which will help keep track of my progress - or I can write the platform after 1.5 - 2 months into the internship. - Either way I aim to have the Web Platform ready by March 12. - + which will be similar to this `[1]`_. I am decently familiar with + Django and hence I will write the platform in Django. There are two + action plans that I have in mind right now. Either I can start writing + the platform simultaneously which will help keep track of my progress + or I can write the platform after 1.5 - 2 months into the internship. + Either way I aim to have the Web Platform ready by March 12. * Functest VNF implementation phase - This is the last phase that will - involve writing a fully functional implementation of an Open Source VNF - into Functest. I will undertake this after I am 3 months into the - internship. I have a decent familiarity with python and hence I think - it shouldn’t be too difficult. I need to decide how complex the VNFI - should undertake this exercise for (e.g. AAA such as free radius sounds - relatively easy, vCDN is much more challenging). - This will be decided in consent with my mentors. - - - + involve writing a fully functional implementation of an Open Source VNF + into Functest. I will undertake this after I am 3 months into the + internship. I have a decent familiarity with python and hence I think + it shouldn’t be too difficult. I need to decide how complex the VNFI + should undertake this exercise for (e.g. AAA such as free radius sounds + relatively easy, vCDN is much more challenging). + This will be decided in consent with my mentors. Schedule: ========= diff --git a/docs/testing/user/configguide/ci.rst b/docs/testing/user/configguide/ci.rst index f3901d867..e4fec51dc 100644 --- a/docs/testing/user/configguide/ci.rst +++ b/docs/testing/user/configguide/ci.rst @@ -1,3 +1,5 @@ +.. SPDX-License-Identifier: CC-BY-4.0 + Integration in CI ================= In CI we use the Docker images and execute the appropriate commands within the @@ -5,7 +7,8 @@ container from Jenkins. 4 steps have been defined:: * functest-cleanup: clean existing functest dockers on the jumphost - * functest-daily: run dockers opnfv/functest-* (healthcheck, smoke, features, vnf) + * functest-daily: run dockers opnfv/functest-* (healthcheck, smoke, features, + vnf) * functest-store-results: push logs to artifacts See `[3]`_ for details. diff --git a/docs/testing/user/configguide/conf.py b/docs/testing/user/configguide/conf.py new file mode 100644 index 000000000..a227d7caa --- /dev/null +++ b/docs/testing/user/configguide/conf.py @@ -0,0 +1,185 @@ +# -*- coding: utf-8 -*- +# +# Functest Installation Guide documentation build configuration file, created +# by sphinx-quickstart on Tue Apr 3 03:51:57 2018. +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) +import sphinx_opnfv_theme + + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +# +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'Functest Installation Guide' +copyright = u'2018, Cédric Ollivier <cedric.ollivier@orange.com>' +author = u'Cédric Ollivier <cedric.ollivier@orange.com>' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = u'master' +# The full version, including alpha/beta/rc tags. +release = u'master' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This patterns also effect to html_static_path and html_extra_path +exclude_patterns = [] + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = False + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'opnfv' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} +html_theme_options = { + 'bootswatch_theme': 'journal', + 'navbar_sidebarrel': False, + 'navbar_title': '', +} + +# Add any paths that contain custom themes here, relative to this directory. +# html_theme_path = [] +html_theme_path = sphinx_opnfv_theme.get_html_theme_path() + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +# html_static_path = [] + +# Custom sidebar templates, must be a dictionary that maps document names +# to template names. +# +# This is required for the alabaster theme +# refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars +html_sidebars = { + '**': [ + 'relations.html', # needs 'show_related': True theme option to display + 'searchbox.html', + ] +} + + +# -- Options for HTMLHelp output ------------------------------------------ + +# Output file base name for HTML help builder. +htmlhelp_basename = 'FunctestInstallationGuidedoc' + + +# -- Options for LaTeX output --------------------------------------------- + +latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # + # 'preamble': '', + + # Latex figure (float) alignment + # + # 'figure_align': 'htbp', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, + 'FunctestInstallationGuide.tex', + u'Functest Installation Guide Documentation', + u'Cédric Ollivier \\textless{}cedric.ollivier@orange.com\\textgreater{}', + 'manual'), +] + + +# -- Options for manual page output --------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, + 'functestinstallationguide', + u'Functest Installation Guide Documentation', + [author], + 1) +] + + +# -- Options for Texinfo output ------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, + 'FunctestInstallationGuide', + u'Functest Installation Guide Documentation', + author, + 'FunctestInstallationGuide', + 'One line description of project.', + 'Miscellaneous'), +] diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst index ef40bf598..a29ff3aaa 100644 --- a/docs/testing/user/configguide/configguide.rst +++ b/docs/testing/user/configguide/configguide.rst @@ -1,4 +1,3 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 Installation and configuration @@ -23,8 +22,9 @@ Docker images are available on the dockerhub: * opnfv/functest-parser * opnfv/functest-restapi -The tag "opnfv-5.0.0" is the official release image in Euphrates, but you can also pull "euphrates" -tag as it is being maintained by Functest team and might include bugfixes. +The tag "opnfv-5.0.0" is the official release image in Euphrates, but you can +also pull "euphrates" tag as it is being maintained by Functest team and might +include bugfixes. The Functest docker container environment can -in principle- be also used with non-OPNFV official installers (e.g. 'devstack'), with the @@ -173,10 +173,10 @@ Testing vnf suite Run vnf suite:: -sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-vnf + sudo docker run --env-file env \ + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ + -v $(pwd)/images:/home/opnfv/functest/images \ + opnfv/functest-vnf Results shall be displayed as follows:: @@ -218,8 +218,8 @@ release note. **NOTE:** The scenario name is mainly used to automatically detect if a test suite is runnable or not (e.g. it will prevent ODL test suite to be -run on 'nosdn' scenarios). If not set, Functest will try to run the default test -cases that might not include SDN controller or a specific feature +run on 'nosdn' scenarios). If not set, Functest will try to run the default +test cases that might not include SDN controller or a specific feature. **NOTE:** An HA scenario means that 3 OpenStack controller nodes are deployed. It does not necessarily mean that the whole system is HA. See @@ -478,6 +478,7 @@ We may distinguish several directories, the first level has 5 directories: Functest directory has 7 sub-directories, which is located under /usr/lib/python2.7/site-packages/functest: + * **api**: This directory is dedicated for the internal Functest API. * **ci**: This directory contains test structure definition files (e.g <filename>.yaml) and bash shell/python scripts used to @@ -605,6 +606,8 @@ If the OpenStack command still does not show anything or complains about connectivity issues, it could be due to an incorrect url given to the OS_AUTH_URL environment variable. Check the deployment settings. +.. _`Proxy support`: + Proxy support ------------- If your Jumphost node is operating behind a http proxy, then there are @@ -614,7 +617,7 @@ succeed: #. Initial installation of docker engine First, try following the official Docker documentation for Proxy settings. Some issues were experienced on CentOS 7 based Jumphost. Some tips are documented - in section: `Docker Installation on CentOS behind http proxy`_ + in section: :ref:`Docker Installation on CentOS behind http proxy` below. If that is the case, make sure the resolv.conf and the needed @@ -677,6 +680,8 @@ commands might not work. You can use the **curl** command instead. (Ignore the content. If command returns a valid HTML page, it proves the connection.) +.. _`Docker Installation on CentOS behind http proxy`: + Docker Installation on CentOS behind http proxy ----------------------------------------------- This section is applicable for CentOS family OS on Jumphost which diff --git a/docs/testing/user/configguide/index.rst b/docs/testing/user/configguide/index.rst index fd997344b..5d7eeb6e4 100644 --- a/docs/testing/user/configguide/index.rst +++ b/docs/testing/user/configguide/index.rst @@ -1,5 +1,4 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 +.. SPDX-License-Identifier: CC-BY-4.0 *************************** Functest Installation Guide @@ -9,101 +8,10 @@ Functest Installation Guide :numbered: :maxdepth: 2 - -Introduction -============ -This document describes how to install and configure Functest in OPNFV. - -High level architecture ------------------------ - -The high level architecture of Functest within OPNFV can be described as -follows:: - - CIMC/Lights+out management Admin Mgmt/API Public Storage Private - PXE - + + + + + + - | | | | | | - | +----------------------------+ | | | | | - | | | | | | | | - +-----+ Jumphost | | | | | | - | | | | | | | | - | | +--------------------+ | | | | | | - | | | | | | | | | | - | | | Tools | +----------------+ | | | - | | | - Rally | | | | | | | - | | | - Robot | | | | | | | - | | | - RefStack | | | | | | | - | | | | |-------------------------+ | | - | | | Testcases | | | | | | | - | | | - VIM | | | | | | | - | | | | | | | | | | - | | | - SDN Controller | | | | | | | - | | | | | | | | | | - | | | - Features | | | | | | | - | | | | | | | | | | - | | | - VNF | | | | | | | - | | | | | | | | | | - | | +--------------------+ | | | | | | - | | Functest Docker + | | | | | - | | | | | | | | - | | | | | | | | - | | | | | | | | - | +----------------------------+ | | | | | - | | | | | | - | +----------------+ | | | | | - | | 1 | | | | | | - +----+ +--------------+-+ | | | | | - | | | 2 | | | | | | - | | | +--------------+-+ | | | | | - | | | | 3 | | | | | | - | | | | +--------------+-+ | | | | | - | | | | | 4 | | | | | | - | +-+ | | +--------------+-+ | | | | | - | | | | | 5 +-------------+ | | | | - | +-+ | | nodes for | | | | | | - | | | | deploying +---------------------+ | | | - | +-+ | OPNFV | | | | | | - | | | +------------------------------+ | | - | +-+ SUT | | | | | | - | | +--------------------------------------+ | - | | | | | | | | - | | +----------------------------------------------+ - | +----------------+ | | | | | - | | | | | | - + + + + + + - SUT = System Under Test - -Note connectivity to management network is not needed for most of the testcases. -But it may be needed for some specific snaps tests. - -All the libraries and dependencies needed by all of the Functest tools are -pre-installed into the Docker images. This allows running Functest on any -platform. - -The automated mechanisms inside the Functest Docker containers will: - - * Prepare the environment according to the System Under Test (SUT) - * Perform the appropriate functional tests - * Push the test results into the OPNFV test result database (optional) - -The OpenStack credentials file must be provided to the container. - -These Docker images can be integrated into CI or deployed independently. - -Please note that the Functest Docker images have been designed for OPNFV, -however, it would be possible to adapt them to any OpenStack based VIM + -controller environment, since most of the test cases are integrated from -upstream communities. - -The functional test cases are described in the Functest User Guide `[2]`_ - -.. include:: ./prerequisites.rst - -.. include:: ./configguide.rst - -.. include:: ./ci.rst - + intro.rst + prerequisites.rst + configguide.rst + ci.rst References ========== diff --git a/docs/testing/user/configguide/intro.rst b/docs/testing/user/configguide/intro.rst new file mode 100644 index 000000000..72730b253 --- /dev/null +++ b/docs/testing/user/configguide/intro.rst @@ -0,0 +1,91 @@ +.. SPDX-License-Identifier: CC-BY-4.0 + +Introduction +============ +This document describes how to install and configure Functest in OPNFV. + +High level architecture +----------------------- + +The high level architecture of Functest within OPNFV can be described as +follows:: + + CIMC/Lights+out management Admin Mgmt/API Public Storage Private + PXE + + + + + + + + | | | | | | + | +----------------------------+ | | | | | + | | | | | | | | + +-----+ Jumphost | | | | | | + | | | | | | | | + | | +--------------------+ | | | | | | + | | | | | | | | | | + | | | Tools | +----------------+ | | | + | | | - Rally | | | | | | | + | | | - Robot | | | | | | | + | | | - RefStack | | | | | | | + | | | | |-------------------------+ | | + | | | Testcases | | | | | | | + | | | - VIM | | | | | | | + | | | | | | | | | | + | | | - SDN Controller | | | | | | | + | | | | | | | | | | + | | | - Features | | | | | | | + | | | | | | | | | | + | | | - VNF | | | | | | | + | | | | | | | | | | + | | +--------------------+ | | | | | | + | | Functest Docker + | | | | | + | | | | | | | | + | | | | | | | | + | | | | | | | | + | +----------------------------+ | | | | | + | | | | | | + | +----------------+ | | | | | + | | 1 | | | | | | + +----+ +--------------+-+ | | | | | + | | | 2 | | | | | | + | | | +--------------+-+ | | | | | + | | | | 3 | | | | | | + | | | | +--------------+-+ | | | | | + | | | | | 4 | | | | | | + | +-+ | | +--------------+-+ | | | | | + | | | | | 5 +-------------+ | | | | + | +-+ | | nodes for | | | | | | + | | | | deploying +---------------------+ | | | + | +-+ | OPNFV | | | | | | + | | | +------------------------------+ | | + | +-+ SUT | | | | | | + | | +--------------------------------------+ | + | | | | | | | | + | | +----------------------------------------------+ + | +----------------+ | | | | | + | | | | | | + + + + + + + + SUT = System Under Test + +Note connectivity to management network is not needed for most of the +testcases. But it may be needed for some specific snaps tests. + +All the libraries and dependencies needed by all of the Functest tools are +pre-installed into the Docker images. This allows running Functest on any +platform. + +The automated mechanisms inside the Functest Docker containers will: + + * Prepare the environment according to the System Under Test (SUT) + * Perform the appropriate functional tests + * Push the test results into the OPNFV test result database (optional) + +The OpenStack credentials file must be provided to the container. + +These Docker images can be integrated into CI or deployed independently. + +Please note that the Functest Docker images have been designed for OPNFV, +however, it would be possible to adapt them to any OpenStack based VIM + +controller environment, since most of the test cases are integrated from +upstream communities. + +The functional test cases are described in the Functest User `[2]`_ + +.. _`[2]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html diff --git a/docs/testing/user/configguide/prerequisites.rst b/docs/testing/user/configguide/prerequisites.rst index 8289803bb..60be1b93b 100644 --- a/docs/testing/user/configguide/prerequisites.rst +++ b/docs/testing/user/configguide/prerequisites.rst @@ -1,3 +1,5 @@ +.. SPDX-License-Identifier: CC-BY-4.0 + Prerequisites ============= The OPNFV deployment is out of the scope of this document but it can be @@ -13,10 +15,11 @@ Several prerequisites are needed for Functest: #. Connectivity from the Jumphost to the SUT public/external network Some specific SNAPS tests may require a connectivity from the Jumphost to the -SUT admin/management network but most of the test cases do not. This requirement -can be changed by overriding the 'interface' attribute (OS_INTERFACE) value -to 'public' in the credentials file. Another means to circumvent this issue -would be to change the 'snaps.use_keystone' value from True to False. +SUT admin/management network but most of the test cases do not. This +requirement can be changed by overriding the 'interface' attribute +(OS_INTERFACE) value to 'public' in the credentials file. Another means to +circumvent this issue would be to change the 'snaps.use_keystone' value from +True to False. WARNING: Connectivity from Jumphost is essential and it is of paramount importance to make sure it is working before even considering to install @@ -29,7 +32,7 @@ deployment has been triggered previously, but it could be any server with proper connectivity to the SUT. NOTE: If your Jumphost is operating behind a company http proxy and/or -firewall, please consult first the section `Proxy Support`_, towards +firewall, please consult first the section :ref:`Proxy support`, towards the end of this document. The section details some tips/tricks which *may* be of help in a proxified environment. @@ -44,8 +47,8 @@ commands below are offered as a short reference. *Tip:* For running docker containers behind the proxy, you need first some extra configuration which is described in section -`Docker Installation on CentOS behind http proxy`_. You should follow -that section before installing the docker engine. +:ref:`Docker Installation on CentOS behind http proxy`. You should follow that +section before installing the docker engine. Docker installation needs to be done as root user. You may use other userid's to create and run the actual containers later if so desired. diff --git a/docs/testing/user/userguide/conf.py b/docs/testing/user/userguide/conf.py new file mode 100644 index 000000000..cd37ed4f0 --- /dev/null +++ b/docs/testing/user/userguide/conf.py @@ -0,0 +1,185 @@ +# -*- coding: utf-8 -*- +# +# Functest User Guide documentation build configuration file, created by +# sphinx-quickstart on Wed Apr 4 04:05:42 2018. +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) +import sphinx_opnfv_theme + + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +# +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'Functest User Guide' +copyright = u'2018, Cédric Ollivier <cedric.ollivier@orange.com>' +author = u'Cédric Ollivier <cedric.ollivier@orange.com>' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = u'master' +# The full version, including alpha/beta/rc tags. +release = u'master' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This patterns also effect to html_static_path and html_extra_path +exclude_patterns = [] + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = False + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'opnfv' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} +html_theme_options = { + 'bootswatch_theme': 'journal', + 'navbar_sidebarrel': False, + 'navbar_title': '', +} + +# Add any paths that contain custom themes here, relative to this directory. +# html_theme_path = [] +html_theme_path = sphinx_opnfv_theme.get_html_theme_path() + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +# html_static_path = [] + +# Custom sidebar templates, must be a dictionary that maps document names +# to template names. +# +# This is required for the alabaster theme +# refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars +html_sidebars = { + '**': [ + 'relations.html', # needs 'show_related': True theme option to display + 'searchbox.html', + ] +} + + +# -- Options for HTMLHelp output ------------------------------------------ + +# Output file base name for HTML help builder. +htmlhelp_basename = 'FunctestUserGuidedoc' + + +# -- Options for LaTeX output --------------------------------------------- + +latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # + # 'preamble': '', + + # Latex figure (float) alignment + # + # 'figure_align': 'htbp', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, + 'FunctestUserGuide.tex', + u'Functest User Guide Documentation', + u'Cédric Ollivier \\textless{}cedric.ollivier@orange.com\\textgreater{}', + 'manual'), +] + + +# -- Options for manual page output --------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, + 'functestuserguide', + u'Functest User Guide Documentation', + [author], + 1) +] + + +# -- Options for Texinfo output ------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, + 'FunctestUserGuide', + u'Functest User Guide Documentation', + author, + 'FunctestUserGuide', + 'One line description of project.', + 'Miscellaneous'), +] diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst index 66dfd3e7b..36951fc66 100644 --- a/docs/testing/user/userguide/index.rst +++ b/docs/testing/user/userguide/index.rst @@ -1,8 +1,7 @@ -.. _functest-userguide: - -.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 +.. _functest-userguide: + ******************* Functest User Guide ******************* @@ -10,32 +9,13 @@ Functest User Guide .. toctree:: :maxdepth: 2 - - -Introduction -============ - -The goal of this document is to describe the OPNFV Functest test cases and to -provide a procedure to execute them. - -**IMPORTANT**: It is assumed here that Functest has been properly deployed -following the installation guide procedure `[1]`_. - -.. include:: ./test_overview.rst - -.. include:: ./test_details.rst - -.. include:: ./runfunctest.rst - -.. include:: ./test_results.rst - -.. include:: ./reporting.rst - -.. figure:: ../../../images/functest-reporting-status.png - :align: center - :alt: Functest reporting portal Fuel status page - -.. include:: ./troubleshooting.rst + intro.rst + test_overview.rst + test_details.rst + runfunctest.rst + test_results.rst + reporting.rst + troubleshooting.rst References diff --git a/docs/testing/user/userguide/intro.rst b/docs/testing/user/userguide/intro.rst new file mode 100644 index 000000000..0c36cd8c7 --- /dev/null +++ b/docs/testing/user/userguide/intro.rst @@ -0,0 +1,12 @@ +.. SPDX-License-Identifier: CC-BY-4.0 + +Introduction +============ + +The goal of this document is to describe the OPNFV Functest test cases and to +provide a procedure to execute them. + +**IMPORTANT**: It is assumed here that Functest has been properly deployed +following the installation guide procedure `[1]`_. + +.. _`[1]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html diff --git a/docs/testing/user/userguide/reporting.rst b/docs/testing/user/userguide/reporting.rst index 88f5e865a..337c203db 100644 --- a/docs/testing/user/userguide/reporting.rst +++ b/docs/testing/user/userguide/reporting.rst @@ -1,4 +1,3 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 Test reporting @@ -42,7 +41,9 @@ and features) corresponding to this scenario. +---------------------+---------+---------+---------+---------+ | copper | X | | | X | +---------------------+---------+---------+---------+---------+ - src: os-odl_l2-nofeature-ha Colorado (see release note for the last matrix version) + + src: os-odl_l2-nofeature-ha Colorado (see release note for the last matrix + version) All the testcases (X) listed in the table are runnable on os-odl_l2-nofeature scenarios. @@ -69,16 +70,20 @@ scoring. In fact the success criteria are not always easy to define and may require specific hardware configuration. Please also note that all the test cases have the same "weight" for the score -calculation whatever the complexity of the test case. Concretely a vping has the -same weith than the 200 tempst tests. +calculation whatever the complexity of the test case. Concretely a vping has +the same weight than the 200 tempest tests. Moreover some installers support more features than others. The more cases your scenario is dealing with, the most difficult to rich a good scoring. Therefore the scoring provides 3 types of indicators: - * the richness of the scenario: if the target scoring is high, it means that the scenario includes lots of features - * the maturity: if the percentage (scoring/target scoring * 100) is high, it means that all the tests are PASS - * the stability: as the number of iteration is included in the calculation, the pecentage can be high only if the scenario is run regularly (at least more than 4 iterations over the last 10 days in CI) + * the richness of the scenario: if the target scoring is high, it means that + the scenario includes lots of features + * the maturity: if the percentage (scoring/target scoring * 100) is high, it + means that all the tests are PASS + * the stability: as the number of iteration is included in the calculation, + the pecentage can be high only if the scenario is run regularly (at least + more than 4 iterations over the last 10 days in CI) In any case, the scoring is used to give feedback to the other projects and does not represent an absolute value of the scenario. @@ -86,5 +91,8 @@ does not represent an absolute value of the scenario. See `reporting page`_ for details. For the status, click on the version, Functest then the Status menu. - .. _`reporting page`: http://testresults.opnfv.org/reporting/ + +.. figure:: ../../../images/functest-reporting-status.png + :align: center + :alt: Functest reporting portal Fuel status page diff --git a/docs/testing/user/userguide/runfunctest.rst b/docs/testing/user/userguide/runfunctest.rst index d5b95101a..07e16efed 100644 --- a/docs/testing/user/userguide/runfunctest.rst +++ b/docs/testing/user/userguide/runfunctest.rst @@ -1,6 +1,4 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - +.. SPDX-License-Identifier: CC-BY-4.0 Executing Functest suites ========================= @@ -38,8 +36,8 @@ The result tables show the results by test case, it can be:: Manual run ========== -If you want to run the test step by step, you may add docker option then run the -different commands within the docker. +If you want to run the test step by step, you may add docker option then run +the different commands within the docker. Considering the healthcheck example, running functest manaully means:: @@ -104,7 +102,7 @@ Note the list of test cases depend on the installer and the scenario. Reporting results to the test Database ====================================== -In OPNFV CI we collect all the results from CI. A test APi shall be available +In OPNFV CI we collect all the results from CI. A test API shall be available as well as a test database `[17]`_. Functest internal API diff --git a/docs/testing/user/userguide/test_details.rst b/docs/testing/user/userguide/test_details.rst index 1ce4fce76..fe7e6a9b6 100644 --- a/docs/testing/user/userguide/test_details.rst +++ b/docs/testing/user/userguide/test_details.rst @@ -1,8 +1,7 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 +.. SPDX-License-Identifier: CC-BY-4.0 - -The different test cases are described in the remaining sections of this document. +The different test cases are described in the remaining sections of this +document. VIM (Virtualized Infrastructure Manager) ---------------------------------------- @@ -59,8 +58,8 @@ Given the script **ping.sh**:: The goal of this test is to establish an SSH connection using a floating IP -on the Public/External network and verify that 2 instances can talk over a Private -Tenant network:: +on the Public/External network and verify that 2 instances can talk over a +Private Tenant network:: vPing_ssh test case +-------------+ +-------------+ @@ -105,7 +104,8 @@ vPing_userdata This test case is similar to vPing_ssh but without the use of Floating IPs and the Public/External network to transfer the ping script. -Instead, it uses Nova metadata service to pass it to the instance at booting time. +Instead, it uses Nova metadata service to pass it to the instance at booting +time. As vPing_ssh, it checks that 2 instances can talk to each other on a Private Tenant network:: @@ -164,16 +164,29 @@ Tiers: * Smoke Tier - Test Case 'tempest_smoke_serial' * Components Tier - Test case 'tempest_full_parallel' * Neutron Trunk Port - Test case 'neutron_trunk' + * OpenStack interop testcases - Test case 'refstack_defcore' + * Testing and verifying RBAC policy enforcement - Test case 'patrole' NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke -tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel' -executes all defined Tempest tests using several concurrent threads -(i.e. parallel mode). The number of threads activated corresponds to the number -of available logical CPUs. - -NOTE: The 'neutron_trunk' test set allows to connect a VM to multiple VLAN separated -networks using a single NIC. The feature neutron trunk ports have been supported -by Apex, Fuel and Compass, so the tempest testcases have been integrated normally. +tests with a single thread (i.e. serial mode). Test case +'tempest_full_parallel' executes all defined Tempest tests using several +concurrent threads (i.e. parallel mode). The number of threads activated +corresponds to the number of available logical CPUs. + +NOTE: The 'neutron_trunk' test set allows to connect a VM to multiple VLAN +separated networks using a single NIC. The feature neutron trunk ports have +been supported by Apex, Fuel and Compass, so the tempest testcases have been +integrated normally. + +NOTE: Rally is also used to run Openstack Interop testcases `[9]`_, which focus +on testing interoperability between OpenStack clouds. + +NOTE: Patrole is a tempest plugin for testing and verifying RBAC policy +enforcement. It runs Tempest-based API tests using specified RBAC roles, thus +allowing deployments to verify that only intended roles have access to those +APIs. Patrole currently offers testing for the following OpenStack services: +Nova, Neutron, Glance, Cinder and Keystone. Currently in functest, only neutron +and glance are tested. The goal of the Tempest test suite is to check the basic functionalities of the different OpenStack components on an OPNFV fresh installation, using the @@ -187,10 +200,11 @@ Rally `[3]`_ is a benchmarking tool that answers the question: *How does OpenStack work at scale?* -The goal of this test suite is to benchmark all the different OpenStack modules and -get significant figures that could help to define Telco Cloud KPIs. +The goal of this test suite is to benchmark all the different OpenStack modules +and get significant figures that could help to define Telco Cloud KPIs. -The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios: +The OPNFV Rally scenarios are based on the collection of the actual Rally +scenarios: * authenticate * cinder @@ -200,7 +214,6 @@ The OPNFV Rally scenarios are based on the collection of the actual Rally scenar * neutron * nova * quotas - * ceilometer A basic SLA (stop test on errors) has been implemented. @@ -213,102 +226,15 @@ NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test cases. Test case 'rally_full' executes the full defined set of Rally tests. -Refstack-client to run OpenStack interop testcases --------------------------------------------------- - -Refstack-client `[8]`_ is a command line utility that allows you to -execute Tempest test runs based on configurations you specify. -It is the official tool to run Openstack Interop (previously known as Defcore) -testcases `[9]`_, which focus on testing interoperability between OpenStack -clouds. - -Refstack-client is integrated in Functest, consumed by Dovetail, which -intends to define and provide a set of OPNFV related validation criteria -that will provide input for the evaluation of the use of OPNFV trademarks. -This progress is under the guideline of Compliance Verification Program(CVP). - -Running methods -^^^^^^^^^^^^^^^ - -Two running methods are provided after refstack-client integrated into -Functest, Functest command line and manually, respectively. - -By default, for Defcore test cases run by Functest command line, -are run followed with automatically generated -configuration file, i.e., refstack_tempest.conf. In some circumstances, -the automatic configuration file may not quite satisfied with the SUT, -Functest also inherits the refstack-client command line and provides a way -for users to set its configuration file according to its own SUT manually. - -*command line* - -Inside the Functest container, first to prepare Functest environment: - -:: - - functest env prepare - -then to run default defcore testcases by using refstack-client: - -:: - - functest testcase run refstack_defcore - -In OPNFV Continuous Integration(CI) system, the command line method is used. - -*manually* - -Prepare the tempest configuration file and the testcases want to run with the SUT, -run the testcases with: - -:: - - ./refstack-client test -c <Path of the tempest configuration file to use> -v --test-list <Path or URL of test list> - -using help for more information: - -:: - - ./refstack-client --help - ./refstack-client test --help - -Reference tempest configuration -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -*command line method* - -When command line method is used, the default tempest configuration file -is generated by Rally. - -*manually* - -When running manually is used, recommended way to generate tempest configuration -file is: - -:: - - cd /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/refstack_client - python tempest_conf.py - -a file called tempest.conf is stored in the current path by default, users can do -some adjustment according to the SUT: - -:: - - vim refstack_tempest.conf - -a reference article can be used `[15]`_. - - snaps_smoke ------------ This test case contains tests that setup and destroy environments with VMs with and without Floating IPs with a newly created user and project. Set the config value snaps.use_floating_ips (True|False) to toggle this functionality. -Please note that When the configuration value of snaps.use_keystone is True, Functest must have access -the cloud's private network. -This suite consists in 38 tests (test duration < 10 minutes) +Please note that When the configuration value of snaps.use_keystone is True, +Functest must have access the cloud's private network. +This suite consists in 120 tests (test duration ~= 50 minutes) SDN Controllers @@ -325,31 +251,40 @@ OpenDaylight and Neutron. The list of tests can be described as follows: * Basic Restconf test cases + * Connect to Restconf URL * Check the HTTP code status * Neutron Reachability test cases + * Get the complete list of neutron resources (networks, subnets, ports) * Neutron Network test cases + * Check OpenStack networks * Check OpenDaylight networks - * Create a new network via OpenStack and check the HTTP status code returned by Neutron + * Create a new network via OpenStack and check the HTTP status code returned + by Neutron * Check that the network has also been successfully created in OpenDaylight * Neutron Subnet test cases + * Check OpenStack subnets * Check OpenDaylight subnets - * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron + * Create a new subnet via OpenStack and check the HTTP status code returned + by Neutron * Check that the subnet has also been successfully created in OpenDaylight * Neutron Port test cases + * Check OpenStack Neutron for known ports * Check OpenDaylight ports - * Create a new port via OpenStack and check the HTTP status code returned by Neutron + * Create a new port via OpenStack and check the HTTP status code returned by + Neutron * Check that the new port has also been successfully created in OpenDaylight * Delete operations + * Delete the port previously created via OpenStack * Check that the port has been also successfully deleted in OpenDaylight * Delete previously subnet created via OpenStack @@ -364,7 +299,7 @@ code returned by OpenDaylight. Features -------- -Functest has been supporting several feature projects since Brahpamutra: +Functest has been supporting several feature projects since Brahmaputra: +-----------------+---------+----------+--------+-----------+ @@ -414,9 +349,9 @@ The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an architectural framework for delivering IP multimedia services. vIMS has been integrated in Functest to demonstrate the capability to deploy a -relatively complex NFV scenario on the OPNFV platform. The deployment of a complete -functional VNF allows the test of most of the essential functions needed for a -NFV platform. +relatively complex NFV scenario on the OPNFV platform. The deployment of a +complete functional VNF allows the test of most of the essential functions +needed for a NFV platform. The goal of this test suite consists of: @@ -438,12 +373,14 @@ This testcase extends the cloudify_ims test case. The first part is similar but the testing part is different. The testing part consists in automating a realistic signaling load on the vIMS using an Ixia loader (proprietary tools) - - You need to have access to an Ixia licence server defined in the configuration - file and have ixia image locally. + + - You need to have access to an Ixia licence server defined in the + configuration file and have ixia image locally. This test case is available but not declared in testcases.yaml. The declaration -of the testcase is simple, connect to your functest-vnf docker, add the following -section in /usr/lib/python2.7/site-packacges/functest/ci/testcases.yaml:: +of the testcase is simple, connect to your functest-vnf docker, add the +following section in +/usr/lib/python2.7/site-packacges/functest/ci/testcases.yaml:: - case_name: cloudify_ims_perf @@ -460,16 +397,6 @@ section in /usr/lib/python2.7/site-packacges/functest/ci/testcases.yaml:: module: 'functest.opnfv_tests.vnf.ims.cloudify_ims_perf' class: 'CloudifyImsPerf' -orchestra_openims -^^^^^^^^^^^^^^^^^ -Orchestra test case deals with the deployment of OpenIMS with OpenBaton -orchestrator. - -orchestra_clearwaterims -^^^^^^^^^^^^^^^^^^^^^^^ -Orchestra test case deals with the deployment of Clearwater vIMS with OpenBaton -orchestrator. - vyos-vrouter ^^^^^^^^^^^^ This test case deals with the deployment and the test of vyos vrouter with @@ -489,14 +416,16 @@ The Workflow is as follows: The vyos-vrouter architecture is described in `[14]`_ +juju_epc +^^^^^^^^ .. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html .. _`[3]`: https://rally.readthedocs.org/en/latest/index.html .. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml .. _`[8]`: https://github.com/openstack/refstack-client +.. _`[9]`: https://github.com/openstack/defcore .. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst .. _`[11]`: http://robotframework.org/ .. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html .. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter -.. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/ diff --git a/docs/testing/user/userguide/test_overview.rst b/docs/testing/user/userguide/test_overview.rst index 8919ed713..91eb34326 100644 --- a/docs/testing/user/userguide/test_overview.rst +++ b/docs/testing/user/userguide/test_overview.rst @@ -1,5 +1,4 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 +.. SPDX-License-Identifier: CC-BY-4.0 Overview of the Functest suites =============================== @@ -21,151 +20,157 @@ healthcheck, smoke, features, components, performance, VNF, Stress tests. All the Healthcheck and smoke tests of a given scenario must be succesful to validate the scenario for the release. -+-------------+---------------+----------------+----------------------------------+ -| Domain | Tier | Test case | Comments | -+=============+===============+================+==================================+ -| VIM | healthcheck | connection | Check OpenStack connectivity | -| | | _check | through SNAPS framework | -| | +----------------+----------------------------------+ -| | | api_check | Check OpenStack API through | -| | | | SNAPS framework | -| | +----------------+----------------------------------+ -| | | snaps_health | basic instance creation, check | -| | | \_check | DHCP | -| +---------------+----------------+----------------------------------+ -| | smoke | vping_ssh | NFV "Hello World" using an SSH | -| | | | connection to a destination VM | -| | | | over a created floating IP | -| | | | address on the SUT Public / | -| | | | External network. Using the SSH | -| | | | connection a test script is then | -| | | | copied to the destination | -| | | | VM and then executed via SSH. | -| | | | The script will ping another | -| | | | VM on a specified IP address over| -| | | | the SUT Private Tenant network | -| | +----------------+----------------------------------+ -| | | vping_userdata | Uses Ping with given userdata | -| | | | to test intra-VM connectivity | -| | | | over the SUT Private Tenant | -| | | | network. The correct operation | -| | | | of the NOVA Metadata service is | -| | | | also verified in this test | -| | +----------------+----------------------------------+ -| | | tempest_smoke | Generate and run a relevant | -| | | \_serial | Tempest Test Suite in smoke mode.| -| | | | The generated test set is | -| | | | dependent on the OpenStack | -| | | | deployment environment | -| | +----------------+----------------------------------+ -| | | rally_sanity | Run a subset of the OpenStack | -| | | | Rally Test Suite in smoke mode | -| | +----------------+----------------------------------+ -| | | snaps_smoke | Run the SNAPS-OO integration | -| | | | tests | -| | +----------------+----------------------------------+ -| | | refstack | Reference RefStack suite | -| | | \_defcore | tempest selection for NFV | -| | +----------------+----------------------------------+ -| | | neutron_trunk | The neutron trunk port testcases | -| | | | have been introduced and they are| -| | | | supported by installers : | -| | | | Apex, Fuel and Compass. | -| +---------------+----------------+----------------------------------+ -| | components | tempest_full | Generate and run a full set of | -| | | \_parallel | the OpenStack Tempest Test Suite.| -| | | | See the OpenStack reference test | -| | | | suite `[2]`_. The generated | -| | | | test set is dependent on the | -| | | | OpenStack deployment environment | -| | +----------------+----------------------------------+ -| | | rally_full | Run the OpenStack testing tool | -| | | | benchmarking OpenStack modules | -| | | | See the Rally documents `[3]`_ | -+-------------+---------------+----------------+----------------------------------+ -| Controllers | smoke | odl | Opendaylight Test suite | -| | | | Limited test suite to check the | -| | | | basic neutron (Layer 2) | -| | | | operations mainly based on | -| | | | upstream testcases. See below | -| | | | for details | -| | +----------------+----------------------------------+ -| | | odl_netvirt | Test Suite for the OpenDaylight | -| | | | SDN Controller when the NetVirt | -| | | | features are installed. It | -| | | | integrates some test suites from | -| | | | upstream using Robot as the test | -| | | | framework | -+-------------+---------------+----------------+----------------------------------+ -| Features | features | bgpvpn | Implementation of the OpenStack | -| | | | bgpvpn API from the SDNVPN | -| | | | feature project. It allows for | -| | | | the creation of BGP VPNs. | -| | | | See `SDNVPN User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | doctor | Doctor platform, as of Colorado | -| | | | release, provides the three | -| | | | features: | -| | | | * Immediate Notification | -| | | | * Consistent resource state | -| | | | awareness for compute host down | -| | | | * Valid compute host status | -| | | | given to VM owner | -| | | | See `Doctor User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | odl-sfc | SFC testing for odl scenarios | -| | | | See `SFC User Guide`_ for details| -| | +----------------+----------------------------------+ -| | | parser | Parser is an integration project | -| | | | which aims to provide | -| | | | placement/deployment templates | -| | | | translation for OPNFV platform, | -| | | | including TOSCA -> HOT, POLICY ->| -| | | | TOSCA and YANG -> TOSCA. it | -| | | | deals with a fake vRNC. | -| | | | See `Parser User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | fds | Test Suite for the OpenDaylight | -| | | | SDN Controller when the GBP | -| | | | features are installed. It | -| | | | integrates some test suites from | -| | | | upstream using Robot as the test | -| | | | framework | -+-------------+---------------+----------------+----------------------------------+ -| VNF | vnf | cloudify_ims | Example of a real VNF deployment | -| | | | to show the NFV capabilities of | -| | | | the platform. The IP Multimedia | -| | | | Subsytem is a typical Telco test | -| | | | case, referenced by ETSI. | -| | | | It provides a fully functional | -| | | | VoIP System | -| | +----------------+----------------------------------+ -| | | orchestra | OpenIMS deployment using | -| | | \_openims | Openbaton orchestrator | -| | +----------------+----------------------------------+ -| | | orchestra | Cleawater IMS deployment using | -| | | \_cleawaterims | Openbaton orchestrator | -| | +----------------+----------------------------------+ -| | | vyos_vrouter | vRouter testing | -| | +----------------+----------------------------------+ -| | | cloudify_ims | Based on cloudify_ims test case | -| | | perf | cloudify_ims_perf substitutes | -| | | | the signaling test suite by an | -| | | | automatic deployment of an Ixia | -| | | | loader and generic SIP stress | -| | | | tests. | -| | | | This work has been initiated | -| | | | during the plugfest and allows | -| | | | realistic load tests on top of | -| | | | cloudify_ims. Please note that | -| | | | this test is available but not | -| | | | declared in testcases.yaml as it | -| | | | requires access to proprietary | -| | | | resources (Ixia loader) | -+-------------+---------------+----------------+----------------------------------+ ++-------------+---------------+------------+----------------------------------+ +| Domain | Tier | Test case | Comments | ++=============+===============+============+==================================+ +| VIM | healthcheck | connection | Check OpenStack connectivity | +| | | \_check | through SNAPS framework | +| | +------------+----------------------------------+ +| | | api_check | Check OpenStack API through | +| | | | SNAPS framework | +| | +------------+----------------------------------+ +| | | snaps | basic instance creation, check | +| | | \_health | DHCP | +| | | \_check | | +| +---------------+------------+----------------------------------+ +| | smoke | vping_ssh | NFV "Hello World" using an SSH | +| | | | connection to a destination VM | +| | | | over a created floating IP | +| | | | address on the SUT Public / | +| | | | External network. Using the SSH | +| | | | connection a test script is then | +| | | | copied to the destination | +| | | | VM and then executed via SSH. | +| | | | The script will ping another | +| | | | VM on a specified IP address over| +| | | | the SUT Private Tenant network | +| | +------------+----------------------------------+ +| | | vping | Uses Ping with given userdata | +| | | \_userdata | to test intra-VM connectivity | +| | | | over the SUT Private Tenant | +| | | | network. The correct operation | +| | | | of the NOVA Metadata service is | +| | | | also verified in this test | +| | +------------+----------------------------------+ +| | | tempest | Generate and run a relevant | +| | | \_smoke | Tempest Test Suite in smoke mode.| +| | | \_serial | The generated test set is | +| | | | dependent on the OpenStack | +| | | | deployment environment | +| | +------------+----------------------------------+ +| | | rally | Run a subset of the OpenStack | +| | | \_sanity | Rally Test Suite in smoke mode | +| | +------------+----------------------------------+ +| | | snaps\ | Run the SNAPS-OO integration | +| | | \_smoke | tests | +| | +------------+----------------------------------+ +| | | refstack | Reference RefStack suite | +| | | \_defcore | tempest selection for NFV | +| | +------------+----------------------------------+ +| | | patrole | Patrole is a tempest plugin for | +| | | | testing and verifying RBAC policy| +| | | | enforcement, which offers testing| +| | | | for the following OpenStack | +| | | | services: Nova, Neutron, Glance, | +| | | | Cinder and Keystone | +| +---------------+------------+----------------------------------+ +| | | neutron | The neutron trunk port testcases | +| | | \_trunk | have been introduced and they are| +| | | | supported by installers : | +| | | | Apex, Fuel and Compass. | +| +---------------+------------+----------------------------------+ +| | components | tempest | Generate and run a full set of | +| | | \_full | the OpenStack Tempest Test Suite.| +| | | \_parallel | See the OpenStack reference test | +| | | | suite `[2]`_. The generated | +| | | | test set is dependent on the | +| | | | OpenStack deployment environment | +| | +------------+----------------------------------+ +| | | rally_full | Run the OpenStack testing tool | +| | | | benchmarking OpenStack modules | +| | | | See the Rally documents `[3]`_ | ++-------------+---------------+------------+----------------------------------+ +| Controllers | smoke | odl | Opendaylight Test suite | +| | | | Limited test suite to check the | +| | | | basic neutron (Layer 2) | +| | | | operations mainly based on | +| | | | upstream testcases. See below | +| | | | for details | +| | +------------+----------------------------------+ +| | | odl | Test Suite for the OpenDaylight | +| | | \_netvirt | SDN Controller when the NetVirt | +| | | | features are installed. It | +| | | | integrates some test suites from | +| | | | upstream using Robot as the test | +| | | | framework | ++-------------+---------------+------------+----------------------------------+ +| Features | features | bgpvpn | Implementation of the OpenStack | +| | | | bgpvpn API from the SDNVPN | +| | | | feature project. It allows for | +| | | | the creation of BGP VPNs. | +| | | | See `SDNVPN User Guide`_ for | +| | | | details | +| | +------------+----------------------------------+ +| | | doctor | Doctor platform, as of Colorado | +| | | | release, provides the three | +| | | | features: | +| | | | * Immediate Notification | +| | | | * Consistent resource state | +| | | | awareness for compute host down | +| | | | * Valid compute host status | +| | | | given to VM owner | +| | | | See `Doctor User Guide`_ for | +| | | | details | +| | +------------+----------------------------------+ +| | | odl-sfc | SFC testing for odl scenarios | +| | | | See `SFC User Guide`_ for details| +| | +------------+----------------------------------+ +| | | parser | Parser is an integration project | +| | | | which aims to provide | +| | | | placement/deployment templates | +| | | | translation for OPNFV platform, | +| | | | including TOSCA -> HOT, POLICY ->| +| | | | TOSCA and YANG -> TOSCA. it | +| | | | deals with a fake vRNC. | +| | | | See `Parser User Guide`_ for | +| | | | details | +| | +------------+----------------------------------+ +| | | fds | Test Suite for the OpenDaylight | +| | | | SDN Controller when the GBP | +| | | | features are installed. It | +| | | | integrates some test suites from | +| | | | upstream using Robot as the test | +| | | | framework | ++-------------+---------------+------------+----------------------------------+ +| VNF | vnf | cloudify | Example of a real VNF deployment | +| | | \_ims | to show the NFV capabilities of | +| | | | the platform. The IP Multimedia | +| | | | Subsytem is a typical Telco test | +| | | | case, referenced by ETSI. | +| | | | It provides a fully functional | +| | | | VoIP System | +| | +------------+----------------------------------+ +| | | vyos | vRouter testing | +| | | \_vrouter | | +| | +------------+----------------------------------+ +| | | juju_epc | vEPC validation with Juju as VNF | +| | | | manager and ABoT as test executor| +| | +------------+----------------------------------+ +| | | cloudify | Based on cloudify_ims test case | +| | | \_ims_perf | cloudify_ims_perf substitutes | +| | | | the signaling test suite by an | +| | | | automatic deployment of an Ixia | +| | | | loader and generic SIP stress | +| | | | tests. | +| | | | This work has been initiated | +| | | | during the plugfest and allows | +| | | | realistic load tests on top of | +| | | | cloudify_ims. Please note that | +| | | | this test is available but not | +| | | | declared in testcases.yaml as it | +| | | | requires access to proprietary | +| | | | resources (Ixia loader) | ++-------------+---------------+------------+----------------------------------+ As shown in the above table, Functest is structured into different 'domains', @@ -174,7 +179,8 @@ As shown in the above table, Functest is structured into different 'domains', Test cases also have an implicit execution order. For example, if the early 'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke' -Tier testcases, there is little point to launch a full testcase execution round. +Tier testcases, there is little point to launch a full testcase execution +round. In Danube, we merged smoke and sdn controller tiers in smoke tier. @@ -190,13 +196,13 @@ are integrated from upstream communities or other OPNFV projects. For example, OpenStack integration test suite and Functest is in charge of the selection, integration and automation of those tests that fit suitably to OPNFV. -The Tempest test suite is the default OpenStack smoke test suite but no new test -cases have been created in OPNFV Functest. +The Tempest test suite is the default OpenStack smoke test suite but no new +test cases have been created in OPNFV Functest. The results produced by the tests run from CI are pushed and collected into a -NoSQL database. The goal is to populate the database with results from different -sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot -of a live Functest Dashboard is shown below: +NoSQL database. The goal is to populate the database with results from +different sources and scenarios and to show them on a `Functest Dashboard`_. A +screenshot of a live Functest Dashboard is shown below: .. figure:: ../../../images/FunctestDashboardEuphrates.png :align: center @@ -222,9 +228,9 @@ combinations (which may change from one version to another): Most of the tests are runnable by any combination, but some tests might have restrictions imposed by the utilized installers or due to the available -deployed features. The system uses the environment variables (INSTALLER_TYPE and -DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given -environment. +deployed features. The system uses the environment variables (INSTALLER_TYPE +and DEPLOY_SCENARIO) to automatically determine the valid test cases, for each +given environment. A convenience Functest CLI utility is also available to simplify setting up the Functest evironment, management of the OpenStack environment (e.g. resource diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst index 3941ba0a1..2d36bdc33 100644 --- a/docs/testing/user/userguide/test_results.rst +++ b/docs/testing/user/userguide/test_results.rst @@ -1,3 +1,5 @@ +.. SPDX-License-Identifier: CC-BY-4.0 + Test results ============ @@ -11,10 +13,10 @@ If you want additional logs, you may configure the logging.ini under /usr/lib/python2.7/site-packages/functest/ci. Automated testing --------------- +----------------- -In automated mode, test results are displayed in jenkins logs, a summary is provided -at the end of the job and can be described as follow:: +In automated mode, test results are displayed in jenkins logs, a summary is +provided at the end of the job and can be described as follow:: +-------------------------+----------------------------------------------------------+ | ENV VAR | VALUE | @@ -25,6 +27,8 @@ at the end of the job and can be described as follow:: | CI_LOOP | daily | +-------------------------+----------------------------------------------------------+ +:: + +------------------------------+------------------+---------------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +------------------------------+------------------+---------------------+------------------+----------------+ diff --git a/docs/testing/user/userguide/troubleshooting.rst b/docs/testing/user/userguide/troubleshooting.rst index 1649ac140..cce9d009d 100644 --- a/docs/testing/user/userguide/troubleshooting.rst +++ b/docs/testing/user/userguide/troubleshooting.rst @@ -1,5 +1,4 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 +.. SPDX-License-Identifier: CC-BY-4.0 Troubleshooting =============== @@ -26,8 +25,8 @@ rally_full). vPing common ^^^^^^^^^^^^ -For both vPing test cases (**vPing_ssh**, and **vPing_userdata**), the first steps are -similar: +For both vPing test cases (**vPing_ssh**, and **vPing_userdata**), the first +steps are similar: * Create Glance image * Create Network @@ -37,15 +36,17 @@ similar: After these actions, the test cases differ and will be explained in their respective section. -These test cases can be run inside the container, using new Functest CLI as follows:: +These test cases can be run inside the container, using new Functest CLI as +follows:: $ functest testcase run vping_ssh $ functest testcase run vping_userdata The Functest CLI is designed to route a call to the corresponding internal -python scripts, located in paths -/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_ssh.py -and /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_userdata.py +python scripts, located in paths:: + + /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_ssh.py + /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_userdata.py Notes: @@ -73,10 +74,11 @@ Some of the common errors that can appear in this test case are:: vPing_ssh- ERROR - There has been a problem when creating the neutron network.... -This means that there has been some problems with Neutron, even before creating the -instances. Try to create manually a Neutron network and a Subnet to see if that works. -The debug messages will also help to see when it failed (subnet and router creation). -Example of Neutron commands (using 10.6.0.0/24 range for example):: +This means that there has been some problems with Neutron, even before creating +the instances. Try to create manually a Neutron network and a Subnet to see if +that works. The debug messages will also help to see when it failed (subnet and +router creation). Example of Neutron commands (using 10.6.0.0/24 range for +example):: neutron net-create net-test neutron subnet-create --name subnet-test --allocation-pool start=10.6.0.2,end=10.6.0.100 \ @@ -85,7 +87,8 @@ Example of Neutron commands (using 10.6.0.0/24 range for example):: neutron router-interface-add <ROUTER_ID> test_subnet neutron router-gateway-set <ROUTER_ID> <EXT_NET_NAME> -Another related error can occur while creating the Security Groups for the instances:: +Another related error can occur while creating the Security Groups for the +instances:: vPing_ssh- ERROR - Failed to create the security group... @@ -100,9 +103,9 @@ In this case, proceed to create it manually. These are some hints:: --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0 The next step is to create the instances. The image used is located in -*/home/opnfv/functest/data/cirros-0.4.0-x86_64-disk.img* and a Glance image is created -with the name **functest-vping**. If booting the instances fails (i.e. the status -is not **ACTIVE**), you can check why it failed by doing:: +*/home/opnfv/functest/data/cirros-0.4.0-x86_64-disk.img* and a Glance image is +created with the name **functest-vping**. If booting the instances fails (i.e. +the status is not **ACTIVE**), you can check why it failed by doing:: nova list nova show <INSTANCE_ID> @@ -113,7 +116,8 @@ It might show some messages about the booting failure. To try that manually:: This will spawn a VM using the network created previously manually. In all the OPNFV tested scenarios from CI, it never has been a problem with the -previous actions. Further possible problems are explained in the following sections. +previous actions. Further possible problems are explained in the following +sections. vPing_SSH @@ -143,19 +147,21 @@ from the DHCP agent. It can be checked by doing:: nova console-log opnfv-vping-2 If the message *Sending discover* and *No lease, failing* is shown, it probably -means that the Neutron dhcp-agent failed to assign an IP or even that it was not -responding. At this point it does not make sense to try to ping the floating IP. +means that the Neutron dhcp-agent failed to assign an IP or even that it was +not responding. At this point it does not make sense to try to ping the +floating IP. -If the instance got an IP properly, try to ping manually the VM from the container:: +If the instance got an IP properly, try to ping manually the VM from the +container:: nova list <grab the public IP> ping <public IP> -If the ping does not return anything, try to ping from the Host where the Docker -container is running. If that solves the problem, check the iptable rules because -there might be some rules rejecting ICMP or TCP traffic coming/going from/to the -container. +If the ping does not return anything, try to ping from the Host where the +Docker container is running. If that solves the problem, check the iptable +rules because there might be some rules rejecting ICMP or TCP traffic +coming/going from/to the container. At this point, if the ping does not work either, try to reproduce the test manually with the steps described above in the vPing common section with the @@ -176,7 +182,8 @@ vPing_userdata This test case does not create any floating IP neither establishes an SSH connection. Instead, it uses nova-metadata service when creating an instance to pass the same script as before (ping.sh) but as 1-line text. This script -will be executed automatically when the second instance **opnfv-vping-2** is booted. +will be executed automatically when the second instance **opnfv-vping-2** is +booted. The only known problem here for this test to fail is mainly the lack of support of cloud-init (nova-metadata service). Check the console of the instance:: @@ -219,39 +226,41 @@ In the upstream OpenStack CI all the Tempest test cases are supposed to pass. If some test cases fail in an OPNFV deployment, the reason is very probably one of the following -+-----------------------------+-----------------------------------------------------+ -| Error | Details | -+=============================+=====================================================+ -| Resources required for test | Such resources could be e.g. an external network | -| case execution are missing | and access to the management subnet (adminURL) from | -| | the Functest docker container. | -+-----------------------------+-----------------------------------------------------+ -| OpenStack components or | Check running services in the controller and compute| -| services are missing or not | nodes (e.g. with "systemctl" or "service" commands).| -| configured properly | Configuration parameters can be verified from the | -| | related .conf files located under '/etc/<component>'| -| | directories. | -+-----------------------------+-----------------------------------------------------+ -| Some resources required for | The tempest.conf file, automatically generated by | -| execution test cases are | Rally in Functest, does not contain all the needed | -| missing | parameters or some parameters are not set properly. | -| | The tempest.conf file is located in directory | -| | 'root/.rally/verification/verifier-<UUID> | -| | /for-deployment-<UUID>' | -| | in the Functest Docker container. Use the "rally | -| | deployment list" command in order to check the UUID | -| | the UUID of the current deployment. | -+-----------------------------+-----------------------------------------------------+ ++----------------------------+------------------------------------------------+ +| Error | Details | ++============================+================================================+ +| Resources required for | Such resources could be e.g. an external | +| testcase execution are | network and access to the management subnet | +| missing | (adminURL) from the Functest docker container. | ++----------------------------+------------------------------------------------+ +| OpenStack components or | Check running services in the controller and | +| services are missing or | compute nodes (e.g. with "systemctl" or | +| not configured properly | "service" commands). | +| | Configuration parameters can be verified from | +| | the related .conf files located under | +| | '/etc/<component>' directories. | ++----------------------------+------------------------------------------------+ +| Some resources required | The tempest.conf file, automatically generated | +| for execution test cases | by Rally in Functest, does not contain all the | +| are missing | needed parameters or some parameters are not | +| | set properly. | +| | The tempest.conf file is located in directory | +| | 'root/.rally/verification/verifier-<UUID> | +| | /for-deployment-<UUID>' | +| | in the Functest Docker container. Use the | +| | "rally deployment list" command in order to | +| | check the UUID of the current deployment. | ++----------------------------+------------------------------------------------+ When some Tempest test case fails, captured traceback and possibly also the -related REST API requests/responses are output to the console. More detailed debug -information can be found from tempest.log file stored into related Rally deployment -folder. +related REST API requests/responses are output to the console. More detailed +debug information can be found from tempest.log file stored into related Rally +deployment folder. Functest offers a possibility to test a customized list of Tempest test cases. -To enable that, add a new entry in docker/components/testcases.yaml on the "components" container -with the following content:: +To enable that, add a new entry in docker/components/testcases.yaml on the +"components" container with the following content:: - case_name: tempest_custom @@ -268,8 +277,8 @@ with the following content:: module: 'functest.opnfv_tests.openstack.tempest.tempest' class: 'TempestCustom' -Also, a list of the Tempest test cases must be provided to the container or modify -the existing one in +Also, a list of the Tempest test cases must be provided to the container or +modify the existing one in /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt Example of custom list of tests 'my-custom-tempest-tests.txt':: @@ -290,8 +299,8 @@ This is an example of running a customized list of Tempest tests in Functest:: Rally ^^^^^ -The same error causes which were mentioned above for Tempest test cases, may also -lead to errors in Rally as well. +The same error causes which were mentioned above for Tempest test cases, may +also lead to errors in Rally as well. Possible scenarios are: * authenticate @@ -301,18 +310,19 @@ Possible scenarios are: * keystone * neutron * nova - * ceilometer * quotas * vm -To know more about what those scenarios are doing, they are defined in directory: +To know more about what those scenarios are doing, they are defined in +directory: /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/rally/scenario -For more info about Rally scenario definition please refer to the Rally official -documentation. `[3]`_ +For more info about Rally scenario definition please refer to the Rally +official documentation. `[3]`_ To check any possible problems with Rally, the logs are stored under */home/opnfv/functest/results/rally/* in the Functest Docker container. +.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html Controllers ----------- @@ -372,7 +382,7 @@ described in the following table: | the VM | the vIMS VNF installation fails | +-----------------------------------+------------------------------------+ -Please note that this test case requires resources (8 VM (2Go) + 1 VM (4Go)), it -is there fore not recommended to run it on a light configuration. +Please note that this test case requires resources (8 VM (2Go) + 1 VM (4Go)), +it is there fore not recommended to run it on a light configuration. .. _`OPNFV Functest Developer Guide`: http://artifacts.opnfv.org/functest/docs/testing_developer_devguide/index.html# |