diff options
| -rw-r--r-- | docs/configguide/configguide.rst | 158 | ||||
| -rw-r--r-- | docs/configguide/index.rst | 78 | ||||
| -rw-r--r-- | docs/devguide/index.rst | 365 |
3 files changed, 347 insertions, 254 deletions
diff --git a/docs/configguide/configguide.rst b/docs/configguide/configguide.rst index ea5b01ea6..c03760c5c 100644 --- a/docs/configguide/configguide.rst +++ b/docs/configguide/configguide.rst @@ -30,9 +30,9 @@ following docker command:: [functester@jumphost ~]$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE - opnfv/functest latest 8cd6683c32ae 2 weeks ago 1.611 GB - opnfv/functest colorado.1.0 94b78faa94f7 4 weeks ago 874.9 MB - hello-world latest 94df4f0ce8a4 7 weeks ago 967 B + opnfv/functest latest 8cd6683c32ae 2 weeks ago 1.321 GB + opnfv/functest danube.1.0 13fa54a1b238 4 weeks ago 1.29 GB + opnfv/functest colorado.1.0 94b78faa94f7 9 weeks ago 968 MB The Functest docker container environment can -in principle- be also used with non-OPNFV official installers (e.g. 'devstack'), with the @@ -104,10 +104,14 @@ recommended parameters for invoking docker container -e "INSTALLER_TYPE=<type>" # Use one of following apex, compass, fuel or joid - #. Functest needs to know the IP of the installer:: + #. Functest needs to know the IP of some installers:: -e "INSTALLER_IP=<Specific IP Address>" + This IP is needed to fetch RC file from deployment, fetch logs, ... + If not provided, there is no way to fetch the RC file. It must be + provided manually as a volume + #. Credentials for accessing the Openstack. Most convenient way of passing them to container is by having a local copy of the credentials file in Jumphost and then using the @@ -138,13 +142,24 @@ recommended parameters for invoking docker container If several features are pertinent then use the underscore character '_' to separate each feature (e.g. ovs_kvm) 'nofeature' indicates no NFV feature is deployed - ha_mode is one of ( ha | noha ) + ha_mode (high availability) is one of ( ha | noha ) **NOTE:** Not all possible combinations of "DEPLOY_SCENARIO" are supported. The name passed in to the Functest Docker container must match the scenario used when the actual OPNFV platform was deployed. See release note to see the list of supported scenarios. + **NOTE:** The scenario name is mainly used to automatically detect + if a test suite is runnable or not (e.g. it will prevent ONOS test suite + to be run on ODL scenarios). If not set, Functest will try to run the + default test cases that might not include SDN controller or a specific + feature + + **NOTE:** A HA scenario means that 3 OpenStack controller nodes are + deployed. It does not necessarily mean that the whole system is HA. See + installer release notes for details. + + Putting all above together, when using installer 'fuel' and an invented INSTALLER_IP of '10.20.0.2', the recommended command to create the Functest Docker container is as follows:: @@ -311,14 +326,25 @@ should now be in place:: | `-- results `-- repos |-- bgpvpn + |-- copper |-- doctor + |-- domino |-- functest - |-- odl_integration + |-- kingbird + |-- moon + |-- odl_test |-- onos + |-- ovno + |-- parser |-- promise |-- rally |-- releng - `-- vims-test + |-- sdnvpn + |-- securityscanning + |-- sfc + |-- tempest + |-- vims_test + `-- vnfs Underneath the '/home/opnfv/' directory, the Functest docker container includes two main directories: @@ -339,24 +365,10 @@ follows:: . |-- INFO |-- LICENSE - |-- __init__.py - |-- ci - | |-- __init__.py - | |-- check_os.sh - | |-- config_functest.yaml - | |-- generate_report.py - | |-- exec_test.sh - | |-- prepare_env.py - | |-- run_tests.py - | |-- testcases.yaml - | |-- tier_builder.py - | `-- tier_handler.py - |-- cli - | |-- __init__.py - | |-- cli_base.py - | |-- commands - | |-- functest-complete.sh - | `-- setup.py + |-- requirements.txt + |-- run_unit_tests.sh + |-- setup.py + |-- test-requirements.txt |-- commons | |-- ims | |-- mobile @@ -364,50 +376,96 @@ follows:: |-- docker | |-- Dockerfile | |-- config_install_env.sh - | `-- requirements.pip + | `-- docker_remote_api |-- docs | |-- com | |-- configguide | |-- devguide | |-- images + | |-- internship | |-- release-notes | |-- results | `--userguide - |-- testcases - | |-- __init__.py - | |-- Controllers - | |-- OpenStack - | |-- features - | |-- security_scan - | `-- vnf - `-- utils + |-- functest |-- __init__.py - |-- functest_logger.py - |-- functest_utils.py - |-- openstack_clean.py - |-- openstack_snapshot.py - `-- openstack_utils.py + |-- ci + | |-- __init__.py + | |-- check_os.sh + | |-- config_functest.yaml + | |-- config_patch.yaml + | |-- exec_test.sh + | |-- generate_report.py + | |-- prepare_env.py + | |-- run_tests.py + | |-- testcases.yaml + | |-- tier_builder.py + | `-- tier_handler.py + |-- cli + | |-- __init__.py + | |-- cli_base.py + | |-- commands + | |-- functest-complete.sh + | `-- setup.py + |-- core + | |-- __init__.py + | |-- feature_base.py + | |-- pytest_suite_runner.py + | |-- testcase_base.py + | |-- vnf_base.py + |-- opnfv_tests + | |-- __init__.py + | |-- features + | |-- mano + | |-- openstack + | |-- sdn + | |-- security_scan + | `-- vnf + |-- tests + | |-- __init__.py + | `-- unit + `-- utils + |-- __init__.py + |-- config.py + |-- constants.py + |-- env.py + |-- functest_constants.py + |-- functest_logger.py + |-- functest_utils.py + |-- openstack + |-- openstack_clean.py + |-- openstack_snapshot.py + |-- openstack_tacker.py + `-- openstack_utils.py + (Note: All *.pyc files removed from above list for brevity...) -We may distinguish 7 different directories: +We may distinguish several directories, the first level has 4 directories: + +* **commons**: This directory is dedicated for storage of traffic + profile or any other test inputs that could be reused by any test + project. +* **docker**: This directory includes the needed files and tools to + build the Funtest Docker image. +* **docs**: This directory includes documentation: Release Notes, + User Guide, Configuration Guide and Developer Guide. Test results + are also located in a sub--directory called 'results'. +* **functest**: This directory contains all the code needed to run + functest internal cases and OPNFV onboarded feature or VNF test cases. +Functest directory has 6 directories: * **ci**: This directory contains test structure definition files (e.g <filename>.yaml) and bash shell/python scripts used to configure and execute Functional tests. The test execution script can be executed under the control of Jenkins CI jobs. * **cli**: This directory holds the python based Functest CLI utility source code, which is based on the Python 'click' framework. - * **commons**: This directory is dedicated for storage of traffic - profile or any other test inputs that could be reused by any test - project. - * **docker**: This directory includes the needed files and tools to - build the Funtest Docker container image. - * **docs**: This directory includes documentation: Release Notes, - User Guide, Configuration Guide and Developer Guide. Test results - are also located in a sub--directory called 'results'. - * **testcases**: This directory includes the scripts required by + * **core**: This directory holds the python based Functest core + source code. Three abstraction classes have been created to ease + the integration of internal, feature or vnf cases. + * **opnfv_tests**: This directory includes the scripts required by Functest internal test cases and other feature projects test cases. + * **tests**: This directory includes the functest unit tests * **utils**: this directory holds Python source code for some general purpose helper utilities, which testers can also re-use in their own test code. See for an example the Openstack helper utility: @@ -452,7 +510,7 @@ It is useful sometimes to remove a container if there are some problems:: Use the *-f* option if the container is still running, it will force to destroy it:: - docker -f rm <CONTAINER_ID> + docker rm -f <CONTAINER_ID> Check the Docker documentation dockerdocs_ for more information. diff --git a/docs/configguide/index.rst b/docs/configguide/index.rst index b61bf8ff5..58e8a1343 100644 --- a/docs/configguide/index.rst +++ b/docs/configguide/index.rst @@ -9,6 +9,21 @@ OPNFV FUNCTEST Configuration Guide :numbered: :maxdepth: 2 +Version history +=============== + ++------------+----------+------------------+----------------------------------+ +| **Date** | **Ver.** | **Author** | **Comment** | +| | | | | ++------------+----------+------------------+----------------------------------+ +| 2016-08-17 | 1.0.0 | Juha Haapavirta | Colorado release | +| | | Column Gaynor | | ++------------+----------+------------------+----------------------------------+ +| 2017-01-19 | 1.0.1 | Morgan Richomme | Adaptations for Danube | +| | | | * update testcase list | +| | | | * update docker command | ++------------+----------+------------------+----------------------------------+ + Introduction ============ This document describes how to install and configure Functest in OPNFV. @@ -43,6 +58,7 @@ follows:: | | | -- healthcheck | | | | | | | | | -- vPing_ssh | | | | | | | | | -- vPing_userdata | | | | | | + | | | -- SNAPS_cases | | | | | | | | | -- Tempest_smoke | | | | | | | | | -- Rally_sanity | | | | | | | | | -- Tempest_full | | | | | | @@ -52,12 +68,13 @@ follows:: | | | -- odl | | | | | | | | | -- onos | | | | | | | | | | | | | | | + | | | - Features | | | | | | + | | | | | | | | | | | | - VNF | | | | | | - | | | -- vIMS | | | | | | | | | | | | | | | | | +--------------------+ | | | | | - | | +-------------------------+ | - | | Functest Docker | | | | | + | | Functest Docker +-------------------------+ | + | | | | | | | | | +---------------------------------+ | | | | | | | | +----------------------------+ | | | | @@ -107,7 +124,7 @@ The functional test cases are described in the Functest User Guide `[2]`_ Prerequisites ============= The OPNFV deployment is out of the scope of this document but it can be -found in http://artifacts.opnfv.org/opnfvdocs/colorado/docs/configguide/index.html. +found in http://docs.opnfv.org. The OPNFV platform is considered as the System Under Test (SUT) in this document. @@ -217,15 +234,14 @@ container from Jenkins. Docker creation in set-functest-env builder `[3]`_:: - envs="INSTALLER_TYPE=${INSTALLER_TYPE} -e INSTALLER_IP=${INSTALLER_IP} -e NODE_NAME=${NODE_NAME}" + envs="-e INSTALLER_TYPE=${INSTALLER_TYPE} -e INSTALLER_IP=${INSTALLER_IP} -e NODE_NAME=${NODE_NAME}" [...] - docker pull opnfv/functest:latest_stable - cmd="docker run -id -e $envs ${labconfig} ${sshkey} ${res_volume} opnfv/functest:latest_stable /bin/bash" + docker pull opnfv/functest:$DOCKER_TAG >/dev/null + cmd="sudo docker run -id ${envs} ${volumes} ${custom_params} ${TESTCASE_OPTIONS} opnfv/functest:${DOCKER_TAG} /bin/bash" echo "Functest: Running docker run command: ${cmd}" - ${cmd} - docker ps -a + ${cmd} >${redirect} sleep 5 - container_id=$(docker ps | grep 'opnfv/functest:latest_stable' | awk '{print $1}' | head -1) + container_id=$(docker ps | grep "opnfv/functest:${DOCKER_TAG}" | awk '{print $1}' | head -1) echo "Container ID=${container_id}" if [ -z ${container_id} ]; then echo "Cannot find opnfv/functest container ID ${container_id}. Please check if it is existing." @@ -235,43 +251,29 @@ Docker creation in set-functest-env builder `[3]`_:: echo "Starting the container: docker start ${container_id}" docker start ${container_id} sleep 5 - docker ps - if [ $(docker ps | grep 'opnfv/functest:latest_stable' | wc -l) == 0 ]; then + docker ps >${redirect} + if [ $(docker ps | grep "opnfv/functest:${DOCKER_TAG}" | wc -l) == 0 ]; then echo "The container opnfv/functest with ID=${container_id} has not been properly started. Exiting..." exit 1 fi - cmd="${FUNCTEST_REPO_DIR}/docker/prepare_env.sh" + + cmd="python ${FUNCTEST_REPO_DIR}/functest/ci/prepare_env.py start" echo "Executing command inside the docker: ${cmd}" docker exec ${container_id} ${cmd} Test execution in functest-all builder `[3]`_:: - echo "Functest: run $FUNCTEST_SUITE_NAME" - cmd="${FUNCTEST_REPO_DIR}/docker/run_tests.sh --test $FUNCTEST_SUITE_NAME ${flag}" - container_id=$(docker ps -a | grep opnfv/functest | awk '{print $1}' | head -1) - docker exec $container_id $cmd - -Docker clean in functest-cleanup builder `[3]`_:: - - echo "Cleaning up docker containers/images..." - # Remove previous running containers if exist - if [[ ! -z $(docker ps -a | grep opnfv/functest) ]]; then - echo "Removing existing opnfv/functest containers..." - docker ps | grep opnfv/functest | awk '{print $1}' | xargs docker stop - docker ps -a | grep opnfv/functest | awk '{print $1}' | xargs docker rm + branch=${GIT_BRANCH##*/} + echo "Functest: run $FUNCTEST_SUITE_NAME on branch ${branch}" + cmd="functest testcase run $FUNCTEST_SUITE_NAME" fi + container_id=$(docker ps -a | grep opnfv/functest | awk '{print $1}' | head -1) + docker exec $container_id $cmd + ret_value=$? + exit $ret_value - # Remove existing images if exist - if [[ ! -z $(docker images | grep opnfv/functest) ]]; then - echo "Docker images to remove:" - docker images | head -1 && docker images | grep opnfv/functest - image_tags=($(docker images | grep opnfv/functest | awk '{print $2}')) - for tag in "${image_tags[@]}"; do - echo "Removing docker image opnfv/functest:$tag..." - docker rmi opnfv/functest:$tag - done - fi +Docker clean in functest-cleanup builder `[3]`_ calling docker rm and docker rmi References @@ -279,7 +281,7 @@ References .. _`[1]`: https://ask.openstack.org/en/question/68144/keystone-unable-to-use-the-public-endpoint/ .. _`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html .. _`[3]`: https://git.opnfv.org/cgit/releng/tree/jjb/functest/functest-ci-jobs.yml -.. _`[4]`: http://artifacts.opnfv.org/opnfvdocs/colorado/docs/configguide/index.html +.. _`[4]`: http://artifacts.opnfv.org/functest/danube/docs/configguide/index.html OPNFV main site: opnfvmain_. @@ -289,4 +291,4 @@ OPNFV functional test page: opnfvfunctest_. IRC support channel: #opnfv-functest .. _opnfvmain: http://www.opnfv.org -.. _opnfvfunctest: https://wiki.opnfv.org/opnfv_functional_testing +.. _opnfvfunctest: https://wiki.opnfv.org/functest diff --git a/docs/devguide/index.rst b/docs/devguide/index.rst index 6b3e1f0e6..6624f8a70 100644 --- a/docs/devguide/index.rst +++ b/docs/devguide/index.rst @@ -13,7 +13,7 @@ Introduction Functest is a project dealing with functional testing. Functest produces its own internal test cases but can also be considered -as a framework to support feature project testing. +as a framework to support feature and VNF onboarding project testing. Functest developed a test API and defined a test collection framework that can be used by any OPNFV project. @@ -60,17 +60,20 @@ Functest can be described as follow:: Functest internal test cases ============================ -The internal test cases in Colorado are: +The internal test cases in Danube are: * healthcheck * vping_ssh * vping_userdata * odl + * connection_check + * api_check + * snaps_smoke * tempest_smoke_serial * rally_sanity * tempest_full_parallel * rally_full - * vims + * cloudify_ims By internal, we mean that this particular test cases have been developped and/or integrated by functest contributors and the associated @@ -79,12 +82,12 @@ An internal case can be fully developped or a simple integration of upstream suites (e.g. Tempest/Rally developped in OpenStack are just integrated in Functest). The structure of this repository is detailed in `[1]`_. -The main internal test cases are in the testcases subfolder of the +The main internal test cases are in the opnfv_tests subfolder of the repository, the internal test cases are: - * Controllers: odl, onos, ocl - * OpenStack: healthcheck, vping_ssh, vping_userdata, tempest_*, rally_* - * VNF: vims + * sdn: odl, onos + * openstack: healthcheck, vping_ssh, vping_userdata, tempest_*, rally_*, connection_check, api_check, snaps_smoke + * vnf: cloudify_ims If you want to create a new test case you will have to create a new folder under the testcases directory. @@ -108,17 +111,15 @@ The external test cases are: * parser * domino * multisite + * opera_ims + * orchestra_ims -Note that security_scan has been bootstraped in Functest but is -considered as an external test case as it gets its own repository. - The code to run these test cases may be directly in the repository of -the project. We have also a **features** sub directory under testcases +the project. We have also a **features** sub directory under opnfv_tests directory that may be used (it can be usefull if you want to reuse Functest library). - Functest framework ================== @@ -132,9 +133,82 @@ Since Colorado, test categories also known as tiers have been created to group similar tests, provide consistant sub-lists and at the end optimize test duration for CI (see How To section). -see http://artifacts.opnfv.org/functest/docs/userguide/index.html for -details. +The definition of the tiers has been agreed by the testing working group. + +The tiers are: + * healthcheck + * smoke + * features + * components + * performance + * vnf + * stress + +Functest abstraction classes +============================ +In order to harmonize test integration, 3 abstraction classes have been +introduced in Danube: + + * testcase_base: base for any test case + * feature_base: abstraction for feature project + * vnf_base: abstraction for vnf onboarding + +The goal is to unify the way to run test from Functest. + +feature_base and vnf_base inherit from testcase_base. + + +-----------------------------------------+ + | | + | Testcase_base | + | | + | - init() | + | - run() | + | - publish_report() | + | - check_criteria() | + | | + +-----------------------------------------+ + | | + V V + +--------------------+ +--------------------------+ + | | | | + | feature_base | | vnf_base | + | | | | + | - prepare() | | - prepare() | + | - post() | | - deploy_orchestrator() | + | - parse_results() | | - deploy_vnf() | + | | | - test_vnf() | + | | | - clean() | + | | | - execute() | + | | | | + +--------------------+ +--------------------------+ + + +Functest util classes +===================== + +In order to simplify the creation of test cases, Functest develops some +functions that can be used by any feature or internal test cases. +Several features are supported such as logger, configuration management and +Openstack capabilities (snapshot, clean, tacker,..). +These functions can be found under <repo>/functest/utils and can be described as +follows: + +functest/utils/ +|-- config.py +|-- constants.py +|-- env.py +|-- functest_constants.py +|-- functest_logger.py +|-- functest_utils.py +|-- openstack_clean.py +|-- openstack_snapshot.py +|-- openstack_tacker.py +`-- openstack_utils.py + +Note that for Openstack, keystone v3 is now deployed by default by compass, +fuel and joid in Danube. All installers still support keysone v2 (deprecated in +next version). Test collection framework ========================= @@ -415,153 +489,131 @@ Results: | GET | /api/v1/results/{result_id}| Get the test result by result_id | +--------+----------------------------+------------------------------------------+ +Scenarios: -Dashboard: + +--------+----------------------------+-----------------------------------------+ + | Method | Path | Description | + +========+============================+=========================================+ + | GET | /api/v1/scenarios | Get the list of declared scenarios | + +--------+----------------------------+-----------------------------------------+ + | POST | /api/v1/scenario | Declare a new scenario | + +--------+----------------------------+-----------------------------------------+ + | GET | /api/v1/scenario? | Get a declared scenario | + | | <query conditions> | | + +--------+----------------------------+-----------------------------------------+ - +--------+----------------------------+-----------------------------------------+ - | Method | Path | Description | - +========+============================+=========================================+ - | GET |/dashboard/v1/results? | Get all the dashboard ready results of | - | |&project={project} | {case} of the project {project} | - | |&case={case} | | - +--------+----------------------------+-----------------------------------------+ - | GET |/dashboard/v1/results? | Get all the dashboard ready results of | - | |&project={project} | {case} of the project {project} | - | |&case={case} | version {version} | - | |&version={version} | | - +--------+----------------------------+-----------------------------------------+ - | GET |/dashboard/v1/results? | Get all the dashboard ready results of | - | |&project={project} | {case} of the project {project} | - | |&case={case} | since {days} days | - | |&period={days} | | - +--------+----------------------------+-----------------------------------------+ - | GET |/dashboard/v1/results? | Get all the dashboard ready results of | - | |&project={project} | {case} of the project {project} | - | |&case={case} | installed by {installer} | - | |&installer={installer} | | - +--------+----------------------------+-----------------------------------------+ - | GET |/dashboard/v1/results? | Get all the dashboard ready results of | - | |&project={project} | {case} of the project {project} | - | |&case={case} | on POD {pod} | - | |&pod={pod} | | - +--------+----------------------------+-----------------------------------------+ - | GET |/dashboard/v1/results? | Get all the dashboard ready results of | - | |&project={project} | {case} of the project {project} | - | |&case={case} | and combined by other query conditions | - | |&<query conditions> | supported above. | - +--------+----------------------------+-----------------------------------------+ - | GET |/dashboard/v1/projects? | Get all the dashboard ready projects | - +--------+----------------------------+-----------------------------------------+ The code of the API is hosted in the releng repository `[6]`_. The test API has been dockerized and may be installed locally in your lab. See `[15]`_ for details. +The deployment of the test API has been automated. +A jenkins job manages: + * the unit tests of the test api + * the cration of a new docker file + * the deployment of the new test api + * the archive of the old test api + * the backup of the Mongo DB + + Automatic reporting + =================== + + An automatic reporting page has been created in order to provide a + consistant view of the scenarios. + In this page, each scenario is evaluated according to test criteria. + The code for the automatic reporting is available at `[8]`_. + + The results are collected from the centralized database every day and, + per scenario. A score is calculated based on the results from the last + 10 days. This score is the addition of single test scores. Each test + case has a success criteria reflected in the criteria field from the + results. + + Considering an instance of a scenario os-odl_l2-nofeature-ha, the + scoring is the addition of the scores of all the runnable tests from the + categories (tiers healthcheck, smoke and features) + corresponding to this scenario. + + + +---------------------+---------+---------+---------+---------+ + | Test | Apex | Compass | Fuel | Joid | + +=====================+=========+=========+=========+=========+ + | vPing_ssh | X | X | X | X | + +---------------------+---------+---------+---------+---------+ + | vPing_userdata | X | X | X | X | + +---------------------+---------+---------+---------+---------+ + | tempest_smoke_serial| X | X | X | X | + +---------------------+---------+---------+---------+---------+ + | rally_sanity | X | X | X | X | + +---------------------+---------+---------+---------+---------+ + | odl | X | X | X | X | + +---------------------+---------+---------+---------+---------+ + | promise | | | X | X | + +---------------------+---------+---------+---------+---------+ + | doctor | X | | X | | + +---------------------+---------+---------+---------+---------+ + | security_scan | X | | | | + +---------------------+---------+---------+---------+---------+ + | parser | | | X | | + +---------------------+---------+---------+---------+---------+ + | moon | | X | | | + +---------------------+---------+---------+---------+---------+ + | copper | X | | | X | + +---------------------+---------+---------+---------+---------+ + + All the testcases listed in the table are runnable on + os-odl_l2-nofeature scenarios. + If no result is available or if all the results are failed, the test + case get 0 point. + If it was succesfull at least once but not anymore during the 4 runs, + the case get 1 point (it worked once). + If at least 3 of the last 4 runs were successful, the case get 2 points. + If the last 4 runs of the test are successful, the test get 3 points. + + In the example above, the target score for fuel/os-odl_l2-nofeature-ha + is 3x6 = 18 points. + + The scenario is validated per installer when we got 3 points for all + individual test cases (e.g 18/18). + Please note that complex or long duration tests are not considered for + the scoring. The success criteria are not always easy to define and may + require specific hardware configuration. These results however provide + a good level of trust on the scenario. + + A web page is automatically generated every day to display the status. + This page can be found at `[9]`_. For the status, click on Status menu, + you may also get feedback for vims and tempest_smoke_serial test cases. + + Any validated scenario is stored in a local file on the web server. In + fact as we are using a sliding windows to get results, it may happen + that a successful scenarios is no more run (because considered as + stable) and then the number of iterations (4 needed) would not be + sufficient to get the green status. + + Please note that other test cases (e.g. sfc_odl, bgpvpn, moon) need also + ODL configuration addons and as a consequence specific scenario. + There are not considered as runnable on the generic odl_l2 scenario. + Dashboard ========= Dashboard is used to provide a consistant view of the results collected in CI. -The results with dashboard method are post-processed from raw results. -Please note that dashboard results are not stored. Only raw results are -stored. +The results showed on the dashboard are post processed from the Database, +which only contains raw results. -Release Brahmaputra -------------------- - -Dashboard url: http://testresults.opnfv.org/dashboard/ - -Release Colorado ----------------- +In Brahmaputra, we created a basic dashboard. Since Colorado, it was decided to adopt ELK framework. Mongo DB results are extracted to feed Elasticsearch database (`[7]`_). -Dashboard url: http://testresults.opnfv.org/kibana_dashboards/ - -Credentials for a guest account: opnfvuser/kibana - -A script has been developped to build elasticsearch data set. This +A script was developed to build elasticsearch data set. This script can be found in `[16]`_. +For next versions, it was decided to integrated bitergia dashboard. +Bitergia already provides a dashboard for code and infrastructure. +A new Test tab will be added. The dataset will be built by consuming +the test API. -Automatic reporting -=================== - -An automatic reporting page has been created in order to provide a -consistant view of the scenarios. -In this page each scenario is evaluated according to test criteria. -The code for the automatic reporting is available at `[8]`_. - -The results are collected from the centralized database every day and, -per scenario. A score is calculated based on the results from the last -50 days. This score is the addition of single test scores. Each test -case has a success criteria reflected in the criteria field from the -results. - -Considering an instance of a scenario os-odl_l2-nofeature-ha, the -scoring is the addition of the scores of all the runnable tests from the -categories (tiers healthcheck, smoke, controller and feature) -corresponding to this scenario. - - - +---------------------+---------+---------+---------+---------+ - | Test | Apex | Compass | Fuel | Joid | - +=====================+=========+=========+=========+=========+ - | vPing_ssh | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | vPing_userdata | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | tempest_smoke_serial| X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | rally_sanity | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | odl | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | promise | | | X | X | - +---------------------+---------+---------+---------+---------+ - | doctor | X | | X | | - +---------------------+---------+---------+---------+---------+ - | security_scan | X | | | | - +---------------------+---------+---------+---------+---------+ - | parser | | | X | | - +---------------------+---------+---------+---------+---------+ - | moon | | X | | | - +---------------------+---------+---------+---------+---------+ - | copper | X | | | X | - +---------------------+---------+---------+---------+---------+ - -All the testcases listed in the table are runnable on -os-odl_l2-nofeature scenarios. -If no result is available or if all the results are failed, the test -case get 0 point. -If it was succesfull at least once but not anymore during the 4 runs, -the case get 1 point (it worked once). -If at least 3 of the last 4 runs were successful, the case get 2 points. -If the last 4 runs of the test are successful, the test get 3 points. - -In the example above, the target score for fuel/os-odl_l2-nofeature-ha -is 3x6 = 18 points. - -The scenario is validated per installer when we got 3 points for all -individual test cases (e.g 18/18). -Please note that complex or long duration tests are not considered for -the scoring. The success criteria are not always easy to define and may -require specific hardware configuration. These results however provide -a good level of trust on the scenario. - -A web page is automatically generated every day to display the status. -This page can be found at `[9]`_. For the status, click on Status menu, -you may also get feedback for vims and tempest_smoke_serial test cases. - -Any validated scenario is stored in a local file on the web server. In -fact as we are using a sliding windows to get results, it may happen -that a successful scenarios is no more run (because considered as -stable) and then the number of iterations (4 needed) would not be -sufficient to get the green status. - -Please note that other test cases e.g. sfc_odl, bgpvpn, moon) need also -ODL configuration addons and as a consequence specific scenario. -There are not considered as runnable on the generic odl_l2 scenario. ======= How TOs @@ -776,7 +828,6 @@ repository. The Functest files to be modified are: * functest/docker/Dockerfile: get your code in Functest container * functest/ci/testcases.yaml: reference your test and its associated constraints - * functest/ci/exec_test.sh: run your test from CI and CLI Dockerfile @@ -810,25 +861,6 @@ If you are integrating test suites from a feature project, the default category is **features**. -exec_test.sh ------------- - -This file is used to start your test. It is used in CI but also by the -CLI. - -You just patch the file in git and add a line:: - - - "<my_super_test_case>") - python ${FUNCTEST_REPO_DIR}/testcases/features/mycase.py - ;; - - -Note you can use python or bash scripts (or any language assuming that -the packages have been properly preinstalled but we recommand python or -bash..). - - How to select my list of tests for CI? ====================================== @@ -841,6 +873,9 @@ Functest jenkins job):: Each case can be configured as daily and/or weekly task. +Weekly tasks are used for long duration or experimental tests. +Daily tasks correspond to the minimum set of test suites to validate a scenario. + When executing run_tests.py, a check based on the jenkins build tag will be considered to detect whether it is a daily and/or a weekly test. @@ -854,8 +889,8 @@ e.g.:: cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t healthcheck,smoke ${flags}" This command will run all the test cases of the first 2 tiers, i.e. -healthcheck, vping_ssh, vping_userdata, tempest_smoke_serial and -rally_sanity. +healthcheck, connection_check, api_check, vping_ssh, vping_userdata, +snaps_somke, tempest_smoke_serial and rally_sanity. How to push your results into the Test Database @@ -938,6 +973,4 @@ _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_varia _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html -_`config_test.py` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.py - _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml |