From 83714aaa9a09e7df27e06d5d95dadc89a8e67f9e Mon Sep 17 00:00:00 2001 From: "jose.lausuch" Date: Tue, 21 Feb 2017 15:21:45 +0100 Subject: Restructure docs DOCS directory restructured according to: https://wiki.opnfv.org/display/DOC/Documentation+Guide Now: - release - release notes - testing - developer - dev guide (this is to be released on docs.opnfv.org) - internship (docs about intern projects) - user (this is to be released on docs.opnfv.org) - config guide - user guide Change-Id: I1851189601aac3c5989f19f99d779efe23dbf3d1 Signed-off-by: jose.lausuch --- docs/configguide/configguide.rst | 821 ----------------- docs/configguide/index.rst | 296 ------ docs/devguide/index.rst | 993 --------------------- docs/internship/security_group/index.rst | 70 -- docs/internship/testapi_evolution/index.rst | 237 ----- docs/internship/unit_tests/index.rst | 70 -- docs/internship/vnf_catalog/index.rst | 170 ---- docs/release-notes/functest-release.rst | 637 ------------- docs/release-notes/index.rst | 15 - docs/release/release-notes/functest-release.rst | 637 +++++++++++++ docs/release/release-notes/index.rst | 15 + docs/testing/developer/devguide/index.rst | 993 +++++++++++++++++++++ .../developer/internship/security_group/index.rst | 70 ++ .../internship/testapi_evolution/index.rst | 237 +++++ .../developer/internship/unit_tests/index.rst | 70 ++ .../developer/internship/vnf_catalog/index.rst | 170 ++++ docs/testing/user/configguide/configguide.rst | 821 +++++++++++++++++ docs/testing/user/configguide/index.rst | 296 ++++++ docs/testing/user/userguide/index.rst | 548 ++++++++++++ docs/testing/user/userguide/introduction.rst | 255 ++++++ docs/testing/user/userguide/runfunctest.rst | 385 ++++++++ docs/testing/user/userguide/troubleshooting.rst | 387 ++++++++ docs/userguide/index.rst | 548 ------------ docs/userguide/introduction.rst | 255 ------ docs/userguide/runfunctest.rst | 385 -------- docs/userguide/troubleshooting.rst | 387 -------- 26 files changed, 4884 insertions(+), 4884 deletions(-) delete mode 100644 docs/configguide/configguide.rst delete mode 100644 docs/configguide/index.rst delete mode 100644 docs/devguide/index.rst delete mode 100644 docs/internship/security_group/index.rst delete mode 100644 docs/internship/testapi_evolution/index.rst delete mode 100644 docs/internship/unit_tests/index.rst delete mode 100644 docs/internship/vnf_catalog/index.rst delete mode 100644 docs/release-notes/functest-release.rst delete mode 100644 docs/release-notes/index.rst create mode 100644 docs/release/release-notes/functest-release.rst create mode 100644 docs/release/release-notes/index.rst create mode 100644 docs/testing/developer/devguide/index.rst create mode 100644 docs/testing/developer/internship/security_group/index.rst create mode 100644 docs/testing/developer/internship/testapi_evolution/index.rst create mode 100644 docs/testing/developer/internship/unit_tests/index.rst create mode 100644 docs/testing/developer/internship/vnf_catalog/index.rst create mode 100644 docs/testing/user/configguide/configguide.rst create mode 100644 docs/testing/user/configguide/index.rst create mode 100644 docs/testing/user/userguide/index.rst create mode 100644 docs/testing/user/userguide/introduction.rst create mode 100644 docs/testing/user/userguide/runfunctest.rst create mode 100644 docs/testing/user/userguide/troubleshooting.rst delete mode 100644 docs/userguide/index.rst delete mode 100644 docs/userguide/introduction.rst delete mode 100644 docs/userguide/runfunctest.rst delete mode 100644 docs/userguide/troubleshooting.rst diff --git a/docs/configguide/configguide.rst b/docs/configguide/configguide.rst deleted file mode 100644 index 08e089c2b..000000000 --- a/docs/configguide/configguide.rst +++ /dev/null @@ -1,821 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - - -Pulling the Docker image ------------------------- -Pull the Functest Docker image ('opnfv/functest') from the public -dockerhub registry under the OPNFV account: [dockerhub_], with the -following docker command:: - - docker pull opnfv/functest: - -where identifies a release of the Functest docker -container image in the public dockerhub registry. There are many tags -created automatically by the CI mechanisms, and you must ensure you -pull an image with the **correct tag** to match the OPNFV software -release installed in your environment. All available tagged images can -be seen from location [FunctestDockerTags_]. For example, when running -on the first official release of the OPNFV Colorado system platform, -tag "colorado.1.0" is needed. Pulling other tags might cause some -problems while running the tests. -Docker images pulled without a tag specifier bear the implicitly -assigned label "latest". If you need to specifically pull the latest -Functest docker image, then omit the tag argument:: - - docker pull opnfv/functest - -After pulling the Docker image, check that it is available with the -following docker command:: - - [functester@jumphost ~]$ docker images - REPOSITORY TAG IMAGE ID CREATED SIZE - opnfv/functest latest 8cd6683c32ae 2 weeks ago 1.321 GB - opnfv/functest danube.1.0 13fa54a1b238 4 weeks ago 1.29 GB - opnfv/functest colorado.1.0 94b78faa94f7 9 weeks ago 968 MB - -The Functest docker container environment can -in principle- be also -used with non-OPNFV official installers (e.g. 'devstack'), with the -**disclaimer** that support for such environments is outside of the -scope and responsibility of the OPNFV project. - -Accessing the Openstack credentials ------------------------------------ -OpenStack credentials are mandatory and can be retrieved in different -ways. From inside the running Functest docker container the -"functest env prepare" command will automatically look for the -Openstack credentials file "/home/opnfv/functest/conf/openstack.creds" -and retrieve it unless the file already exists. This Functest -environment preparation step is described later in this document. - -WARNING: When the installer type is "joid" you have to have the -credentials file inside the running container **before** initiating the -functest environment preparation. For that reason you have to choose -either one of the options below, since the automated copying does not -work for "joid". - -You can also specifically pass in the needed file prior to running the -environment preparation either: - - * by using the -v option when creating the Docker container. This is - referred to in docker documentation as "Bind Mounting". See the - usage of this parameter in the following chapter. - * or creating a local file '/home/opnfv/functest/conf/openstack.creds' - inside the running container with the credentials in it. Consult - your installer guide for further details. This is however not - instructed in this document. - -NOTE: When the installer type is "fuel" and virtualized deployment -is used, there you have to explicitly fetch the credentials file -executing the following sequence - - #. Create a container as described in next chapter but do not - "Bind Mount" the credentials - #. Log in to container and execute the following command. Replace - the IP with installer address after the "-a" parameter:: - - $REPOS_DIR/releng/utils/fetch_os_creds.sh \ - -d /home/opnfv/functest/conf/openstack.creds \ - -i fuel \ - -a 10.20.0.2 \ - -v - ( -d specifies the full path to the Openstack credential file - -i specifies the INSTALLER_TYPE - -a specifies the INSTALLER_IP - -v indicates a virtualized environment and takes no arguments ) - - #. Continue with your testing, initiate functest environment - preparation, run tests etc. - -In proxified environment you may need to change the credentials file. -There are some tips in chapter: `Proxy support`_ - -Functest Docker parameters --------------------------- -This chapter explains how to run a container for executing functest -test suites. Numbered list below explains some details of the -recommended parameters for invoking docker container - - #. It is a good practice to assign a precise container name through - the **--name** option. - - #. Assign parameter for installer type:: - - -e "INSTALLER_TYPE=" - # Use one of following apex, compass, fuel or joid - - #. Functest needs to know the IP of some installers:: - - -e "INSTALLER_IP=" - - This IP is needed to fetch RC file from deployment, fetch logs, ... - If not provided, there is no way to fetch the RC file. It must be - provided manually as a volume - - #. Credentials for accessing the Openstack. - Most convenient way of passing them to container is by having a - local copy of the credentials file in Jumphost and then using the - **-v** option. In the example we have local file by the name of - "overcloudrc" and we are using that as an argument:: - - -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds - - The credentials file needs to exist in the Docker container - under the path: '/home/opnfv/functest/conf/openstack.creds'. - - **WARNING:** If you are using the Joid installer, you must pass the - credentials using the **-v** option: - -v /var/lib/jenkins/admin-openrc:/home/opnfv/functest/conf/openstack.creds. - See the section `Accessing the Openstack credentials`_ above. - - #. Passing deployment scenario - When running Functest against any of the supported OPNFV scenarios, - it is recommended to include also the environment variable - **DEPLOY_SCENARIO**. The **DEPLOY_SCENARIO** environment variable - is passed with the format:: - - -e "DEPLOY_SCENARIO=os---" - where: - os = OpenStack (No other VIM choices currently available) - controller is one of ( nosdn | odl_l2 | odl_l3 | onos | ocl) - nfv_feature is one or more of ( ovs | kvm | sfc | bgpvpn | nofeature ) - If several features are pertinent then use the underscore - character '_' to separate each feature (e.g. ovs_kvm) - 'nofeature' indicates no NFV feature is deployed - ha_mode (high availability) is one of ( ha | noha ) - - **NOTE:** Not all possible combinations of "DEPLOY_SCENARIO" are - supported. The name passed in to the Functest Docker container - must match the scenario used when the actual OPNFV platform was - deployed. See release note to see the list of supported scenarios. - - **NOTE:** The scenario name is mainly used to automatically detect - if a test suite is runnable or not (e.g. it will prevent ONOS test suite - to be run on ODL scenarios). If not set, Functest will try to run the - default test cases that might not include SDN controller or a specific - feature - - **NOTE:** A HA scenario means that 3 OpenStack controller nodes are - deployed. It does not necessarily mean that the whole system is HA. See - installer release notes for details. - - -Putting all above together, when using installer 'fuel' and an invented -INSTALLER_IP of '10.20.0.2', the recommended command to create the -Functest Docker container is as follows:: - - docker run --name "FunctestContainer" -it \ - -e "INSTALLER_IP=10.20.0.2" \ - -e "INSTALLER_TYPE=fuel" \ - -e "DEPLOY_SCENARIO=os-odl_l2-ovs_kvm-ha" \ - -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds \ - opnfv/functest /bin/bash - -After the *run* command, a new prompt appears which means that we are inside -the container and ready to move to the next step. - -For tips on how to set up container with installer Apex, see chapter -`Apex Installer Tips`_. - -Finally, three additional environment variables can also be passed in -to the Functest Docker Container, using the -e -"=" mechanism. The first two of these are -only relevant to Jenkins CI invoked testing and **should not be used** -when performing manual test scenarios:: - - -e "NODE_NAME=" \ - -e "BUILD_TAG=" \ - -e "CI_DEBUG=" - where: - = Symbolic name of the POD where the tests are run. - Visible in test results files, which are stored - to the database. This option is only used when - tests are activated under Jenkins CI control. - It indicates the POD/hardware where the test has - been run. If not specified, then the POD name is - defined as "Unknown" by default. - DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS. - = Symbolic name of the Jenkins Build Job. - Visible in test results files, which are stored - to the database. This option is only set when - tests are activated under Jenkins CI control. - It enables the correlation of test results, - which - are independently pushed to the results datbase - from different Jenkins jobs. - DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS. - = "true" or "false" - Default = "false", if not specified - If "true" is specified, then additional debug trace - text can be sent to the test results file / log files - and also to the standard console output. - -Apex Installer Tips -------------------- -Some specific tips are useful for the Apex Installer case. If not using -Apex Installer; ignore this section. - -In case of Triple-O based installer (like Apex) the docker container -needs to connect to the installer VM, so it is then required that some -known SSH keys are present in docker container. Since the Jumphost root -SSH keys are already known, easiest way is to use those using the -'Bind mount' method. See below for sample parameter:: - - -v /root/.ssh/id_rsa:/root/.ssh/id_rsa - - NOTE: You need the "sudo" when creating the container to access root - users ssh credentials even the docker command itself might not - require that. - -HINT! In case of Triple-O installers you can find value for the -INSTALLER_IP parameter by executing command and note the returned IP -address:: - - inst=$(sudo virsh list | grep -iEo "undercloud|instack") - sudo virsh domifaddr ${inst} - - NOTE: In releases prior to Colorado, the name 'instack' was - used. Currently the name 'undercloud' is used. - -You can copy the credentials file from the "stack" users home directory -in installer VM to Jumphost. Please check the correct IP from the -command above. In the example below we are using invented IP address -"192.168.122.89":: - - scp stack@192.168.122.89:overcloudrc . - -Here is an example of the full docker command invocation for an Apex -installed system, using latest Functest docker container, for -illustration purposes:: - - sudo docker run -it --name "ApexFuncTestODL" \ - -e "INSTALLER_IP=192.168.122.89" \ - -e "INSTALLER_TYPE=apex" \ - -e "DEPLOY_SCENARIO=os-odl_l2-nofeature-ha" \ - -v /root/.ssh/id_rsa:/root/.ssh/id_rsa \ - -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds \ - opnfv/functest /bin/bash - -Compass installer local development env usage Tips --------------------------------------------------- -In the compass-functest local test case check and development environment, -in order to get openstack service inside the functest container, some -parameters should be configured during container creation, which are -hard to guess for freshman. This section will provide the guideline, the -parameters values are defaults here, which should be adjusted according -to the settings, the complete steps are given here so as not to appear -too abruptly. - -1, Pull Functest docker image from public dockerhub:: - - docker pull opnfv/functest: - - here can be "brahmaputra.1.0", "colorado.1.0", etc. -Tag omitted means the latest docker image:: - - docker pull opnfv/functest - -2, Functest Docker container creation - -To make a file used for the environment, such as 'functest-docker-env':: - - OS_AUTH_URL=http://172.16.1.222:35357/v2.0 - OS_USERNAME=admin - OS_PASSWORD=console - OS_TENANT_NAME=admin - OS_VOLUME_API_VERSION=2 - OS_PROJECT_NAME=admin - INSTALLER_TYPE=compass - INSTALLER_IP=192.168.200.2 - EXTERNAL_NETWORK=ext-net - -Note: please adjust the content according to the environment, such as -'TENANT_ID' maybe used for some special cases. - -Then to create the Functest docker:: - - docker run --privileged=true --rm -t \ - --env-file functest-docker-env \ - --name \ - opnfv/functest: /bin/bash - -3, To attach Functest container - -Before trying to attach the Functest container, the status can be checked by:: - - docker ps -a - -to attach the 'Up' status Functest container and start bash mode:: - - docker exec -it bash - -4, Functest environemnt preparation and check - -To see the Section below `Preparing the Functest environment`_. - -Functest docker container directory structure ---------------------------------------------- -Inside the Functest docker container, the following directory structure -should now be in place:: - - `-- home - `-- opnfv - |-- functest - | |-- conf - | |-- data - | `-- results - `-- repos - |-- bgpvpn - |-- copper - |-- doctor - |-- domino - |-- functest - |-- kingbird - |-- odl_test - |-- onos - |-- ovno - |-- parser - |-- promise - |-- rally - |-- releng - |-- sdnvpn - |-- securityscanning - |-- sfc - |-- tempest - |-- vims_test - `-- vnfs - -Underneath the '/home/opnfv/' directory, the Functest docker container -includes two main directories: - - * The **functest** directory stores configuration files (e.g. the - OpenStack creds are stored in path '/home/opnfv/functest/conf/openstack.creds'), - the **data** directory stores a 'cirros' test image used in some - functional tests and the **results** directory stores some temporary - result log files - * The **repos** directory holds various repositories. The directory - '/home/opnfv/repos/functest' is used to prepare the needed Functest - environment and to run the tests. The other repository directories - are used for the installation of the needed tooling (e.g. rally) or - for the retrieval of feature projects scenarios (e.g. promise) - -The structure under the **functest** repository can be described as -follows:: - - . |-- INFO - |-- LICENSE - |-- requirements.txt - |-- run_unit_tests.sh - |-- setup.py - |-- test-requirements.txt - |-- commons - | |-- ims - | |-- mobile - | `--traffic-profile-guidelines.rst - |-- docker - | |-- Dockerfile - | |-- config_install_env.sh - | `-- docker_remote_api - |-- docs - | |-- com - | |-- configguide - | |-- devguide - | |-- images - | |-- internship - | |-- release-notes - | |-- results - | `--userguide - |-- functest - |-- __init__.py - |-- ci - | |-- __init__.py - | |-- check_os.sh - | |-- config_functest.yaml - | |-- config_patch.yaml - | |-- exec_test.sh - | |-- generate_report.py - | |-- prepare_env.py - | |-- run_tests.py - | |-- testcases.yaml - | |-- tier_builder.py - | `-- tier_handler.py - |-- cli - | |-- __init__.py - | |-- cli_base.py - | |-- commands - | |-- functest-complete.sh - | `-- setup.py - |-- core - | |-- __init__.py - | |-- feature_base.py - | |-- pytest_suite_runner.py - | |-- testcase_base.py - | |-- vnf_base.py - |-- opnfv_tests - | |-- __init__.py - | |-- features - | |-- mano - | |-- openstack - | |-- sdn - | |-- security_scan - | `-- vnf - |-- tests - | |-- __init__.py - | `-- unit - `-- utils - |-- __init__.py - |-- config.py - |-- constants.py - |-- env.py - |-- functest_constants.py - |-- functest_logger.py - |-- functest_utils.py - |-- openstack - |-- openstack_clean.py - |-- openstack_snapshot.py - |-- openstack_tacker.py - `-- openstack_utils.py - - - (Note: All *.pyc files removed from above list for brevity...) - -We may distinguish several directories, the first level has 4 directories: - -* **commons**: This directory is dedicated for storage of traffic - profile or any other test inputs that could be reused by any test - project. -* **docker**: This directory includes the needed files and tools to - build the Funtest Docker image. -* **docs**: This directory includes documentation: Release Notes, - User Guide, Configuration Guide and Developer Guide. Test results - are also located in a sub--directory called 'results'. -* **functest**: This directory contains all the code needed to run - functest internal cases and OPNFV onboarded feature or VNF test cases. - -Functest directory has 6 directories: - * **ci**: This directory contains test structure definition files - (e.g .yaml) and bash shell/python scripts used to - configure and execute Functional tests. The test execution script - can be executed under the control of Jenkins CI jobs. - * **cli**: This directory holds the python based Functest CLI utility - source code, which is based on the Python 'click' framework. - * **core**: This directory holds the python based Functest core - source code. Three abstraction classes have been created to ease - the integration of internal, feature or vnf cases. - * **opnfv_tests**: This directory includes the scripts required by - Functest internal test cases and other feature projects test cases. - * **tests**: This directory includes the functest unit tests - * **utils**: this directory holds Python source code for some general - purpose helper utilities, which testers can also re-use in their - own test code. See for an example the Openstack helper utility: - 'openstack_utils.py'. - -Useful Docker commands ----------------------- -When typing **exit** in the container prompt, this will cause exiting -the container and probably stopping it. When stopping a running Docker -container all the changes will be lost, there is a keyboard shortcut -to quit the container without stopping it: -P + -Q. To -reconnect to the running container **DO NOT** use the *run* command -again (since it will create a new container), use the *exec* or *attach* -command instead:: - - docker ps # - docker exec -ti /bin/bash - -There are other useful Docker commands that might be needed to manage possible -issues with the containers. - -List the running containers:: - - docker ps - -List all the containers including the stopped ones:: - - docker ps -a - -Start a stopped container named "FunTest":: - - docker start FunTest - -Attach to a running container named "StrikeTwo":: - - docker attach StrikeTwo - -It is useful sometimes to remove a container if there are some problems:: - - docker rm - -Use the *-f* option if the container is still running, it will force to -destroy it:: - - docker rm -f - -Check the Docker documentation dockerdocs_ for more information. - -Preparing the Functest environment ----------------------------------- -Once the Functest docker container is up and running, the required -Functest environment needs to be prepared. A custom built **functest** -CLI utility is available to perform the needed environment preparation -action. Once the environment is prepared, the **functest** CLI utility -can be used to run different functional tests. The usage of the -**functest** CLI utility to run tests is described further in the -Functest User Guide `OPNFV_FuncTestUserGuide`_ - -Prior to commencing the Functest environment preparation, we can check -the initial status of the environment. Issue the **functest env status** -command at the prompt:: - - functest env status - Functest environment is not installed. - - Note: When the Functest environment is prepared, the command will - return the status: "Functest environment ready to run tests." - -To prepare the Functest docker container for test case execution, issue -the **functest env prepare** command at the prompt:: - - functest env prepare - -This script will make sure that the requirements to run the tests are -met and will install the needed libraries and tools by all Functest -test cases. It should be run only once every time the Functest docker -container is started from scratch. If you try to run this command, on -an already prepared enviroment, you will be prompted whether you really -want to continue or not:: - - functest env prepare - It seems that the environment has been already prepared. - Do you want to do it again? [y|n] - - (Type 'n' to abort the request, or 'y' to repeat the - environment preparation) - - -To list some basic information about an already prepared Functest -docker container environment, issue the **functest env show** at the -prompt:: - - functest env show - +======================================================+ - | Functest Environment info | - +======================================================+ - | INSTALLER: apex, 192.168.122.89 | - | SCENARIO: os-odl_l2-nofeature-ha | - | POD: localhost | - | GIT BRANCH: master | - | GIT HASH: 5bf1647dec6860464eeb082b2875798f0759aa91 | - | DEBUG FLAG: false | - +------------------------------------------------------+ - | STATUS: ready | - +------------------------------------------------------+ - - Where: - - INSTALLER: Displays the INSTALLER_TYPE value - - here = "apex" - and the INSTALLER_IP value - - here = "192.168.122.89" - SCENARIO: Displays the DEPLOY_SCENARIO value - - here = "os-odl_l2-nofeature-ha" - POD: Displays the value passed in NODE_NAME - - here = "localhost" - GIT BRANCH: Displays the git branch of the OPNFV Functest - project repository included in the Functest - Docker Container. - - here = "master" - (In first official colorado release - would be "colorado.1.0") - GIT HASH: Displays the git hash of the OPNFV Functest - project repository included in the Functest - Docker Container. - - here = "5bf1647dec6860464eeb082b2875798f0759aa91" - DEBUG FLAG: Displays the CI_DEBUG value - - here = "false" - - NOTE: In Jenkins CI runs, an additional item "BUILD TAG" - would also be listed. The valaue is set by Jenkins CI. - -Finally, the **functest** CLI has a **--help** options: - -Some examples:: - - functest --help Usage: functest [OPTIONS] COMMAND [ARGS]... - - Options: - --version Show the version and exit. - -h, --help Show this message and exit. - - Commands: - env - openstack - testcase - tier - - functest env --help - Usage: functest env [OPTIONS] COMMAND [ARGS]... - - Options: - -h, --help Show this message and exit. - - Commands: - prepare Prepares the Functest environment. - show Shows information about the current... - status Checks if the Functest environment is ready... - -Checking Openstack and credentials ----------------------------------- -It is recommended and fairly straightforward to check that Openstack -and credentials are working as expected. - -Once the credentials are there inside the container, they should be -sourced before running any Openstack commands:: - - source /home/opnfv/functest/conf/openstack.creds - -After this, try to run any OpenStack command to see if you get any -output, for instance:: - - openstack user list - -This will return a list of the actual users in the OpenStack -deployment. In any other case, check that the credentials are sourced:: - - env|grep OS_ - -This command must show a set of environment variables starting with -*OS_*, for example:: - - OS_REGION_NAME=RegionOne - OS_DEFAULT_DOMAIN=default - OS_PROJECT_NAME=admin - OS_PASSWORD=admin - OS_AUTH_STRATEGY=keystone - OS_AUTH_URL=http://172.30.10.3:5000/v2.0 - OS_USERNAME=admin - OS_TENANT_NAME=admin - OS_ENDPOINT_TYPE=internalURL - OS_NO_CACHE=true - -If the OpenStack command still does not show anything or complains -about connectivity issues, it could be due to an incorrect url given to -the OS_AUTH_URL environment variable. Check the deployment settings. - -SSL Support ------------ -If you need to connect to a server that is TLS-enabled (the auth URL -begins with "https") and it uses a certificate from a private CA or a -self-signed certificate, then you will need to specify the path to an -appropriate CA certificate to use, to validate the server certificate -with the environment variable OS_CACERT:: - - echo $OS_CACERT - /etc/ssl/certs/ca.crt - -However, this certificate does not exist in the container by default. -It has to be copied manually from the OpenStack deployment. This can be -done in 2 ways: - - #. Create manually that file and copy the contents from the OpenStack - controller. - #. (Recommended) Add the file using a Docker volume when starting the - container:: - - -v :/etc/ssl/certs/ca.cert - -You might need to export OS_CACERT environment variable inside the -container:: - - export OS_CACERT=/etc/ssl/certs/ca.crt - -Certificate verification can be turned off using OS_INSECURE=true. For -example, Fuel uses self-signed cacerts by default, so an pre step would -be:: - - export OS_INSECURE=true - -Proxy support -------------- -If your Jumphost node is operating behind a http proxy, then there are -2 places where some special actions may be needed to make operations -succeed: - - #. Initial installation of docker engine First, try following the - official Docker documentation for Proxy settings. Some issues were - experienced on CentOS 7 based Jumphost. Some tips are documented - in section: `Docker Installation on CentOS behind http proxy`_ - below. - - #. Execution of the Functest environment preparation inside the - created docker container Functest needs internet access to - download some resources for some test cases. This might not - work properly if the Jumphost is connecting to internet - through a http Proxy. - -If that is the case, make sure the resolv.conf and the needed -http_proxy and https_proxy environment variables, as well as the -'no_proxy' environment variable are set correctly:: - - # Make double sure that the 'no_proxy=...' line in the - # 'openstack.creds' file is commented out first. Otherwise, the - # values set into the 'no_proxy' environment variable below will - # be ovewrwritten, each time the command - # 'source ~/functest/conf/openstack.creds' is issued. - - cd ~/functest/conf/ - sed -i 's/export no_proxy/#export no_proxy/' openstack.creds - source ./openstack.creds - - # Next calculate some IP addresses for which http_proxy - # usage should be excluded: - - publicURL_IP=$(echo $OS_AUTH_URL | grep -Eo "([0-9]+\.){3}[0-9]+") - - adminURL_IP=$(openstack catalog show identity | \ - grep adminURL | grep -Eo "([0-9]+\.){3}[0-9]+") - - export http_proxy="" - export https_proxy="" - export no_proxy="127.0.0.1,localhost,$publicURL_IP,$adminURL_IP" - - # Ensure that "git" uses the http_proxy - # This may be needed if your firewall forbids SSL based git fetch - git config --global http.sslVerify True - git config --global http.proxy - -Validation check: Before running **'functest env prepare'** CLI command, -make sure you can reach http and https sites from inside the Functest -docker container. - -For example, try to use the **nc** command from inside the functest -docker container:: - - nc -v opnfv.org 80 - Connection to opnfv.org 80 port [tcp/http] succeeded! - - nc -v opnfv.org 443 - Connection to opnfv.org 443 port [tcp/https] succeeded! - -Note: In a Jumphost node based on the CentOS family OS, the **nc** -commands might not work. You can use the **curl** command instead. - - curl http://www.opnfv.org:80 - - - curl https://www.opnfv.org:443 - - - (Ignore the content. If command returns a valid HTML page, it proves - the connection.) - -Docker Installation on CentOS behind http proxy ------------------------------------------------ -This section is applicable for CentOS family OS on Jumphost which -itself is behind a proxy server. In that case, the instructions below -should be followed **before** installing the docker engine:: - - 1) # Make a directory '/etc/systemd/system/docker.service.d' - # if it does not exist - sudo mkdir /etc/systemd/system/docker.service.d - - 2) # Create a file called 'env.conf' in that directory with - # the following contents: - [Service] - EnvironmentFile=-/etc/sysconfig/docker - - 3) # Set up a file called 'docker' in directory '/etc/sysconfig' - # with the following contents: - HTTP_PROXY="" - HTTPS_PROXY="" - http_proxy="${HTTP_PROXY}" - https_proxy="${HTTPS_PROXY}" - - 4) # Reload the daemon - systemctl daemon-reload - - 5) # Sanity check - check the following docker settings: - systemctl show docker | grep -i env - - Expected result: - ---------------- - EnvironmentFile=/etc/sysconfig/docker (ignore_errors=yes) - DropInPaths=/etc/systemd/system/docker.service.d/env.conf - -Now follow the instructions in [`InstallDockerCentOS`_] to download -and install the **docker-engine**. The instructions conclude with a -"test pull" of a sample "Hello World" docker container. This should now -work with the above pre-requisite actions. - -.. _dockerdocs: https://docs.docker.com/ -.. _dockerhub: https://hub.docker.com/r/opnfv/functest/ -.. _Proxy: https://docs.docker.com/engine/admin/systemd/#http-proxy -.. _FunctestDockerTags: https://hub.docker.com/r/opnfv/functest/tags/ -.. _InstallDockerCentOS: https://docs.docker.com/engine/installation/linux/centos/ -.. _OPNFV_FuncTestUserGuide: http://artifacts.opnfv.org/functest/docs/userguide/index.html diff --git a/docs/configguide/index.rst b/docs/configguide/index.rst deleted file mode 100644 index f12739e31..000000000 --- a/docs/configguide/index.rst +++ /dev/null @@ -1,296 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - -================================== -OPNFV FUNCTEST Configuration Guide -================================== - -.. toctree:: - :numbered: - :maxdepth: 2 - -Version history -=============== - -+------------+----------+------------------+----------------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+----------------------------------+ -| 2016-08-17 | 1.0.0 | Juha Haapavirta | Colorado release | -| | | Column Gaynor | | -+------------+----------+------------------+----------------------------------+ -| 2017-01-19 | 1.0.1 | Morgan Richomme | Adaptations for Danube | -| | | | * update testcase list | -| | | | * update docker command | -+------------+----------+------------------+----------------------------------+ - -Introduction -============ -This document describes how to install and configure Functest in OPNFV. -The Functest CLI is utilized during the Functest environment preparation -step. The given example commands should work in both virtual and bare -metal cases alike. - -High level architecture ------------------------ - -The high level architecture of Functest within OPNFV can be described as -follows:: - - CIMC/Lights+out management Admin Mgmt/API Public Storage Private - PXE - + + + + + + - | | | | | | - | +----------------------------+ | | | | | - | | | | | | | | - +-----+ Jumphost | | | | | | - | | +--------+ | | | | - | | | | | | | | - | | +--------------------+ | | | | | | - | | | | | | | | | | - | | | Tools | +----------------+ | | | - | | | - Rally | | | | | | | - | | | - Robot | | | | | | | - | | | - TestON | | | | | | | - | | | | |-------------------------+ | | - | | | Testcases | | | | | | | - | | | - VIM | | | | | | | - | | | -- healthcheck | | | | | | | - | | | -- vPing_ssh | | | | | | | - | | | -- vPing_userdata | | | | | | | - | | | -- SNAPS_cases | | | | | | | - | | | -- Tempest_smoke | | | | | | | - | | | -- Rally_sanity | | | | | | | - | | | -- Tempest_full | | | | | | | - | | | -- Rally_full | | | | | | | - | | | | | | | | | | - | | | - SDN Controller | | | | | | | - | | | -- odl | | | | | | | - | | | -- onos | | | | | | | - | | | | | | | | | | - | | | - Features | | | | | | | - | | | | | | | | | | - | | | - VNF | | | | | | | - | | | | | | | | | | - | | +--------------------+ | | | | | | - | | Functest Docker + | | | | | - | | | | | | | | - | | | | | | | | - | | | | | | | | - | +----------------------------+ | | | | | - | | | | | | - | +----------------+ | | | | | - | | 1 | | | | | | - +----+ +--------------+-+ | | | | | - | | | 2 | | | | | | - | | | +--------------+-+ | | | | | - | | | | 3 | | | | | | - | | | | +--------------+-+ | | | | | - | | | | | 4 | | | | | | - | +-+ | | +--------------+-+ | | | | | - | | | | | 5 +-------------+ | | | | - | +-+ | | nodes for | | | | | | - | | | | deploying +---------------------+ | | | - | +-+ | OPNFV | | | | | | - | | | +------------------------------+ | | - | +-+ SUT | | | | | | - | | +--------------------------------------+ | - | | | | | | | | - | | +----------------------------------------------+ - | +----------------+ | | | | | - | | | | | | - + + + + + + - SUT = System Under Test - -All the libraries and dependencies needed by all of the Functest tools -are pre-installed into the Docker image. This allows running Functest -on any platform on any Operating System. - -The automated mechanisms inside the Functest Docker container will: - - * Retrieve OpenStack credentials - * Prepare the environment according to the SUT - * Perform the appropriate functional tests - * Push the test results into the OPNFV test result database - -This Docker image can be integrated into CI or deployed independently. - -Please note that the Functest Docker container has been designed for -OPNFV, however, it would be possible to adapt it to any VIM + controller -environment, since most of the test cases are integrated from upstream -communities. - -The functional test cases are described in the Functest User Guide `[2]`_ - - -Prerequisites -============= -The OPNFV deployment is out of the scope of this document but it can be -found in http://docs.opnfv.org. -The OPNFV platform is considered as the System Under Test (SUT) in this -document. - -Several prerequisites are needed for Functest: - - #. A Jumphost to run Functest on - #. A Docker daemon shall be installed on the Jumphost - #. A public/external network created on the SUT - #. An admin/management network created on the SUT - #. Connectivity from the Jumphost to the SUT public/external network - #. Connectivity from the Jumphost to the SUT admin/management network - -WARNING: Connectivity from Jumphost is essential and it is of paramount -importance to make sure it is working before even considering to install -and run Functest. Make also sure you understand how your networking is -designed to work. - -NOTE: **Jumphost** refers to any server which meets the previous -requirements. Normally it is the same server from where the OPNFV -deployment has been triggered previously. - -NOTE: If your Jumphost is operating behind a company http proxy and/or -Firewall, please consult first the section `Proxy Support`_, towards -the end of this document. The section details some tips/tricks which -*may* be of help in a proxified environment. - -Docker installation -------------------- -Docker installation and configuration is only needed to be done once -through the life cycle of Jumphost. - -If your Jumphost is based on Ubuntu, RHEL or CentOS linux, please -consult the references below for more detailed instructions. The -commands below are offered as a short reference. - -*Tip:* For running docker containers behind the proxy, you need first -some extra configuration which is described in section -`Docker Installation on CentOS behind http proxy`_. You should follow -that section before installing the docker engine. - -Docker installation needs to be done as root user. You may use other -userid's to create and run the actual containers later if so desired. -Log on to your Jumphost as root user and install the Docker Engine -(e.g. for CentOS family):: - - curl -sSL https://get.docker.com/ | sh - systemctl start docker - - *Tip:* If you are working through proxy, please set the https_proxy - environment variable first before executing the curl command. - -Add your user to docker group to be able to run commands without sudo:: - - sudo usermod -aG docker - -A reconnect is needed. There are 2 ways for this: - #. Re-login to your account - #. su - - -References - Installing Docker Engine on different Linux Operating Systems: - * Ubuntu_ - * RHEL_ - * CentOS_ - -.. _Ubuntu: https://docs.docker.com/engine/installation/linux/ubuntulinux/ -.. _RHEL: https://docs.docker.com/engine/installation/linux/rhel/ -.. _CentOS: https://docs.docker.com/engine/installation/linux/centos/ - -Public/External network on SUT ------------------------------- -Some of the tests against the VIM (Virtual Infrastructure Manager) need -connectivity through an existing public/external network in order to -succeed. This is needed, for example, to create floating IPs to access -VM instances through the public/external network (i.e. from the Docker -container). - -By default, the four OPNFV installers provide a fresh installation with -a public/external network created along with a router. Make sure that -the public/external subnet is reachable from the Jumphost. - -*Hint:* For the given OPNFV Installer in use, the IP sub-net address -used for the public/external network is usually a planning item and -should thus be known. Consult the OPNFV Configuration guide `[4]`_, and -ensure you can reach each node in the SUT, from the Jumphost using the -'ping' command using the respective IP address on the public/external -network for each node in the SUT. The details of how to determine the -needed IP addresses for each node in the SUT may vary according to the -used installer and are therefore ommitted here. - -Connectivity to Admin/Management network on SUT ------------------------------------------------ -Some of the Functest tools need to have access to the OpenStack -admin/management network of the controllers `[1]`_. - -For this reason, check the connectivity from the Jumphost to all the -controllers in cluster in the OpenStack admin/management network range. - -Installation and configuration -============================== - -.. include:: ./configguide.rst - -Integration in CI -================= -In CI we use the Docker image and execute the appropriate commands within the -container from Jenkins. - -Docker creation in set-functest-env builder `[3]`_:: - - envs="-e INSTALLER_TYPE=${INSTALLER_TYPE} -e INSTALLER_IP=${INSTALLER_IP} -e NODE_NAME=${NODE_NAME}" - [...] - docker pull opnfv/functest:$DOCKER_TAG >/dev/null - cmd="sudo docker run -id ${envs} ${volumes} ${custom_params} ${TESTCASE_OPTIONS} opnfv/functest:${DOCKER_TAG} /bin/bash" - echo "Functest: Running docker run command: ${cmd}" - ${cmd} >${redirect} - sleep 5 - container_id=$(docker ps | grep "opnfv/functest:${DOCKER_TAG}" | awk '{print $1}' | head -1) - echo "Container ID=${container_id}" - if [ -z ${container_id} ]; then - echo "Cannot find opnfv/functest container ID ${container_id}. Please check if it is existing." - docker ps -a - exit 1 - fi - echo "Starting the container: docker start ${container_id}" - docker start ${container_id} - sleep 5 - docker ps >${redirect} - if [ $(docker ps | grep "opnfv/functest:${DOCKER_TAG}" | wc -l) == 0 ]; then - echo "The container opnfv/functest with ID=${container_id} has not been properly started. Exiting..." - exit 1 - fi - - cmd="python ${FUNCTEST_REPO_DIR}/functest/ci/prepare_env.py start" - echo "Executing command inside the docker: ${cmd}" - docker exec ${container_id} ${cmd} - - -Test execution in functest-all builder `[3]`_:: - - branch=${GIT_BRANCH##*/} - echo "Functest: run $FUNCTEST_SUITE_NAME on branch ${branch}" - cmd="functest testcase run $FUNCTEST_SUITE_NAME" - fi - container_id=$(docker ps -a | grep opnfv/functest | awk '{print $1}' | head -1) - docker exec $container_id $cmd - ret_value=$? - exit $ret_value - -Docker clean in functest-cleanup builder `[3]`_ calling docker rm and docker rmi - - -References -========== -.. _`[1]`: https://ask.openstack.org/en/question/68144/keystone-unable-to-use-the-public-endpoint/ -.. _`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html -.. _`[3]`: https://git.opnfv.org/cgit/releng/tree/jjb/functest/functest-ci-jobs.yml -.. _`[4]`: http://artifacts.opnfv.org/functest/danube/docs/configguide/index.html - - -OPNFV main site: opnfvmain_. - -OPNFV functional test page: opnfvfunctest_. - -IRC support channel: #opnfv-functest - -.. _opnfvmain: http://www.opnfv.org -.. _opnfvfunctest: https://wiki.opnfv.org/functest diff --git a/docs/devguide/index.rst b/docs/devguide/index.rst deleted file mode 100644 index eee013678..000000000 --- a/docs/devguide/index.rst +++ /dev/null @@ -1,993 +0,0 @@ -****************************** -OPNFV FUNCTEST developer guide -****************************** - -.. toctree:: - :numbered: - :maxdepth: 2 - - -============ -Introduction -============ - -Functest is a project dealing with functional testing. -Functest produces its own internal test cases but can also be considered -as a framework to support feature and VNF onboarding project testing. -Functest developed a test API and defined a test collection framework -that can be used by any OPNFV project. - -Therefore there are many ways to contribute to Functest. You can: - - * Develop new internal test cases - * Integrate the tests from your feature project - * Develop the framework to ease the integration of external test cases - * Develop the API / Test collection framework - * Develop dashboards or automatic reporting portals - -This document describes how, as a developer, you may interact with the -Functest project. The first section details the main working areas of -the project. The Second part is a list of "How to" to help you to join -the Functest family whatever your field of interest is. - - -======================== -Functest developer areas -======================== - - -Functest High level architecture -================================ - -Functest is project delivering a test container dedicated to OPNFV. -It includes the tools, the scripts and the test scenarios. - -Functest can be described as follow:: - - +----------------------+ - | | - | +--------------+ | +-------------------+ - | | | | Public | | - | | Tools | +------------------+ OPNFV | - | | Scripts | | | System Under Test | - | | Scenarios | +------------------+ | - | | | | Management | | - | +--------------+ | +-------------------+ - | | - | Functest Docker | - | | - +----------------------+ - -Functest internal test cases -============================ -The internal test cases in Danube are: - - * healthcheck - * connection_check - * api_check - * vping_ssh - * vping_userdata - * odl - * snaps_smoke - * tempest_smoke_serial - * rally_sanity - * tempest_full_parallel - * rally_full - * cloudify_ims - -By internal, we mean that this particular test cases have been -developped and/or integrated by functest contributors and the associated -code is hosted in the Functest repository. -An internal case can be fully developped or a simple integration of -upstream suites (e.g. Tempest/Rally developped in OpenStack are just -integrated in Functest). -The structure of this repository is detailed in `[1]`_. -The main internal test cases are in the opnfv_tests subfolder of the -repository, the internal test cases are: - - * sdn: odl, onos - * openstack: healthcheck, vping_ssh, vping_userdata, tempest_*, rally_*, connection_check, api_check, snaps_smoke - * vnf: cloudify_ims - -If you want to create a new test case you will have to create a new -folder under the testcases directory. - -Functest external test cases -============================ -The external test cases are inherited from other OPNFV projects, -especially the feature projects. - -The external test cases are: - - * promise - * doctor - * onos - * bgpvpn - * copper - * security_scan - * sfc-odl - * sfc-onos - * parser - * domino - * multisite - * opera_ims - * orchestra_ims - - -The code to run these test cases may be directly in the repository of -the project. We have also a **features** sub directory under opnfv_tests -directory that may be used (it can be usefull if you want to reuse -Functest library). - -Functest framework -================== - -Functest can be considered as a framework. -Functest is release as a docker file, including tools, scripts and a CLI -to prepare the environement and run tests. -It simplifies the integration of external test suites in CI pipeline -and provide commodity tools to collect and display results. - -Since Colorado, test categories also known as tiers have been created to -group similar tests, provide consistant sub-lists and at the end optimize -test duration for CI (see How To section). - -The definition of the tiers has been agreed by the testing working group. - -The tiers are: - * healthcheck - * smoke - * features - * components - * performance - * vnf - * stress - -Functest abstraction classes -============================ - -In order to harmonize test integration, 3 abstraction classes have been -introduced in Danube: - - * testcase_base: base for any test case - * feature_base: abstraction for feature project - * vnf_base: abstraction for vnf onboarding - -The goal is to unify the way to run test from Functest. - -feature_base and vnf_base inherit from testcase_base. - - +-----------------------------------------+ - | | - | Testcase_base | - | | - | - init() | - | - run() | - | - publish_report() | - | - check_criteria() | - | | - +-----------------------------------------+ - | | - V V - +--------------------+ +--------------------------+ - | | | | - | feature_base | | vnf_base | - | | | | - | - prepare() | | - prepare() | - | - post() | | - deploy_orchestrator() | - | - parse_results() | | - deploy_vnf() | - | | | - test_vnf() | - | | | - clean() | - | | | - execute() | - | | | | - +--------------------+ +--------------------------+ - - -Functest util classes -===================== - -In order to simplify the creation of test cases, Functest develops some -functions that can be used by any feature or internal test cases. -Several features are supported such as logger, configuration management and -Openstack capabilities (snapshot, clean, tacker,..). -These functions can be found under /functest/utils and can be described as -follows: - -functest/utils/ -|-- config.py -|-- constants.py -|-- env.py -|-- functest_constants.py -|-- functest_logger.py -|-- functest_utils.py -|-- openstack_clean.py -|-- openstack_snapshot.py -|-- openstack_tacker.py -`-- openstack_utils.py - -Note that for Openstack, keystone v3 is now deployed by default by compass, -fuel and joid in Danube. All installers still support keysone v2 (deprecated in -next version). - -Test collection framework -========================= - -The OPNFV testing group created a test collection database to collect -the test results from CI: - - - http://testresults.opnfv.org/test/swagger/spec.html - - Authentication: opnfv/api@opnfv - -Any test project running on any lab integrated in CI can push the -results to this database. -This database can be used to see the evolution of the tests and compare -the results versus the installers, the scenarios or the labs. - - -Overall Architecture --------------------- -The Test result management can be summarized as follows:: - - +-------------+ +-------------+ +-------------+ - | | | | | | - | Test | | Test | | Test | - | Project #1 | | Project #2 | | Project #N | - | | | | | | - +-------------+ +-------------+ +-------------+ - | | | - V V V - +-----------------------------------------+ - | | - | Test Rest API front end | - | http://testresults.opnfv.org/test | - | | - +-----------------------------------------+ - A | - | V - | +-------------------------+ - | | | - | | Test Results DB | - | | Mongo DB | - | | | - | +-------------------------+ - | - | - +----------------------+ - | | - | test Dashboard | - | | - +----------------------+ - -Test API description --------------------- -The Test API is used to declare pods, projects, test cases and test -results. Pods are the pods used to run the tests. -The results pushed in the database are related to pods, projects and -cases. If you try to push results of test done on non referenced pod, -the API will return an error message. - -An additional method dashboard has been added to post-process -the raw results in release Brahmaputra (deprecated in Colorado). - -The data model is very basic, 4 objects are created: - - * Pods - * Projects - * Testcases - * Results - -Pods:: - - { - "id": , - "details": , - "creation_date": "YYYY-MM-DD HH:MM:SS", - "name": , - "mode": , - "role": - }, - -Projects:: - - { - "id": , - "name": , - "creation_date": "YYYY-MM-DD HH:MM:SS", - "description": - }, - -Testcases:: - - { - "id": , - "name":, - "project_name":, - "creation_date": "YYYY-MM-DD HH:MM:SS", - "description": , - "url": - }, - -Results:: - - { - "_id": , - "case_name": , - "project_name": , - "pod_name": , - "installer": , - "version": , - "start_date": "YYYY-MM-DD HH:MM:SS", - "stop_date": "YYYY-MM-DD HH:MM:SS", - "build_tag": , - "scenario": , - "criteria": , - "trust_indicator": { - "current": 0, - "histories": [] - } - } - -The API can described as follows. For detailed information, please go to - - http://testresults.opnfv.org/test/swagger/spec.html - - Authentication: opnfv/api@opnfv - -Version: - - +--------+--------------------------+-----------------------------------------+ - | Method | Path | Description | - +========+==========================+=========================================+ - | GET | /versions | Get all supported API versions | - +--------+--------------------------+-----------------------------------------+ - - -Pods: - - +--------+----------------------------+-----------------------------------------+ - | Method | Path | Description | - +========+============================+=========================================+ - | GET | /api/v1/pods | Get the list of declared Labs (PODs) | - +--------+----------------------------+-----------------------------------------+ - | POST | /api/v1/pods | Declare a new POD | - | | | Content-Type: application/json | - | | | { | - | | | "name": "pod_foo", | - | | | "mode": "metal", | - | | | "role": "ci-pod", | - | | | "details": "it is a ci pod" | - | | | } | - +--------+----------------------------+-----------------------------------------+ - | GET | /api/v1/pods/{pod_name} | Get a declared POD | - +--------+----------------------------+-----------------------------------------+ - -Projects: - - +--------+----------------------------+-----------------------------------------+ - | Method | Path | Description | - +========+============================+=========================================+ - | GET | /api/v1/projects | Get the list of declared projects | - +--------+----------------------------+-----------------------------------------+ - | POST | /api/v1/projects | Declare a new test project | - | | | Content-Type: application/json | - | | | { | - | | | "name": "project_foo", | - | | | "description": "whatever you want" | - | | | } | - +--------+----------------------------+-----------------------------------------+ - | DELETE | /api/v1/projects/{project} | Delete a test project | - +--------+----------------------------+-----------------------------------------+ - | GET | /api/v1/projects/{project} | Get details on a {project} | - | | | | - +--------+----------------------------+-----------------------------------------+ - | PUT | /api/v1/projects/{project} | Update a test project | - | | | | - | | | Content-Type: application/json | - | | | { | - | | | | - | | | } | - +--------+----------------------------+-----------------------------------------+ - - -Testcases: - - +--------+----------------------------+-----------------------------------------+ - | Method | Path | Description | - +========+============================+=========================================+ - | GET | /api/v1/projects/{project}/| Get the list of testcases of {project} | - | | cases | | - +--------+----------------------------+-----------------------------------------+ - | POST | /api/v1/projects/{project}/| Add a new test case to {project} | - | | cases | Content-Type: application/json | - | | | { | - | | | "name": "case_foo", | - | | | "description": "whatever you want" | - | | | "url": "whatever you want" | - | | | } | - +--------+----------------------------+-----------------------------------------+ - | DELETE | /api/v1/projects/{project}/| Delete a test case | - | | cases/{case} | | - +--------+----------------------------+-----------------------------------------+ - | GET | /api/v1/projects/{project}/| Get a declared test case | - | | cases/{case} | | - +--------+----------------------------+-----------------------------------------+ - | PUT | /api/v1/projects/{project}?| Modify a test case of {project} | - | | cases/{case} | | - | | | Content-Type: application/json | - | | | { | - | | | | - | | | } | - +--------+----------------------------+-----------------------------------------+ - -Results: - - +--------+----------------------------+------------------------------------------+ - | Method | Path | Description | - +========+============================+==========================================+ - | GET | /api/v1/results | Get all the test results | - +--------+----------------------------+------------------------------------------+ - | POST | /api/v1/results | Add a new test results | - | | | Content-Type: application/json | - | | | { | - | | | "project_name": "project_foo", | - | | | "scenario": "odl-l2", | - | | | "stop_date": "2016-05-28T14:42:58.384Z", | - | | | "trust_indicator": 0.5, | - | | | "case_name": "vPing", | - | | | "build_tag": "", | - | | | "version": "Colorado", | - | | | "pod_name": "pod_foo", | - | | | "criteria": "PASS", | - | | | "installer": "fuel", | - | | | "start_date": "2016-05-28T14:41:58.384Z",| - | | | "details": | - | | | } | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of {case} | - | | case={case} | | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of build_tag | - | | build_tag={tag} | {tag}. | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get last {N} records of test results | - | | last={N} | | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of scenario | - | | scenario={scenario} | {scenario}. | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of trust_indicator | - | | trust_indicator={ind} | {ind}. | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of last days | - | | period={period} | {period}. | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of {project} | - | | project={project} | | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of version | - | | version={version} | {version}. | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of criteria | - | | criteria={criteria} | {criteria}. | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | get the results on pod {pod} | - | | pod={pod} | | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of installer {inst} | - | | installer={inst} | | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the results according to combined | - | | | query conditions supported above | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results/{result_id}| Get the test result by result_id | - +--------+----------------------------+------------------------------------------+ - -Scenarios: - - +--------+----------------------------+-----------------------------------------+ - | Method | Path | Description | - +========+============================+=========================================+ - | GET | /api/v1/scenarios | Get the list of declared scenarios | - +--------+----------------------------+-----------------------------------------+ - | POST | /api/v1/scenario | Declare a new scenario | - +--------+----------------------------+-----------------------------------------+ - | GET | /api/v1/scenario? | Get a declared scenario | - | | | | - +--------+----------------------------+-----------------------------------------+ - - -The code of the API is hosted in the releng repository `[6]`_. -The static documentation of the API can be found at `[17]`_. -The test API has been dockerized and may be installed locally in your -lab. See `[15]`_ for details. - -The deployment of the test API has been automated. -A jenkins job manages: - * the unit tests of the test api - * the creation of a new docker file - * the deployment of the new test api - * the archive of the old test api - * the backup of the Mongo DB - -Test API Authorization -~~~~~~~~~~~~~~~~~~~~~~ - -PUT/DELETE/POST operations of the testapi now require token based authorization. The token needs -to be added in the request using a header 'X-Auth-Token' for access to the database. - -e.g:: - headers['X-Auth-Token'] - -The value of the header i.e the token can be accessed in the jenkins environment variable -*TestApiToken*. The token value is added as a masked password. - -.. code-block:: python - - headers['X-Auth-Token'] = os.environ.get('TestApiToken') - -The above example is in Python. Token based authentication has been added so that only ci pods -jenkins job can have access to the database. - -Please note that currently token authorization is implemented but is not yet enabled. - - Automatic reporting - =================== - - An automatic reporting page has been created in order to provide a - consistant view of the scenarios. - In this page, each scenario is evaluated according to test criteria. - The code for the automatic reporting is available at `[8]`_. - - The results are collected from the centralized database every day and, - per scenario. A score is calculated based on the results from the last - 10 days. This score is the addition of single test scores. Each test - case has a success criteria reflected in the criteria field from the - results. - - Considering an instance of a scenario os-odl_l2-nofeature-ha, the - scoring is the addition of the scores of all the runnable tests from the - categories (tiers healthcheck, smoke and features) - corresponding to this scenario. - - - +---------------------+---------+---------+---------+---------+ - | Test | Apex | Compass | Fuel | Joid | - +=====================+=========+=========+=========+=========+ - | vPing_ssh | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | vPing_userdata | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | tempest_smoke_serial| X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | rally_sanity | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | odl | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | promise | | | X | X | - +---------------------+---------+---------+---------+---------+ - | doctor | X | | X | | - +---------------------+---------+---------+---------+---------+ - | security_scan | X | | | | - +---------------------+---------+---------+---------+---------+ - | parser | | | X | | - +---------------------+---------+---------+---------+---------+ - | copper | X | | | X | - +---------------------+---------+---------+---------+---------+ - - All the testcases listed in the table are runnable on - os-odl_l2-nofeature scenarios. - If no result is available or if all the results are failed, the test - case get 0 point. - If it was succesfull at least once but not anymore during the 4 runs, - the case get 1 point (it worked once). - If at least 3 of the last 4 runs were successful, the case get 2 points. - If the last 4 runs of the test are successful, the test get 3 points. - - In the example above, the target score for fuel/os-odl_l2-nofeature-ha - is 3x6 = 18 points. - - The scenario is validated per installer when we got 3 points for all - individual test cases (e.g 18/18). - Please note that complex or long duration tests are not considered for - the scoring. The success criteria are not always easy to define and may - require specific hardware configuration. These results however provide - a good level of trust on the scenario. - - A web page is automatically generated every day to display the status. - This page can be found at `[9]`_. For the status, click on Status menu, - you may also get feedback for vims and tempest_smoke_serial test cases. - - Any validated scenario is stored in a local file on the web server. In - fact as we are using a sliding windows to get results, it may happen - that a successful scenarios is no more run (because considered as - stable) and then the number of iterations (4 needed) would not be - sufficient to get the green status. - - Please note that other test cases (e.g. sfc_odl, bgpvpn) need also - ODL configuration addons and as a consequence specific scenario. - There are not considered as runnable on the generic odl_l2 scenario. - -Dashboard -========= - -Dashboard is used to provide a consistant view of the results collected -in CI. -The results showed on the dashboard are post processed from the Database, -which only contains raw results. - -In Brahmaputra, we created a basic dashboard. -Since Colorado, it was decided to adopt ELK framework. Mongo DB results -are extracted to feed Elasticsearch database (`[7]`_). - -A script was developed to build elasticsearch data set. This -script can be found in `[16]`_. - -For next versions, it was decided to integrated bitergia dashboard. -Bitergia already provides a dashboard for code and infrastructure. -A new Test tab will be added. The dataset will be built by consuming -the test API. - - -======= -How TOs -======= - -How Functest works? -=================== - -The installation and configuration of the Functest docker image is -described in `[1]`_. - -The procedure to start tests is described in `[2]`_ - - -How can I contribute to Functest? -================================= - -If you are already a contributor of any OPNFV project, you can -contribute to functest. If you are totally new to OPNFV, you must first -create your Linux Foundation account, then contact us in order to -declare you in the repository database. - -We distinguish 2 levels of contributors: - - * the standard contributor can push patch and vote +1/0/-1 on any Functest patch - * The commitor can vote -2/-1/0/+1/+2 and merge - -Functest commitors are promoted by the Functest contributors. - - -Where can I find some help to start? -==================================== - -This guide is made for you. You can also have a look at the project wiki -page `[10]`_. -There are references on documentation, video tutorials, tips... - -You can also directly contact us by mail with [Functest] prefix in the -title at opnfv-tech-discuss@lists.opnfv.org or on the IRC chan -#opnfv-functest. - - -What kind of testing do you do in Functest? -=========================================== - -Functest is focusing on Functional testing. The results must be PASS or -FAIL. We do not deal with performance and/or qualification tests. -We consider OPNFV as a black box and execute our tests from the jumphost -according to Pharos reference technical architecture. - -Upstream test suites are integrated (Rally/Tempest/ODL/ONOS,...). -If needed Functest may bootstrap temporarily testing activities if they -are identified but not covered yet by an existing testing project (e.g -security_scan before the creation of the security repository) - - -How test constraints are defined? -================================= - -Test constraints are defined according to 2 paramaters: - - * The scenario (DEPLOY_SCENARIO env variable) - * The installer (INSTALLER_TYPE env variable) - -A scenario is a formal description of the system under test. -The rules to define a scenario are described in `[4]`_ - -These 2 constraints are considered to determinate if the test is runnable -or not (e.g. no need to run onos suite on odl scenario). - -In the test declaration for CI, the test owner shall indicate these 2 -constraints. The file testcases.yaml `[5]`_ must be patched in git to -include new test cases. A more elaborated system based on template is -planned for next releases - -For each dependency, it is possible to define a regex:: - - name: promise - criteria: 'success_rate == 100%' - description: >- - Test suite from Promise project. - dependencies: - installer: '(fuel)|(joid)' - scenario: '' - -In the example above, it means that promise test will be runnable only -with joid or fuel installers on any scenario. - -The vims criteria means any installer and exclude onos and odl with -bgpvpn scenarios:: - - name: vims - criteria: 'status == "PASS"' - description: >- - This test case deploys an OpenSource vIMS solution from Clearwater - using the Cloudify orchestrator. It also runs some signaling traffic. - dependencies: - installer: '' - scenario: '(ocl)|(nosdn)|^(os-odl)((?!bgpvpn).)*$' - - -How to write and check constaint regex? -======================================= - -Regex are standard regex. You can have a look at `[11]`_ - -You can also easily test your regex via an online regex checker such as `[12]`_. -Put your scenario in the TEST STRING window (e.g. os-odl_l3-ovs-ha), put -your regex in the REGULAR EXPRESSION window, then you can test your rule -. - - -How to know which test I can run? -================================= - -You can use the API `[13]`_. The static declaration is in git `[5]`_ - -If you are in a Functest docker container (assuming that the -environement has been prepared): just use the CLI. - -You can get the list per Test cases or by Tier:: - - # functest testcase list - healthcheck - vping_ssh - vping_userdata - tempest_smoke_serial - rally_sanity - odl - doctor - security_scan - tempest_full_parallel - rally_full - vims - # functest tier list - - 0. healthcheck: - ['healthcheck'] - - 1. smoke: - ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity'] - - 2. sdn_suites: - ['odl'] - - 3. features: - ['doctor', 'security_scan'] - - 4. openstack: - ['tempest_full_parallel', 'rally_full'] - - 5. vnf: - ['vims'] - - -How to manually start Functest tests? -===================================== - -Assuming that you are connected on the jumphost and that the system is -"Pharos compliant", i.e the technical architecture is compatible with -the one defined in the Pharos project:: - - # docker pull opnfv/functest:latest - # envs="-e INSTALLER_TYPE=fuel -e INSTALLER_IP=10.20.0.2 -e DEPLOY_SCENARIO=os-odl_l2-nofeature-ha -e CI_DEBUG=true" - # sudo docker run --privileged=true -id ${envs} opnfv/functest:latest /bin/bash - - -Then you must connect to the docker container and source the -credentials:: - - # docker ps (copy the id) - # docker exec -ti bash - # source $creds - - -You must first check if the environment is ready:: - - # functest env status - Functest environment ready to run tests. - - -If not ready, prepare the env by launching:: - - # functest env prepare - Functest environment ready to run tests. - -Once the Functest env is ready, you can use the CLI to start tests. - -You can run test cases per test case or per tier: - # functest testcase run or # functest tier run - - -e.g:: - - # functest testcase run tempest_smoke_serial - # functest tier run features - - -If you want to run all the tests you can type:: - - # functest testcase run all - - -If you want to run all the tiers (same at the end that running all the -test cases) you can type:: - - # functest tier run all - - -How to declare my tests in Functest? -==================================== - -If you want to add new internal test cases, you can submit patch under -the testcases directory of Functest repository. - -For feature test integration, the code can be kept into your own -repository. The Functest files to be modified are: - - * functest/docker/Dockerfile: get your code in Functest container - * functest/ci/testcases.yaml: reference your test and its associated constraints - - -Dockerfile ----------- - -This file lists the repositories (internal or external) to be cloned in -the Functest container. You can also add external packages:: - - RUN git clone https://gerrit.opnfv.org/gerrit/ ${REPOS_DIR}/ - -testcases.yaml --------------- - -All the test cases that must be run from CI / CLI must be declared in -ci/testcases.yaml. - -This file is used to get the constraints related to the test:: - - name: - criteria: 'PASS', 'rate > 90%' - description: >- - - dependencies: - installer: regex related to installer e.g. 'fuel', '(apex)||(joid)' - scenario: regex related to the scenario e.g. 'ovs*no-ha' - - -You must declare your test case in one of the category (tier). - -If you are integrating test suites from a feature project, the default -category is **features**. - - -How to select my list of tests for CI? -====================================== - -Functest can be run automatically from CI, a jenkins job is usually -called after an OPNFV fresh installation. -By default we try to run all the possible tests (see `[14]` called from -Functest jenkins job):: - - cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t all ${flags}" - - -Each case can be configured as daily and/or weekly task. -Weekly tasks are used for long duration or experimental tests. -Daily tasks correspond to the minimum set of test suites to validate a scenario. - -When executing run_tests.py, a check based on the jenkins build tag will -be considered to detect whether it is a daily and/or a weekly test. - -in your CI you can customize the list of test you want to run by case or -by tier, just change the line:: - - cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t ${flags}" - -e.g.:: - - cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t healthcheck,smoke ${flags}" - -This command will run all the test cases of the first 2 tiers, i.e. -healthcheck, connection_check, api_check, vping_ssh, vping_userdata, -snaps_somke, tempest_smoke_serial and rally_sanity. - - -How to push your results into the Test Database -=============================================== - -The test database is used to collect test results. By default it is -enabled only for CI tests from Production CI pods. - -The architecture and associated API is described in previous chapter. -If you want to push your results from CI, you just have to call the API -at the end of your script. - -You can also reuse a python function defined in functest_utils.py:: - - def push_results_to_db(db_url, case_name, logger, pod_name,version, payload): - """ - POST results to the Result target DB - """ - url = db_url + "/results" - installer = get_installer_type(logger) - params = {"project_name": "functest", "case_name": case_name, - "pod_name": pod_name, "installer": installer, - "version": version, "details": payload} - - headers = {'Content-Type': 'application/json'} - try: - r = requests.post(url, data=json.dumps(params), headers=headers) - if logger: - logger.debug(r) - return True - except Exception, e: - print "Error [push_results_to_db('%s', '%s', '%s', '%s', '%s')]:" \ - % (db_url, case_name, pod_name, version, payload), e - return False - - -========== -References -========== - -_`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest configuration guide - -_`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide - -_`[3]`: https://wiki.opnfv.org/opnfv_test_dashboard Brahmaputra dashboard - -_`[4]`: https://wiki.opnfv.org/display/INF/CI+Scenario+Naming - -_`[5]`: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml - -_`[6]`: https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api - -_`[7]`: https://git.opnfv.org/cgit/releng/tree/utils/test/scripts - -_`[8]`: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting/functest - -_`[9]`: http://testresults.opnfv.org/reporting/ - -_`[10]`: https://wiki.opnfv.org/opnfv_functional_testing - -_`[11]`: https://docs.python.org/2/howto/regex.html - -_`[12]`: https://regex101.com/ - -_`[13]`: http://testresults.opnfv.org/test/api/v1/projects/functest/cases - -_`[14]`: https://git.opnfv.org/cgit/releng/tree/jjb/functest/functest-daily.sh - -_`[15]`: https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api/README.rst - -_`[16]`: https://git.opnfv.org/cgit/releng/tree/utils/test/scripts/mongo_to_elasticsearch.py - -_`[17]`: http://artifacts.opnfv.org/releng/docs/testapi.html - -OPNFV main site: http://www.opnfv.org - -OPNFV functional test page: https://wiki.opnfv.org/opnfv_functional_testing - -IRC support chan: #opnfv-functest - -_`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html - -_`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html - -_`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml diff --git a/docs/internship/security_group/index.rst b/docs/internship/security_group/index.rst deleted file mode 100644 index d1cdbdd8f..000000000 --- a/docs/internship/security_group/index.rst +++ /dev/null @@ -1,70 +0,0 @@ -======= -License -======= - -Functest Docs are licensed under a Creative Commons Attribution 4.0 -International License. -You should have received a copy of the license along with this. -If not, see . - -================================== -Functest Security group test cases -================================== - -Author: Girish Sukhatankar -mentors: D.Blaisonneau, J.Lausuch, M.Richomme - -Abstract -======== - - -Version history -=============== - -+------------+----------+------------------+------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+------------------------+ -| 2016-??-?? | 0.0.1 | Morgan Richomme | Beginning of the | -| | | (Orange) | Internship | -+------------+----------+------------------+------------------------+ - - -Overview: -========= - - - - -Problem Statement: ------------------- - - - -Curation Phase --------------- - - - - - -Schedule: -========= - - - -+--------------------------+------------------------------------------+ -| **Date** | **Comment** | -| | | -+--------------------------+------------------------------------------+ -| December - January | ........ | -+--------------------------+------------------------------------------+ -| January - february | ........ | -+--------------------------+------------------------------------------+ - - -References: -=========== - -.. _`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Security+groups+test+case+in+Functest - diff --git a/docs/internship/testapi_evolution/index.rst b/docs/internship/testapi_evolution/index.rst deleted file mode 100644 index 6a1cde7df..000000000 --- a/docs/internship/testapi_evolution/index.rst +++ /dev/null @@ -1,237 +0,0 @@ -======= -License -======= - -Functest Docs are licensed under a Creative Commons Attribution 4.0 -International License. -You should have received a copy of the license along with this. -If not, see . - -================== -Test API evolution -================== - -Author: Sakala Venkata Krishna Rohit -Mentors: S. Feng, J.Lausuch, M.Richomme - -Abstract -======== - -The testapi is used by all the test opnfv projects to report results. -It is also used to declare projects, test cases and labs. A major refactoring -has been done in Colorado with the introduction of swagger. The testapi is defined in Functest -developer guide. The purpose of this project is to add more features to the testapi that automate -the tasks that are done manually now, though there are tasks other than automation. - -Version history -=============== - -+------------+----------+------------------+------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+------------------------+ -| 2016-11-14 | 0.0.1 | Morgan Richomme | Beginning of the | -| | | (Orange) | Internship | -+------------+----------+------------------+------------------------+ -| 2017-02-17 | 0.0.2 | S.V.K Rohit | End of the Internship | -| | | (IIIT Hyderabad) | | -+------------+----------+------------------+------------------------+ - -Overview: -========= - -The internhip time period was from Nov 14th to Feb 17th. The project prosposal page is here `[1]`_. -The intern project was assigned to Svk Rohit and was mentored by S. Feng, J.Lausuch, M.Richomme. -The link to the patches submitted is `[2]`_. The internship was successfully completed and the -documentation is as follows. - -Problem Statement: ------------------- - -The problem statement could be divided into pending features that needed to be added into testapi -repo. The following were to be accomplished within the internship time frame. - -* **Add verification jenkins job for the testapi code** - The purpose of this job is to verify whehter the unit tests are successful or not with the - inclusion of the patchset submitted. - -* **Automatic update of opnfv/testapi docker image** - The docker image of testapi is hosted in the opnfv docker hub. To ensure that the testapi image - is always updated with the repository, automatic updation of the image is necessary and a job - is triggered whenever a new patch gets merged. - -* **Automation deployment of testresults.opnfv.org/test/ website** - In the same manner as the docker image of testapi is updated, the testapi website needs to be - in sync with the repository code. So, a job has been added to the opnfv jenkins ci for the - updation of the testresults website. - -* **Generate static documentation of testapi calls** - The purpose of this is to give an static/offline view of testapi. If someone wants to have a - look at the Restful apis of testapi, he/she does't need to go to the website, he can download - a html page and view it anytime. - -* **Backup MongoDB of testapi** - The mongoDB needs to be backed up every week. Till now it was done manually, but due to this - internship, it is now automated using a jenkins job. - -* **Add token based authorization to the testapi calls** - The token based authorization was implemented to ensure that only ci_pods could access the - database. Authentication has been added to only delete/put/post requests. - -Curation Phase: ---------------- - -The curation phase was the first 3 to 4 weeks of the internship. This phase was to get familiar -with the testapi code and functionality and propose the solutions/tools for the tasks mentioned -above. Swagger codegen was choosen out of the four tools proposed `[3]`_ for generating static -documentaion. - -Also, specific amount of time was spent on the script flow of the jenkins jobs. The automatic -deployment task involves accessing a remote server from inside the jenkins build. The deployment -had to be done only after the docker image update is done. For these constraints to satisfy, a -multijob jenkins job was choosen instead of a freestyle job. - -Important Links: ----------------- - -* MongoDB Backup Link - `[4]`_ -* Static Documentation - `[5]`_ -* TestAPI Token addition to ci_pods - `[6]`_ - -Schedule: -========= - -The progress and completion of the tasks is described in the below table. - -+--------------------------+------------------------------------------+ -| **Date** | **Comment** | -| | | -+--------------------------+------------------------------------------+ -| Nov 14th - Dec 31st | Understand Testapi code and the | -| | requirements. | -+--------------------------+------------------------------------------+ -| Jan 1st - Jan 7th | Add jenkins job to create static | -| | documentation and write build scripts. | -+--------------------------+------------------------------------------+ -| Jan 8th - Jan 21st | Add verification jenkins job for unit | -| | tests. | -+--------------------------+------------------------------------------+ -| Jan 22nd - Jan 28th | Add jenkins job for mongodb backup | -| | | -+--------------------------+------------------------------------------+ -| Jan 29th - Feb 11th | Enable automatic deployment of | -| | testresults.opnfv.org/test/ | -+--------------------------+------------------------------------------+ -| Feb 12th - Feb 17th | Add token based authentication | -| | | -+--------------------------+------------------------------------------+ - -FAQ's -===== - -This section lists the problems that I have faced and the understanding that I have acquired during -the internship. This section may help other developers in solving any errors casused because of the -code written as a part of this internship. - - -Test Api --------- - -What is the difference between defining data_file as "/etc/.." and "etc/.." in setup.cfg ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If in the setup.cfg, it is defined as - -[files] -data_files = -etc/a.conf = etc/a.conf.sample - -then it ends up installed in the /usr/etc/. With this configuration, it would be installed -correctly within a venv. but when it is defined as - -[files] -data_files = -/etc/a.conf = etc/a.conf.sample - -then it ends up installed on the root of the filesystem instead of properly be installed within the -venv. - -Which attribute does swagger-codegen uses as the title in the generation of document generation ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -It uses the nickname of the api call in swagger as the title in the generation of the document -generation. - -Does swagger-codegen take more than one yaml file as input ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -No, swagger-codegen only takes one yaml file as input to its jar file. If there more than one yaml -file, one needs to merge them and give it as an input keeping mind the swagger specs. - - -Jenkins & JJB -------------- - -Which scm macro is used for verification jenkins jobs ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -There are two macros for scm one is git-scm and other git-scm-gerrit. git-scm-gerrit is used for -verification jenkins job. - -Does the virtualenv created in one build script exists in other build scripts too ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -No, the virtualenv created in one build script only exists in that build script/shell. - -What parameters are needed for the scm macros ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Project and Branch are the two parameters needed for scm macros. - -What is the directory inside the jenkins build ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The directory of the jenkins build is the directory of the repo. `ls $WORKSPACE` command will give -you all the contents of the directory. - -How to include a bash script in jenkins job yaml file ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An example might be apt here as an answer. - -builders: - - shell: - !include-raw: include-raw001-hello-world.sh - - -How do you make a build server run on a specific machine ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -It can be done by defining a label parameter 'SLAVE_LABEL' or in OPNFV , there are macros for each -server, one can use those parameter macros. -Ex: opnfv-build-defaults. Note, if we use macro, then no need to define GIT_BASE, but if one uses -SLAVE_LABEL, one needs to define a parameter GIT_BASE. This is because macro already has GIT_BASE -defined. - -What job style should be used when there is a situation like one build should trigger other builds -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -or when different build scripts need to be run on different machines ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -MultiJob style should be used as it has phases where each phase can be taken as a build scipt and -can have its own parameters by which one can define the SLAVE_LABEL parameter. - -References: -=========== - -_`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+testapi+evolution - -_`[2]` : https://gerrit.opnfv.org/gerrit/#/q/status:merged+owner:%22Rohit+Sakala+%253Crohitsakala%2540gmail.com%253E%22 - -_`[3]` : https://docs.google.com/document/d/1jWwVZ1ZpKgKcOS_zSz2KzX1nwg4BXxzBxcwkesl7krw/edit?usp=sharing - -_`[4]` : http://artifacts.opnfv.org/testapibackup.html - -_`[5]` : http://artifacts.opnfv.org/releng/docs/testapi.html - -_`[6]` : http://artifacts.opnfv.org/functest/docs/devguide/index.html#test-api-authorization diff --git a/docs/internship/unit_tests/index.rst b/docs/internship/unit_tests/index.rst deleted file mode 100644 index f969aa72d..000000000 --- a/docs/internship/unit_tests/index.rst +++ /dev/null @@ -1,70 +0,0 @@ -======= -License -======= - -Functest Docs are licensed under a Creative Commons Attribution 4.0 -International License. -You should have received a copy of the license along with this. -If not, see . - -=================== -Functest Unit tests -=================== - -Author: Ashish Kumar -Mentors: H.Yao, J.Lausuch, M.Richomme - -Abstract -======== - - -Version history -=============== - -+------------+----------+------------------+------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+------------------------+ -| 2016-??-?? | 0.0.1 | Morgan Richomme | Beginning of the | -| | | (Orange) | Internship | -+------------+----------+------------------+------------------------+ - - -Overview: -========= - - - - -Problem Statement: ------------------- - - - -Curation Phase --------------- - - - - - -Schedule: -========= - - - -+--------------------------+------------------------------------------+ -| **Date** | **Comment** | -| | | -+--------------------------+------------------------------------------+ -| December - January | ........ | -+--------------------------+------------------------------------------+ -| January - february | ........ | -+--------------------------+------------------------------------------+ - - -References: -=========== - -.. _`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Functest+unit+tests - diff --git a/docs/internship/vnf_catalog/index.rst b/docs/internship/vnf_catalog/index.rst deleted file mode 100644 index df7633391..000000000 --- a/docs/internship/vnf_catalog/index.rst +++ /dev/null @@ -1,170 +0,0 @@ -======= -License -======= - -Functest Docs are licensed under a Creative Commons Attribution 4.0 -International License. -You should have received a copy of the license along with this. -If not, see . - -======================= -Open Source VNF Catalog -======================= - -Author: Kumar Rishabh -Mentors: B.Souville, M.Richomme, J.Lausuch - -Abstract -======== - - - -Version hissory -=============== - -+------------+----------+------------------+------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+------------------------+ -| 2016-12-12 | 0.0.1 | Morgan Richomme | Beginning of the | -| | | (Orange) | Internship | -+------------+----------+------------------+------------------------+ - - -Overview: -========= - - -This project aims to create an Open Source catalog for reference and -classification of Virtual Network Functions (VNFs)s available on -Internet. The classification method proposed will be in sync with the -requirements of Telcos active in NFV landscape. The project aims to have -running web platform similar to [1] by the mid of internship (2nd week -of March). By the penultimate month of internship I aim to have fully -functional implementation of an Open Source VNF in functest. - - -Problem Statement: ------------------- - -OPNFV aims to be the reference platform for development, -standardization and integration of Open Source NFV components across -various Open Source Platforms. It mainly deals with the infrastructure -through the Network Function Virtualization Infrastructure (NFVI) and -Virtual Infrastructure manager (VIM). The MANO (Management and -orchestration) stacks have been introduced recently. VNFs are not -directly in OPNFV scope, however VNFs are needed to test and qualify the -infrastructure. In this regard having a common curated Open Source -Reference VNF catalog would be of immense importance to community. - -Since major focus of OPNFV is Telcos, a curated platform targeted from -industry point of view would be very useful. We plan to divide the -entire project into three major phases(with some iterative improvements -and overlaps) - - -Curation Phase --------------- -This phase pertains to studying various Open Source VNFs available and -classification of them based on certain parameters. The parameters that -I currently have in mind are: - * Developer Metrics: These pertain to repo characteristics of VNF under - study - * Usage Statistics - Activity, Number of Commits, stars - * Maturity Statistics - For instance if an NFV community decides code - coverage is important for them, it shows the NFV community is serious - about taking the project forward - * Technical Tagging: These are the tags that pertain to technical - characteristics of a VNF - * Broad Use Cases - Whether the VNF fits strictly in IaaS, PaaS or - SaaS layer or is an hybrid of two/all. - * Generic Use Cases - This in my opinion is the broadest - classification category. For instance a VNF could be built with a - broad idea of powering IOT devices at home or from usage perspective - of Telco Operators (vFW, vEPC, vIMS, vCDN, vAAA, vCPE,...).`[2]`_ - * Fields of Application - * Library Status - Whether APIs are standardized, support RESTful - services. - * Dependency Forwarding Graph - This is pretty complex tagging - mechanism. It essentially tries to establish a graph relationship - between the VNFs (elementary VNFs are used in Service Function - Chaining chains such as Firewall, DPI, content enrichment,..). In my - opinion this is useful immensely. This will allow users to go to - platform and ask a question like - “I have this X tech stack to - support, Y and Z are my use cases, which NFVs should I use to support - this. - * Visitor Score - Based on `[1]`_ I plan to evolve a visitor score for - the platform. This will allow users to score an NFV on certain - parameters, may be post comments. - -**I plan to use the above three scores and evolve cumulative score which -will be displayed next to each of the NFV on the platform.** - - * Platform building phase - This will involve erecting a Web Platform - which will be similar to this `[1]`_. I am decently familiar with - Django and hence I will write the platform in Django. There are two - action plans that I have in mind right now. Either I can start writing - the platform simultaneously which will help keep track of my progress - or I can write the platform after 1.5 - 2 months into the internship. - Either way I aim to have the Web Platform ready by March 12. - - * Functest VNF implementation phase - This is the last phase that will - involve writing a fully functional implementation of an Open Source VNF - into Functest. I will undertake this after I am 3 months into the - internship. I have a decent familiarity with python and hence I think - it shouldn’t be too difficult. I need to decide how complex the VNFI - should undertake this exercise for (e.g. AAA such as free radius sounds - relatively easy, vCDN is much more challenging). - This will be decided in consent with my mentors. - - - - -Schedule: -========= -I plan to take this project in 6 months time frame as I want to use it -as a chance to read more about NFVs in particular and SDN in general - - -+--------------------------+------------------------------------------+ -| **Date** | **Comment** | -| | | -+--------------------------+------------------------------------------+ -| December 12 - January 12 | Study the above mentioned metrics | -| | Decide which of them are important for | -| | community (and which are not). | -+--------------------------+------------------------------------------+ -| January 12 - January 27 | Make a database for the above studied | -| | metrics and evolve it further based on | -| | Mentors’ input. + associated API | -+--------------------------+------------------------------------------+ -| January 27 - February 5 | Compile the data collected above and make| -| | it public. Although I can keep everything| -| | public from the beginning too. My | -| | rationale of not making the entire data | -| | public in initial stage as the errors | -| | caused by me could be misleading for | -| | developers. | -+--------------------------+------------------------------------------+ -| February 5 - March 5 | Erect the Web Platform and release it | -| | for restricted group for alpha testing. | -+--------------------------+------------------------------------------+ -| March 5 - March 12 | Make it public. Release it to public for | -| | beta testing. Fix Bugs. | -+--------------------------+------------------------------------------+ -| March 12 - April 12 | Start working on implementation of an | -| | Open Source VNF in Functest. | -+--------------------------+------------------------------------------+ -| April 12 - May 12 | I will decided what to do here based on | -| | discussion with mentors. | -+--------------------------+------------------------------------------+ - - -References: -=========== - -.. _`[1]` : Openhub: https://www.openhub.net/explore/projects - -.. _`[2]` : ETSI NFV White Paper: https://portal.etsi.org/Portals/0/TBpages/NFV/Docs/NFV_White_Paper3.pdf - -.. _`[3]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Open+Source+VNF+catalog diff --git a/docs/release-notes/functest-release.rst b/docs/release-notes/functest-release.rst deleted file mode 100644 index 07389bb5b..000000000 --- a/docs/release-notes/functest-release.rst +++ /dev/null @@ -1,637 +0,0 @@ -======= -License -======= - -OPNFV Colorado release note for Functest Docs -(c) by Morgan Richomme (Orange) - -OPNFV Colorado release note for Functest Docs -are licensed under a Creative Commons Attribution 4.0 International License. -You should have received a copy of the license along with this. -If not, see . - -=========================================== -OPNFV Colorado1.0 release note for Functest -=========================================== - -Abstract -======== - -This document describes the release note of Functest project. - - -Version history -=============== - -+------------+----------+------------------+------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+------------------------+ -| 2016-08-17 | 1.0.0 | Morgan Richomme | Functest for Colorado | -| | | (Orange) | release | -+------------+----------+------------------+------------------------+ - -OPNFV Colorado Release -========================= - -Functest deals with functional testing of the OPNFV solution. -It includes test cases developed within the project and test cases developed in -other OPNFV projects and other upstream communities. - -The internal test cases are: - - * healthcheck - * vPing ssh - * vPing userdata - * Tempest Smoke Serial - * Rally Sanity - * ODL - * Tempest full parallel - * Rally full - * vIMS - -The OPNFV projects integrated into Functest framework for automation are: - - * bgpvpn - * Copper - * Doctor - * Domino - * Moon - * Multisite - * ONOSFW - * ONOS-sfc - * ODL-sfc - * Parser - * Promise - * Security scan - -The validation of a scenario requires a subset of these tests depending -on the installer and the scenario. - -The 3 last internal test cases (tempest full parallel, Rally full and -vIMS) are not considered for scenario validation. - -Release Data -============ - -+--------------------------------------+--------------------------------------+ -| **Project** | functest | -| | | -+--------------------------------------+--------------------------------------+ -| **Repo/tag** | colorado.1.0 | -| | | -+--------------------------------------+--------------------------------------+ -| **Release designation** | Colorado base release | -| | | -+--------------------------------------+--------------------------------------+ -| **Release date** | September 22 2016 | -| | | -+--------------------------------------+--------------------------------------+ -| **Purpose of the delivery** | Colorado base release | -| | | -+--------------------------------------+--------------------------------------+ - -Deliverables -============ - -Software --------- - - - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: colorado.1.0) - - - The testapi Docker image: https://hub.docker.com/r/opnfv/testapi (tag:colorado.1.0) - - -Documents ---------- - - - Installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html - - - User Guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html - - - Developer Guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html - - -Version change -============== - -Feature evolution ------------------ - - - refactoring of ODL functional tests (with upstream modifications) - - - refactoring of testapi (update, swagger documentation, dockerization) - - - jenkins logs improvement - - - update integration of Doctor, Promise and SDNVPN projects - - - split Tempest and rally into 2 different tests: smoke and full - - - vIMS test suite integration - - - adoption of Kibana for dashboarding - - -New features ------------- - - - Functest CLI to prepare and run the tests - - - creation of the healthcheck test case - - - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, fdio, multisite) - - - integration of new OPNFV feature projects (copper, domino, multisite, - moon, parser, onos-sfc, odl-sfc, security scan) - - - introduction of test tiers in functest framework - - - automatic reporting - - - introduction of a jenkins summary table - - - support of ARM architecture - - -Scenario Matrix -=============== - -For Colorado 1.0, Functest was tested on the following scenarios (if not -precised, the scenario is a HA scenario): - -+---------------------+---------+---------+---------+---------+ -| Scenario | Apex | Compass | Fuel | Joid | -+=====================+=========+=========+=========+=========+ -| nosdn | X | X | X | X | -+---------------------+---------+---------+---------+---------+ -| nosdn-noha | | | X | X | -+---------------------+---------+---------+---------+---------+ -| kvm | | | X | | -+---------------------+---------+---------+---------+---------+ -| kvm-noha | | | X | | -+---------------------+---------+---------+---------+---------+ -| lxd | | | | X | -+---------------------+---------+---------+---------+---------+ -| lxd-noha | | | | X | -+---------------------+---------+---------+---------+---------+ -| ovs-noha (dpdk) | | | X | | -+---------------------+---------+---------+---------+---------+ -| odl_l2 | X | X | X | X | -+---------------------+---------+---------+---------+---------+ -| odl_l2-noha | | | X | | -+---------------------+---------+---------+---------+---------+ -| odl_l2-bgpvpn | X | | X | | -+---------------------+---------+---------+---------+---------+ -| odl_l2-bgpvpn-noha| | | X | | -+---------------------+---------+---------+---------+---------+ -| odl_l2-fdio-noha | X | | | | -+---------------------+---------+---------+---------+---------+ -| odl_l2-moon | | X | | | -+---------------------+---------+---------+---------+---------+ -| odl_l2-sfc-noha | X | | X | | -+---------------------+---------+---------+---------+---------+ -| odl_l3 | X | X | X | | -+---------------------+---------+---------+---------+---------+ -| odl_l3-noha | | | X | | -+---------------------+---------+---------+---------+---------+ -| onos | | X | X | X | -+---------------------+---------+---------+---------+---------+ -| onos-noha | | | X | | -+---------------------+---------+---------+---------+---------+ -| onos-sfc | | X | X | X | -+---------------------+---------+---------+---------+---------+ -| onos-sfc-noha | | | X | | -+---------------------+---------+---------+---------+---------+ -| multisite | | | X | | -+---------------------+---------+---------+---------+---------+ - -For Colorado 2.0, the following scenarios have been re-released: - -+---------------------+---------+---------+---------+---------+ -| Scenario | Apex | Compass | Fuel | Joid | -+=====================+=========+=========+=========+=========+ -| onos | X | X | X | X | -+---------------------+---------+---------+---------+---------+ -| odl_l2-sfc-ha | | | X | | -+---------------------+---------+---------+---------+---------+ - - -In Colorado, the functional tests have been sliced in 6 different -categories: - -+----------------+-----------------------------------------------+ -| Category | Description | -+================+===============================================+ -| healthcheck | Basic OpenStack commands | -+----------------+-----------------------------------------------+ -| smoke | vPings, Tempest and rally smoke tests | -+----------------+-----------------------------------------------+ -| sdn_suites | Specific SDN feature tests | -+----------------+-----------------------------------------------+ -| features | OPNFV feature project functional test suites | -+----------------+-----------------------------------------------+ -| openstack | Advanced, long duration OpenStack tests | -| | (Tempest and Rally full suite). Each test may | -| | last several hours | -+----------------+-----------------------------------------------+ -| vnf | Complex scenarios dealing with orchestration, | -| | VNF deployment and tests (e.g. vIMS) | -+----------------+-----------------------------------------------+ - -For the scenario validation, we consider only the healthcheck, smoke, -sdn_suites and features categories. These tests are run systematically -in the CI daily loops. - -Success criteria have been defined for these test cases, they can be -PASS/FAIL or a success rate may be declared (100%, > 90%) -All the criteria, as well as the test dependencies are declared in the -ci/testcases.yaml file. - -The scoring for the Colorado release per installer can be described as -follows. - -The scoring is an indicator showing how many feature project test suites -have been integrated on the scenario. - -The scoring equals the number of tests * succesful iteration of each -test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617 - - e.g. - apex/odl_l2-nofeature-ha - tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper - Scoring = 21/21 = 7 * 3 - -By default, if not specified, the scenarios are HA. -HA means OpenStack High Availability (main services). Note that not -all VIM (e.g. OpenStack) services are deployed in HA mode, as that -depends upon support of the specific service for HA deployment. -For example, in the Colorado release, the Congress service -is deployed in non-HA mode even for HA OPNFV scenarios, as explicit -support for HA operation has not yet been verified. -See the release notes of the installers for more details. - - -Apex ----- - -+------------------+---------+---------+-------------------+ -| Scenario | Scoring | Success | Results | -| | | rate | | -+==================+=========+=========+===================+ -| nosdn | 17/18 | 95% | `apex-res-174`_ | -+------------------+---------+---------+-------------------+ -| odl_l2 | 21/21 | 100% | `apex-res-175`_ | -+------------------+---------+---------+-------------------+ -| odl_l3 | 15/18 | 83% | `apex-res-176`_ | -+------------------+---------+---------+-------------------+ -| odl_l2-bgpvpn | 14/18 | 78% | `apex-res-235`_ | -+------------------+---------+---------+-------------------+ -| odl_l2-fdio-noha | 12/15 | 80% | `apex-res-6`_ | -+------------------+---------+---------+-------------------+ -| odl_l2-sfc-noha | 18/24 | 75% | `apex-res-217`_ | -+------------------+---------+---------+-------------------+ -| onos-nofeature-ha| 20/21 | 95% | `apex-res-423`_ | -+------------------+---------+---------+-------------------+ - - -Compass -------- - -+------------------+---------+---------+-------------------+ -| Scenario | Scoring | Success | Results | -| | | rate | | -+==================+=========+=========+===================+ -| nosdn | 12/12 | 100% | `compass-res-55`_ | -+------------------+---------+---------+-------------------+ -| odl_l2 | 15/15 | 100% | `compass-res-59`_ | -+------------------+---------+---------+-------------------+ -| odl_l3 | 9/12 | 75% | `compass-res-73`_ | -+------------------+---------+---------+-------------------+ -| odl_l2-moon | 15/18 | 83% | `compass-res-567`_| -+------------------+---------+---------+-------------------+ -| onos-ha | 15/15 | 100% | `compass-res-285`_| -+------------------+---------+---------+-------------------+ -| onos-sfc-ha | 17/18 | 95% | `compass-res-76`_ | -+------------------+---------+---------+-------------------+ - -Note: all the Compass tests for Colorado have been executed on virtual -environment. Bare metal resources were used for Master branch. - - -Fuel ----- - -+---------------------+---------+---------+-------------------+ -| Scenario | Scoring | Success | Results | -| | | rate | | -+=====================+=========+=========+===================+ -| nosdn | 18/18 | 100% | `fuel-res-129`_ | -+---------------------+---------+---------+-------------------+ -| nosdn-noha | 15/15 | 100% | `fuel-res-154`_ | -+---------------------+---------+---------+-------------------+ -| nosdn-kvm | 18/18 | 100% | `fuel-res-128`_ | -+---------------------+---------+---------+-------------------+ -| nosdn-kvm-noha | 15/15 | 100% | `fuel-res-161`_ | -+---------------------+---------+---------+-------------------+ -| nosdn-ovs | 12/18 | 67%* | `fuel-res-213`_ | -+---------------------+---------+---------+-------------------+ -| nosdn-ovs-noha | 15/15 | 100% | `fuel-res-162`_ | -+---------------------+---------+---------+-------------------+ -| odl_l2 | 21/21 | 100% | `fuel-res-123`_ | -+---------------------+---------+---------+-------------------+ -| odl_l2-noha | 17/18 | 94% | `fuel-res-155`_ | -+---------------------+---------+---------+-------------------+ -| odl_l2-bgpvpn | 14/18 | 78% | `fuel-res-119`_ | -+---------------------+---------+---------+-------------------+ -| odl_l2-bgpvpn-noha | 14/15 | 93% | `fuel-res-160`_ | -+---------------------+---------+---------+-------------------+ -| odl_l2-sfc-noha | 6/21 | 29% | `fuel-res-219`_ | -+---------------------+---------+---------+-------------------+ -| odl_l2-sfc-ha | 16/21 | 76% | `fuel-res-376`_ | -+---------------------+---------+---------+-------------------+ -| odl_l3 | 15/18 | 83% | `fuel-res-115`_ | -+---------------------+---------+---------+-------------------+ -| odl_l3-noha | 12/15 | 80% | `fuel-res-164`_ | -+---------------------+---------+---------+-------------------+ -| onos | 20/21 | 95% | `fuel-res-492`_ | -+---------------------+---------+---------+-------------------+ -| onos-noha | 18/18 | 100% | `fuel-res-166`_ | -+---------------------+---------+---------+-------------------+ -| onos-sfc | 24/24 | 100% | `fuel-res-124`_ | -+---------------------+---------+---------+-------------------+ -| onos-sfc-noha | 21/21 | 100% | `fuel-res-129`_ | -+---------------------+---------+---------+-------------------+ -| multisite | N.R | 100% | `fuel-res-8`_ | -+---------------------+---------+---------+-------------------+ - -*: all results passed, lacking iterations to reach the full score - -Results of Functest on AArch64 Colorado 3.0 - -+---------------------+---------+---------+----------------------+ -| Scenario | Scoring | Success | Results | -| | | rate | | -+=====================+=========+=========+======================+ -| nosdn | 18/18 | 100% | `fuel-arm-res-128`_ | -+---------------------+---------+---------+----------------------+ -| odl_l2 | 21/21 | 100% | `fuel-arm-res-122`_ | -+---------------------+---------+---------+----------------------+ -| odl_l2-noha | 18/18 | 100% | `fuel-arm-res-129`_ | -+---------------------+---------+---------+----------------------+ -| odl_l3 | 17/18 | 95% | `fuel-arm-res-135`_ | -+---------------------+---------+---------+----------------------+ - - -Joid ----- - -+---------------------+---------+---------+-----------------+ -| Scenario | Scoring | Success | Results | -| | | rate | | -+=====================+=========+=========+=================+ -| nosdn | 18/18 | 100% | `joid-res-102`_ | -+---------------------+---------+---------+-----------------+ -| nosdn-noha | 17/18 | 95% | `joid-res-93`_ | -+---------------------+---------+---------+-----------------+ -| nosdn-lxd | 12/12 | 100% | `joid-res-104`_ | -+---------------------+---------+---------+-----------------+ -| nosdn-lxd-noha | 12/12 | 100% | `joid-res-91`_ | -+---------------------+---------+---------+-----------------+ -| odl_l2 | 21/21 | 100% | `joid-res-103`_ | -+---------------------+---------+---------+-----------------+ -| onos | 21/21 | 100% | `joid-res-345`_ | -+---------------------+---------+---------+-----------------+ -| onos-sfc | 24/24 | 100% | `joid-res-97`_ | -+---------------------+---------+---------+-----------------+ - -It is highly recommended to install a json viewer in your browser -(e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/) - -You can get additional details through test logs on http://artifacts.opnfv.org/. -As no search engine is available on the OPNFV artifact web site you must -retrieve the pod identifier on which the tests have been executed (see -field pod in any of the results) then click on the selected POD and look -for the date of the test you are interested in. - -The reporting pages can be found at: - - * apex: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html - * compass: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-compass.html - * fuel: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html - * joid: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-joid.html - -Colorado known restrictions/issues -================================== - -+-----------+-----------+----------------------------------------------+ -| Installer | Scenario | Issue | -+===========+===========+==============================================+ -| any | odl_l3-* | Tempest cases related to using floating IP | -| | | addresses fail because of a known ODL bug. | -| | | vPing_ssh test case is excluded for the same | -| | | reason. | -| | | https://jira.opnfv.org/browse/APEX-112 | -| | | https://jira.opnfv.org/browse/FUNCTEST-445 | -+-----------+-----------+----------------------------------------------+ -| apex/fuel | *-bgpvpn | vPing_ssh (floating ips not supported) and | -| | | vIMS excluded. Some Tempest cases related to | -| | | floating ips also excluded. Some performance | -| | | issues have been detected in this scenario | -| | | (i.e. BGPVPN extension enabled) when running | -| | | commands against the OpenStack APIs, thus | -| | | Rally sanity test case has been disabled. | -| | | Performance issues seem to be connected to | -| | | the ODL version. It is planned to reintroduce| -| | | Rally sanity in Colorado 2.0 with the | -| | | adoption of ODL Boron release. | -+-----------+-----------+----------------------------------------------+ -| apex | *-fdio | Due to late integration, fdio decided to | -| | | focus on mandatory tests and exclude feature | -| | | tests (copper, doctor, security_scan) from | -| | | its scenarios | -+-----------+-----------+----------------------------------------------+ -| compass | moon | First ODL test FAILS because ODL/Openstack | -| | | federation done in moon is partial. Only | -| | | MD-SAL is federated (not AD-SAL) | -+-----------+-----------+----------------------------------------------+ -| fuel | any | TestServerBasicOps test case skipped | -| | | https://gerrit.opnfv.org/gerrit/#/c/19635/ | -+-----------+-----------+----------------------------------------------+ -| fuel | kvm | vPing_ssh and vIMS excluded (metadata related| -| | | scenarios) | -+-----------+-----------+----------------------------------------------+ -| fuel | multisite | random errors running multisite. A known bug | -| | | in keystone mitaka, due to which memcache | -| | | raises exception and keystone becomes | -| | | unresponsive | -| | | bugs.launchpad.net/keystone/+bug/1600394 | -| | | workaround consists in restarting memcache on| -| | | server | -+-----------+-----------+----------------------------------------------+ -| joid | any | Tempest cases related to object storage | -| | | excluded | -| | | https://gerrit.opnfv.org/gerrit/#/c/17871/ | -+-----------+-----------+----------------------------------------------+ -| joid | domino | Domino tests are skipped in CI. However the | -| | | test case can be run by manually setting | -| | | IS_IPandKEY_CONFIGURED=true after manually | -| | | setting properly the IP addresses of the 3 | -| | | Controller nodes in the configuration file | -| | | /repos/domino/tests/run_multinode.sh | -+-----------+-----------+----------------------------------------------+ - - -Test and installer/scenario dependencies -======================================== - -It is not always possible to run all the test cases on all the scenarios. -The following table details the dependencies of the test cases per -scenario. The scenario dependencies (installer or scenario) are detailed -in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml - -Test results -============ - -Test results are available in: - - - test results document: http://artifacts.opnfv.org/functest - - - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/ - - - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/ - - - -Open JIRA tickets -================= - -+------------------+-----------------------------------------------+ -| JIRA | Description | -+==================+===============================================+ -| `FUNCTEST-446`_ | Cleanup ODL-SFC output in Functest execution | -| | Impact on odl_l2-sfc scenarios | -+------------------+-----------------------------------------------+ -| `FUNCTEST-454`_ | Cleanup failures when using HA networks in | -| | Neutron | -+------------------+-----------------------------------------------+ - -Useful links -============ - - - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing - - - wiki Functest Colorado page: https://wiki.opnfv.org/display/functest/Functest+Colorado - - - Functest repo: https://git.opnfv.org/cgit/functest - - - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/ - - - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611 - - - Functest IRC chan: #opnfv-functest - - - Functest reporting: http://testresults.opnfv.org/reporting - - - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml - - - Functest Colorado user guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html - - - Functest installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html - - - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html - -.. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446 - -.. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454 - -.. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6 - -.. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174 - -.. _`apex-res-175`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-175 - -.. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176 - -.. _`apex-res-217`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-217 - -.. _`apex-res-235`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-235 - -.. _`apex-res-423`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-423 - -.. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55 - -.. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59 - -.. _`compass-res-73`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-73 - -.. _`compass-res-76`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-76 - -.. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77 - -.. _`compass-res-567`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-567 - -.. _`compass-res-285`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-285 - -.. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console - -.. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115 - -.. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117 - -.. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119 - -.. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123 - -.. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124 - -.. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128 - -.. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129 - -.. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154 - -.. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155 - -.. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160 - -.. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161 - -.. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162 - -.. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164 - -.. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166 - -.. _`fuel-res-213`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-213 - -.. _`fuel-res-219`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-219 - -.. _`fuel-res-376`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-376 - -.. _`fuel-res-492`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-492 - -.. _`fuel-arm-res-128`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/128/console - -.. _`fuel-arm-res-122`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/122/console - -.. _`fuel-arm-res-129`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/129/console - -.. _`fuel-arm-res-135`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/135/console - -.. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93 - -.. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91 - -.. _`joid-res-99`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-99 - -.. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97 - -.. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102 - -.. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103 - -.. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104 - -.. _`joid-res-345`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-345 - diff --git a/docs/release-notes/index.rst b/docs/release-notes/index.rst deleted file mode 100644 index 6c6d18109..000000000 --- a/docs/release-notes/index.rst +++ /dev/null @@ -1,15 +0,0 @@ -********************** -Functest Release Notes -********************** - -.. toctree:: - :numbered: - :maxdepth: 4 - - functest-release.rst - -Revision: _sha1_ - -:Author: Morgan Richomme (morgan.richomme@orange.com) - -Build date: |today| diff --git a/docs/release/release-notes/functest-release.rst b/docs/release/release-notes/functest-release.rst new file mode 100644 index 000000000..07389bb5b --- /dev/null +++ b/docs/release/release-notes/functest-release.rst @@ -0,0 +1,637 @@ +======= +License +======= + +OPNFV Colorado release note for Functest Docs +(c) by Morgan Richomme (Orange) + +OPNFV Colorado release note for Functest Docs +are licensed under a Creative Commons Attribution 4.0 International License. +You should have received a copy of the license along with this. +If not, see . + +=========================================== +OPNFV Colorado1.0 release note for Functest +=========================================== + +Abstract +======== + +This document describes the release note of Functest project. + + +Version history +=============== + ++------------+----------+------------------+------------------------+ +| **Date** | **Ver.** | **Author** | **Comment** | +| | | | | ++------------+----------+------------------+------------------------+ +| 2016-08-17 | 1.0.0 | Morgan Richomme | Functest for Colorado | +| | | (Orange) | release | ++------------+----------+------------------+------------------------+ + +OPNFV Colorado Release +========================= + +Functest deals with functional testing of the OPNFV solution. +It includes test cases developed within the project and test cases developed in +other OPNFV projects and other upstream communities. + +The internal test cases are: + + * healthcheck + * vPing ssh + * vPing userdata + * Tempest Smoke Serial + * Rally Sanity + * ODL + * Tempest full parallel + * Rally full + * vIMS + +The OPNFV projects integrated into Functest framework for automation are: + + * bgpvpn + * Copper + * Doctor + * Domino + * Moon + * Multisite + * ONOSFW + * ONOS-sfc + * ODL-sfc + * Parser + * Promise + * Security scan + +The validation of a scenario requires a subset of these tests depending +on the installer and the scenario. + +The 3 last internal test cases (tempest full parallel, Rally full and +vIMS) are not considered for scenario validation. + +Release Data +============ + ++--------------------------------------+--------------------------------------+ +| **Project** | functest | +| | | ++--------------------------------------+--------------------------------------+ +| **Repo/tag** | colorado.1.0 | +| | | ++--------------------------------------+--------------------------------------+ +| **Release designation** | Colorado base release | +| | | ++--------------------------------------+--------------------------------------+ +| **Release date** | September 22 2016 | +| | | ++--------------------------------------+--------------------------------------+ +| **Purpose of the delivery** | Colorado base release | +| | | ++--------------------------------------+--------------------------------------+ + +Deliverables +============ + +Software +-------- + + - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: colorado.1.0) + + - The testapi Docker image: https://hub.docker.com/r/opnfv/testapi (tag:colorado.1.0) + + +Documents +--------- + + - Installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html + + - User Guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html + + - Developer Guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html + + +Version change +============== + +Feature evolution +----------------- + + - refactoring of ODL functional tests (with upstream modifications) + + - refactoring of testapi (update, swagger documentation, dockerization) + + - jenkins logs improvement + + - update integration of Doctor, Promise and SDNVPN projects + + - split Tempest and rally into 2 different tests: smoke and full + + - vIMS test suite integration + + - adoption of Kibana for dashboarding + + +New features +------------ + + - Functest CLI to prepare and run the tests + + - creation of the healthcheck test case + + - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, fdio, multisite) + + - integration of new OPNFV feature projects (copper, domino, multisite, + moon, parser, onos-sfc, odl-sfc, security scan) + + - introduction of test tiers in functest framework + + - automatic reporting + + - introduction of a jenkins summary table + + - support of ARM architecture + + +Scenario Matrix +=============== + +For Colorado 1.0, Functest was tested on the following scenarios (if not +precised, the scenario is a HA scenario): + ++---------------------+---------+---------+---------+---------+ +| Scenario | Apex | Compass | Fuel | Joid | ++=====================+=========+=========+=========+=========+ +| nosdn | X | X | X | X | ++---------------------+---------+---------+---------+---------+ +| nosdn-noha | | | X | X | ++---------------------+---------+---------+---------+---------+ +| kvm | | | X | | ++---------------------+---------+---------+---------+---------+ +| kvm-noha | | | X | | ++---------------------+---------+---------+---------+---------+ +| lxd | | | | X | ++---------------------+---------+---------+---------+---------+ +| lxd-noha | | | | X | ++---------------------+---------+---------+---------+---------+ +| ovs-noha (dpdk) | | | X | | ++---------------------+---------+---------+---------+---------+ +| odl_l2 | X | X | X | X | ++---------------------+---------+---------+---------+---------+ +| odl_l2-noha | | | X | | ++---------------------+---------+---------+---------+---------+ +| odl_l2-bgpvpn | X | | X | | ++---------------------+---------+---------+---------+---------+ +| odl_l2-bgpvpn-noha| | | X | | ++---------------------+---------+---------+---------+---------+ +| odl_l2-fdio-noha | X | | | | ++---------------------+---------+---------+---------+---------+ +| odl_l2-moon | | X | | | ++---------------------+---------+---------+---------+---------+ +| odl_l2-sfc-noha | X | | X | | ++---------------------+---------+---------+---------+---------+ +| odl_l3 | X | X | X | | ++---------------------+---------+---------+---------+---------+ +| odl_l3-noha | | | X | | ++---------------------+---------+---------+---------+---------+ +| onos | | X | X | X | ++---------------------+---------+---------+---------+---------+ +| onos-noha | | | X | | ++---------------------+---------+---------+---------+---------+ +| onos-sfc | | X | X | X | ++---------------------+---------+---------+---------+---------+ +| onos-sfc-noha | | | X | | ++---------------------+---------+---------+---------+---------+ +| multisite | | | X | | ++---------------------+---------+---------+---------+---------+ + +For Colorado 2.0, the following scenarios have been re-released: + ++---------------------+---------+---------+---------+---------+ +| Scenario | Apex | Compass | Fuel | Joid | ++=====================+=========+=========+=========+=========+ +| onos | X | X | X | X | ++---------------------+---------+---------+---------+---------+ +| odl_l2-sfc-ha | | | X | | ++---------------------+---------+---------+---------+---------+ + + +In Colorado, the functional tests have been sliced in 6 different +categories: + ++----------------+-----------------------------------------------+ +| Category | Description | ++================+===============================================+ +| healthcheck | Basic OpenStack commands | ++----------------+-----------------------------------------------+ +| smoke | vPings, Tempest and rally smoke tests | ++----------------+-----------------------------------------------+ +| sdn_suites | Specific SDN feature tests | ++----------------+-----------------------------------------------+ +| features | OPNFV feature project functional test suites | ++----------------+-----------------------------------------------+ +| openstack | Advanced, long duration OpenStack tests | +| | (Tempest and Rally full suite). Each test may | +| | last several hours | ++----------------+-----------------------------------------------+ +| vnf | Complex scenarios dealing with orchestration, | +| | VNF deployment and tests (e.g. vIMS) | ++----------------+-----------------------------------------------+ + +For the scenario validation, we consider only the healthcheck, smoke, +sdn_suites and features categories. These tests are run systematically +in the CI daily loops. + +Success criteria have been defined for these test cases, they can be +PASS/FAIL or a success rate may be declared (100%, > 90%) +All the criteria, as well as the test dependencies are declared in the +ci/testcases.yaml file. + +The scoring for the Colorado release per installer can be described as +follows. + +The scoring is an indicator showing how many feature project test suites +have been integrated on the scenario. + +The scoring equals the number of tests * succesful iteration of each +test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617 + + e.g. + apex/odl_l2-nofeature-ha + tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper + Scoring = 21/21 = 7 * 3 + +By default, if not specified, the scenarios are HA. +HA means OpenStack High Availability (main services). Note that not +all VIM (e.g. OpenStack) services are deployed in HA mode, as that +depends upon support of the specific service for HA deployment. +For example, in the Colorado release, the Congress service +is deployed in non-HA mode even for HA OPNFV scenarios, as explicit +support for HA operation has not yet been verified. +See the release notes of the installers for more details. + + +Apex +---- + ++------------------+---------+---------+-------------------+ +| Scenario | Scoring | Success | Results | +| | | rate | | ++==================+=========+=========+===================+ +| nosdn | 17/18 | 95% | `apex-res-174`_ | ++------------------+---------+---------+-------------------+ +| odl_l2 | 21/21 | 100% | `apex-res-175`_ | ++------------------+---------+---------+-------------------+ +| odl_l3 | 15/18 | 83% | `apex-res-176`_ | ++------------------+---------+---------+-------------------+ +| odl_l2-bgpvpn | 14/18 | 78% | `apex-res-235`_ | ++------------------+---------+---------+-------------------+ +| odl_l2-fdio-noha | 12/15 | 80% | `apex-res-6`_ | ++------------------+---------+---------+-------------------+ +| odl_l2-sfc-noha | 18/24 | 75% | `apex-res-217`_ | ++------------------+---------+---------+-------------------+ +| onos-nofeature-ha| 20/21 | 95% | `apex-res-423`_ | ++------------------+---------+---------+-------------------+ + + +Compass +------- + ++------------------+---------+---------+-------------------+ +| Scenario | Scoring | Success | Results | +| | | rate | | ++==================+=========+=========+===================+ +| nosdn | 12/12 | 100% | `compass-res-55`_ | ++------------------+---------+---------+-------------------+ +| odl_l2 | 15/15 | 100% | `compass-res-59`_ | ++------------------+---------+---------+-------------------+ +| odl_l3 | 9/12 | 75% | `compass-res-73`_ | ++------------------+---------+---------+-------------------+ +| odl_l2-moon | 15/18 | 83% | `compass-res-567`_| ++------------------+---------+---------+-------------------+ +| onos-ha | 15/15 | 100% | `compass-res-285`_| ++------------------+---------+---------+-------------------+ +| onos-sfc-ha | 17/18 | 95% | `compass-res-76`_ | ++------------------+---------+---------+-------------------+ + +Note: all the Compass tests for Colorado have been executed on virtual +environment. Bare metal resources were used for Master branch. + + +Fuel +---- + ++---------------------+---------+---------+-------------------+ +| Scenario | Scoring | Success | Results | +| | | rate | | ++=====================+=========+=========+===================+ +| nosdn | 18/18 | 100% | `fuel-res-129`_ | ++---------------------+---------+---------+-------------------+ +| nosdn-noha | 15/15 | 100% | `fuel-res-154`_ | ++---------------------+---------+---------+-------------------+ +| nosdn-kvm | 18/18 | 100% | `fuel-res-128`_ | ++---------------------+---------+---------+-------------------+ +| nosdn-kvm-noha | 15/15 | 100% | `fuel-res-161`_ | ++---------------------+---------+---------+-------------------+ +| nosdn-ovs | 12/18 | 67%* | `fuel-res-213`_ | ++---------------------+---------+---------+-------------------+ +| nosdn-ovs-noha | 15/15 | 100% | `fuel-res-162`_ | ++---------------------+---------+---------+-------------------+ +| odl_l2 | 21/21 | 100% | `fuel-res-123`_ | ++---------------------+---------+---------+-------------------+ +| odl_l2-noha | 17/18 | 94% | `fuel-res-155`_ | ++---------------------+---------+---------+-------------------+ +| odl_l2-bgpvpn | 14/18 | 78% | `fuel-res-119`_ | ++---------------------+---------+---------+-------------------+ +| odl_l2-bgpvpn-noha | 14/15 | 93% | `fuel-res-160`_ | ++---------------------+---------+---------+-------------------+ +| odl_l2-sfc-noha | 6/21 | 29% | `fuel-res-219`_ | ++---------------------+---------+---------+-------------------+ +| odl_l2-sfc-ha | 16/21 | 76% | `fuel-res-376`_ | ++---------------------+---------+---------+-------------------+ +| odl_l3 | 15/18 | 83% | `fuel-res-115`_ | ++---------------------+---------+---------+-------------------+ +| odl_l3-noha | 12/15 | 80% | `fuel-res-164`_ | ++---------------------+---------+---------+-------------------+ +| onos | 20/21 | 95% | `fuel-res-492`_ | ++---------------------+---------+---------+-------------------+ +| onos-noha | 18/18 | 100% | `fuel-res-166`_ | ++---------------------+---------+---------+-------------------+ +| onos-sfc | 24/24 | 100% | `fuel-res-124`_ | ++---------------------+---------+---------+-------------------+ +| onos-sfc-noha | 21/21 | 100% | `fuel-res-129`_ | ++---------------------+---------+---------+-------------------+ +| multisite | N.R | 100% | `fuel-res-8`_ | ++---------------------+---------+---------+-------------------+ + +*: all results passed, lacking iterations to reach the full score + +Results of Functest on AArch64 Colorado 3.0 + ++---------------------+---------+---------+----------------------+ +| Scenario | Scoring | Success | Results | +| | | rate | | ++=====================+=========+=========+======================+ +| nosdn | 18/18 | 100% | `fuel-arm-res-128`_ | ++---------------------+---------+---------+----------------------+ +| odl_l2 | 21/21 | 100% | `fuel-arm-res-122`_ | ++---------------------+---------+---------+----------------------+ +| odl_l2-noha | 18/18 | 100% | `fuel-arm-res-129`_ | ++---------------------+---------+---------+----------------------+ +| odl_l3 | 17/18 | 95% | `fuel-arm-res-135`_ | ++---------------------+---------+---------+----------------------+ + + +Joid +---- + ++---------------------+---------+---------+-----------------+ +| Scenario | Scoring | Success | Results | +| | | rate | | ++=====================+=========+=========+=================+ +| nosdn | 18/18 | 100% | `joid-res-102`_ | ++---------------------+---------+---------+-----------------+ +| nosdn-noha | 17/18 | 95% | `joid-res-93`_ | ++---------------------+---------+---------+-----------------+ +| nosdn-lxd | 12/12 | 100% | `joid-res-104`_ | ++---------------------+---------+---------+-----------------+ +| nosdn-lxd-noha | 12/12 | 100% | `joid-res-91`_ | ++---------------------+---------+---------+-----------------+ +| odl_l2 | 21/21 | 100% | `joid-res-103`_ | ++---------------------+---------+---------+-----------------+ +| onos | 21/21 | 100% | `joid-res-345`_ | ++---------------------+---------+---------+-----------------+ +| onos-sfc | 24/24 | 100% | `joid-res-97`_ | ++---------------------+---------+---------+-----------------+ + +It is highly recommended to install a json viewer in your browser +(e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/) + +You can get additional details through test logs on http://artifacts.opnfv.org/. +As no search engine is available on the OPNFV artifact web site you must +retrieve the pod identifier on which the tests have been executed (see +field pod in any of the results) then click on the selected POD and look +for the date of the test you are interested in. + +The reporting pages can be found at: + + * apex: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html + * compass: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-compass.html + * fuel: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html + * joid: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-joid.html + +Colorado known restrictions/issues +================================== + ++-----------+-----------+----------------------------------------------+ +| Installer | Scenario | Issue | ++===========+===========+==============================================+ +| any | odl_l3-* | Tempest cases related to using floating IP | +| | | addresses fail because of a known ODL bug. | +| | | vPing_ssh test case is excluded for the same | +| | | reason. | +| | | https://jira.opnfv.org/browse/APEX-112 | +| | | https://jira.opnfv.org/browse/FUNCTEST-445 | ++-----------+-----------+----------------------------------------------+ +| apex/fuel | *-bgpvpn | vPing_ssh (floating ips not supported) and | +| | | vIMS excluded. Some Tempest cases related to | +| | | floating ips also excluded. Some performance | +| | | issues have been detected in this scenario | +| | | (i.e. BGPVPN extension enabled) when running | +| | | commands against the OpenStack APIs, thus | +| | | Rally sanity test case has been disabled. | +| | | Performance issues seem to be connected to | +| | | the ODL version. It is planned to reintroduce| +| | | Rally sanity in Colorado 2.0 with the | +| | | adoption of ODL Boron release. | ++-----------+-----------+----------------------------------------------+ +| apex | *-fdio | Due to late integration, fdio decided to | +| | | focus on mandatory tests and exclude feature | +| | | tests (copper, doctor, security_scan) from | +| | | its scenarios | ++-----------+-----------+----------------------------------------------+ +| compass | moon | First ODL test FAILS because ODL/Openstack | +| | | federation done in moon is partial. Only | +| | | MD-SAL is federated (not AD-SAL) | ++-----------+-----------+----------------------------------------------+ +| fuel | any | TestServerBasicOps test case skipped | +| | | https://gerrit.opnfv.org/gerrit/#/c/19635/ | ++-----------+-----------+----------------------------------------------+ +| fuel | kvm | vPing_ssh and vIMS excluded (metadata related| +| | | scenarios) | ++-----------+-----------+----------------------------------------------+ +| fuel | multisite | random errors running multisite. A known bug | +| | | in keystone mitaka, due to which memcache | +| | | raises exception and keystone becomes | +| | | unresponsive | +| | | bugs.launchpad.net/keystone/+bug/1600394 | +| | | workaround consists in restarting memcache on| +| | | server | ++-----------+-----------+----------------------------------------------+ +| joid | any | Tempest cases related to object storage | +| | | excluded | +| | | https://gerrit.opnfv.org/gerrit/#/c/17871/ | ++-----------+-----------+----------------------------------------------+ +| joid | domino | Domino tests are skipped in CI. However the | +| | | test case can be run by manually setting | +| | | IS_IPandKEY_CONFIGURED=true after manually | +| | | setting properly the IP addresses of the 3 | +| | | Controller nodes in the configuration file | +| | | /repos/domino/tests/run_multinode.sh | ++-----------+-----------+----------------------------------------------+ + + +Test and installer/scenario dependencies +======================================== + +It is not always possible to run all the test cases on all the scenarios. +The following table details the dependencies of the test cases per +scenario. The scenario dependencies (installer or scenario) are detailed +in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml + +Test results +============ + +Test results are available in: + + - test results document: http://artifacts.opnfv.org/functest + + - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/ + + - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/ + + + +Open JIRA tickets +================= + ++------------------+-----------------------------------------------+ +| JIRA | Description | ++==================+===============================================+ +| `FUNCTEST-446`_ | Cleanup ODL-SFC output in Functest execution | +| | Impact on odl_l2-sfc scenarios | ++------------------+-----------------------------------------------+ +| `FUNCTEST-454`_ | Cleanup failures when using HA networks in | +| | Neutron | ++------------------+-----------------------------------------------+ + +Useful links +============ + + - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing + + - wiki Functest Colorado page: https://wiki.opnfv.org/display/functest/Functest+Colorado + + - Functest repo: https://git.opnfv.org/cgit/functest + + - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/ + + - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611 + + - Functest IRC chan: #opnfv-functest + + - Functest reporting: http://testresults.opnfv.org/reporting + + - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml + + - Functest Colorado user guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html + + - Functest installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html + + - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html + +.. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446 + +.. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454 + +.. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6 + +.. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174 + +.. _`apex-res-175`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-175 + +.. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176 + +.. _`apex-res-217`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-217 + +.. _`apex-res-235`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-235 + +.. _`apex-res-423`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-423 + +.. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55 + +.. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59 + +.. _`compass-res-73`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-73 + +.. _`compass-res-76`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-76 + +.. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77 + +.. _`compass-res-567`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-567 + +.. _`compass-res-285`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-285 + +.. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console + +.. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115 + +.. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117 + +.. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119 + +.. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123 + +.. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124 + +.. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128 + +.. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129 + +.. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154 + +.. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155 + +.. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160 + +.. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161 + +.. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162 + +.. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164 + +.. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166 + +.. _`fuel-res-213`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-213 + +.. _`fuel-res-219`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-219 + +.. _`fuel-res-376`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-376 + +.. _`fuel-res-492`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-492 + +.. _`fuel-arm-res-128`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/128/console + +.. _`fuel-arm-res-122`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/122/console + +.. _`fuel-arm-res-129`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/129/console + +.. _`fuel-arm-res-135`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/135/console + +.. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93 + +.. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91 + +.. _`joid-res-99`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-99 + +.. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97 + +.. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102 + +.. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103 + +.. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104 + +.. _`joid-res-345`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-345 + diff --git a/docs/release/release-notes/index.rst b/docs/release/release-notes/index.rst new file mode 100644 index 000000000..6c6d18109 --- /dev/null +++ b/docs/release/release-notes/index.rst @@ -0,0 +1,15 @@ +********************** +Functest Release Notes +********************** + +.. toctree:: + :numbered: + :maxdepth: 4 + + functest-release.rst + +Revision: _sha1_ + +:Author: Morgan Richomme (morgan.richomme@orange.com) + +Build date: |today| diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst new file mode 100644 index 000000000..eee013678 --- /dev/null +++ b/docs/testing/developer/devguide/index.rst @@ -0,0 +1,993 @@ +****************************** +OPNFV FUNCTEST developer guide +****************************** + +.. toctree:: + :numbered: + :maxdepth: 2 + + +============ +Introduction +============ + +Functest is a project dealing with functional testing. +Functest produces its own internal test cases but can also be considered +as a framework to support feature and VNF onboarding project testing. +Functest developed a test API and defined a test collection framework +that can be used by any OPNFV project. + +Therefore there are many ways to contribute to Functest. You can: + + * Develop new internal test cases + * Integrate the tests from your feature project + * Develop the framework to ease the integration of external test cases + * Develop the API / Test collection framework + * Develop dashboards or automatic reporting portals + +This document describes how, as a developer, you may interact with the +Functest project. The first section details the main working areas of +the project. The Second part is a list of "How to" to help you to join +the Functest family whatever your field of interest is. + + +======================== +Functest developer areas +======================== + + +Functest High level architecture +================================ + +Functest is project delivering a test container dedicated to OPNFV. +It includes the tools, the scripts and the test scenarios. + +Functest can be described as follow:: + + +----------------------+ + | | + | +--------------+ | +-------------------+ + | | | | Public | | + | | Tools | +------------------+ OPNFV | + | | Scripts | | | System Under Test | + | | Scenarios | +------------------+ | + | | | | Management | | + | +--------------+ | +-------------------+ + | | + | Functest Docker | + | | + +----------------------+ + +Functest internal test cases +============================ +The internal test cases in Danube are: + + * healthcheck + * connection_check + * api_check + * vping_ssh + * vping_userdata + * odl + * snaps_smoke + * tempest_smoke_serial + * rally_sanity + * tempest_full_parallel + * rally_full + * cloudify_ims + +By internal, we mean that this particular test cases have been +developped and/or integrated by functest contributors and the associated +code is hosted in the Functest repository. +An internal case can be fully developped or a simple integration of +upstream suites (e.g. Tempest/Rally developped in OpenStack are just +integrated in Functest). +The structure of this repository is detailed in `[1]`_. +The main internal test cases are in the opnfv_tests subfolder of the +repository, the internal test cases are: + + * sdn: odl, onos + * openstack: healthcheck, vping_ssh, vping_userdata, tempest_*, rally_*, connection_check, api_check, snaps_smoke + * vnf: cloudify_ims + +If you want to create a new test case you will have to create a new +folder under the testcases directory. + +Functest external test cases +============================ +The external test cases are inherited from other OPNFV projects, +especially the feature projects. + +The external test cases are: + + * promise + * doctor + * onos + * bgpvpn + * copper + * security_scan + * sfc-odl + * sfc-onos + * parser + * domino + * multisite + * opera_ims + * orchestra_ims + + +The code to run these test cases may be directly in the repository of +the project. We have also a **features** sub directory under opnfv_tests +directory that may be used (it can be usefull if you want to reuse +Functest library). + +Functest framework +================== + +Functest can be considered as a framework. +Functest is release as a docker file, including tools, scripts and a CLI +to prepare the environement and run tests. +It simplifies the integration of external test suites in CI pipeline +and provide commodity tools to collect and display results. + +Since Colorado, test categories also known as tiers have been created to +group similar tests, provide consistant sub-lists and at the end optimize +test duration for CI (see How To section). + +The definition of the tiers has been agreed by the testing working group. + +The tiers are: + * healthcheck + * smoke + * features + * components + * performance + * vnf + * stress + +Functest abstraction classes +============================ + +In order to harmonize test integration, 3 abstraction classes have been +introduced in Danube: + + * testcase_base: base for any test case + * feature_base: abstraction for feature project + * vnf_base: abstraction for vnf onboarding + +The goal is to unify the way to run test from Functest. + +feature_base and vnf_base inherit from testcase_base. + + +-----------------------------------------+ + | | + | Testcase_base | + | | + | - init() | + | - run() | + | - publish_report() | + | - check_criteria() | + | | + +-----------------------------------------+ + | | + V V + +--------------------+ +--------------------------+ + | | | | + | feature_base | | vnf_base | + | | | | + | - prepare() | | - prepare() | + | - post() | | - deploy_orchestrator() | + | - parse_results() | | - deploy_vnf() | + | | | - test_vnf() | + | | | - clean() | + | | | - execute() | + | | | | + +--------------------+ +--------------------------+ + + +Functest util classes +===================== + +In order to simplify the creation of test cases, Functest develops some +functions that can be used by any feature or internal test cases. +Several features are supported such as logger, configuration management and +Openstack capabilities (snapshot, clean, tacker,..). +These functions can be found under /functest/utils and can be described as +follows: + +functest/utils/ +|-- config.py +|-- constants.py +|-- env.py +|-- functest_constants.py +|-- functest_logger.py +|-- functest_utils.py +|-- openstack_clean.py +|-- openstack_snapshot.py +|-- openstack_tacker.py +`-- openstack_utils.py + +Note that for Openstack, keystone v3 is now deployed by default by compass, +fuel and joid in Danube. All installers still support keysone v2 (deprecated in +next version). + +Test collection framework +========================= + +The OPNFV testing group created a test collection database to collect +the test results from CI: + + + http://testresults.opnfv.org/test/swagger/spec.html + + Authentication: opnfv/api@opnfv + +Any test project running on any lab integrated in CI can push the +results to this database. +This database can be used to see the evolution of the tests and compare +the results versus the installers, the scenarios or the labs. + + +Overall Architecture +-------------------- +The Test result management can be summarized as follows:: + + +-------------+ +-------------+ +-------------+ + | | | | | | + | Test | | Test | | Test | + | Project #1 | | Project #2 | | Project #N | + | | | | | | + +-------------+ +-------------+ +-------------+ + | | | + V V V + +-----------------------------------------+ + | | + | Test Rest API front end | + | http://testresults.opnfv.org/test | + | | + +-----------------------------------------+ + A | + | V + | +-------------------------+ + | | | + | | Test Results DB | + | | Mongo DB | + | | | + | +-------------------------+ + | + | + +----------------------+ + | | + | test Dashboard | + | | + +----------------------+ + +Test API description +-------------------- +The Test API is used to declare pods, projects, test cases and test +results. Pods are the pods used to run the tests. +The results pushed in the database are related to pods, projects and +cases. If you try to push results of test done on non referenced pod, +the API will return an error message. + +An additional method dashboard has been added to post-process +the raw results in release Brahmaputra (deprecated in Colorado). + +The data model is very basic, 4 objects are created: + + * Pods + * Projects + * Testcases + * Results + +Pods:: + + { + "id": , + "details": , + "creation_date": "YYYY-MM-DD HH:MM:SS", + "name": , + "mode": , + "role": + }, + +Projects:: + + { + "id": , + "name": , + "creation_date": "YYYY-MM-DD HH:MM:SS", + "description": + }, + +Testcases:: + + { + "id": , + "name":, + "project_name":, + "creation_date": "YYYY-MM-DD HH:MM:SS", + "description": , + "url": + }, + +Results:: + + { + "_id": , + "case_name": , + "project_name": , + "pod_name": , + "installer": , + "version": , + "start_date": "YYYY-MM-DD HH:MM:SS", + "stop_date": "YYYY-MM-DD HH:MM:SS", + "build_tag": , + "scenario": , + "criteria": , + "trust_indicator": { + "current": 0, + "histories": [] + } + } + +The API can described as follows. For detailed information, please go to + + http://testresults.opnfv.org/test/swagger/spec.html + + Authentication: opnfv/api@opnfv + +Version: + + +--------+--------------------------+-----------------------------------------+ + | Method | Path | Description | + +========+==========================+=========================================+ + | GET | /versions | Get all supported API versions | + +--------+--------------------------+-----------------------------------------+ + + +Pods: + + +--------+----------------------------+-----------------------------------------+ + | Method | Path | Description | + +========+============================+=========================================+ + | GET | /api/v1/pods | Get the list of declared Labs (PODs) | + +--------+----------------------------+-----------------------------------------+ + | POST | /api/v1/pods | Declare a new POD | + | | | Content-Type: application/json | + | | | { | + | | | "name": "pod_foo", | + | | | "mode": "metal", | + | | | "role": "ci-pod", | + | | | "details": "it is a ci pod" | + | | | } | + +--------+----------------------------+-----------------------------------------+ + | GET | /api/v1/pods/{pod_name} | Get a declared POD | + +--------+----------------------------+-----------------------------------------+ + +Projects: + + +--------+----------------------------+-----------------------------------------+ + | Method | Path | Description | + +========+============================+=========================================+ + | GET | /api/v1/projects | Get the list of declared projects | + +--------+----------------------------+-----------------------------------------+ + | POST | /api/v1/projects | Declare a new test project | + | | | Content-Type: application/json | + | | | { | + | | | "name": "project_foo", | + | | | "description": "whatever you want" | + | | | } | + +--------+----------------------------+-----------------------------------------+ + | DELETE | /api/v1/projects/{project} | Delete a test project | + +--------+----------------------------+-----------------------------------------+ + | GET | /api/v1/projects/{project} | Get details on a {project} | + | | | | + +--------+----------------------------+-----------------------------------------+ + | PUT | /api/v1/projects/{project} | Update a test project | + | | | | + | | | Content-Type: application/json | + | | | { | + | | | | + | | | } | + +--------+----------------------------+-----------------------------------------+ + + +Testcases: + + +--------+----------------------------+-----------------------------------------+ + | Method | Path | Description | + +========+============================+=========================================+ + | GET | /api/v1/projects/{project}/| Get the list of testcases of {project} | + | | cases | | + +--------+----------------------------+-----------------------------------------+ + | POST | /api/v1/projects/{project}/| Add a new test case to {project} | + | | cases | Content-Type: application/json | + | | | { | + | | | "name": "case_foo", | + | | | "description": "whatever you want" | + | | | "url": "whatever you want" | + | | | } | + +--------+----------------------------+-----------------------------------------+ + | DELETE | /api/v1/projects/{project}/| Delete a test case | + | | cases/{case} | | + +--------+----------------------------+-----------------------------------------+ + | GET | /api/v1/projects/{project}/| Get a declared test case | + | | cases/{case} | | + +--------+----------------------------+-----------------------------------------+ + | PUT | /api/v1/projects/{project}?| Modify a test case of {project} | + | | cases/{case} | | + | | | Content-Type: application/json | + | | | { | + | | | | + | | | } | + +--------+----------------------------+-----------------------------------------+ + +Results: + + +--------+----------------------------+------------------------------------------+ + | Method | Path | Description | + +========+============================+==========================================+ + | GET | /api/v1/results | Get all the test results | + +--------+----------------------------+------------------------------------------+ + | POST | /api/v1/results | Add a new test results | + | | | Content-Type: application/json | + | | | { | + | | | "project_name": "project_foo", | + | | | "scenario": "odl-l2", | + | | | "stop_date": "2016-05-28T14:42:58.384Z", | + | | | "trust_indicator": 0.5, | + | | | "case_name": "vPing", | + | | | "build_tag": "", | + | | | "version": "Colorado", | + | | | "pod_name": "pod_foo", | + | | | "criteria": "PASS", | + | | | "installer": "fuel", | + | | | "start_date": "2016-05-28T14:41:58.384Z",| + | | | "details": | + | | | } | + +--------+----------------------------+------------------------------------------+ + | GET | /api/v1/results? | Get the test results of {case} | + | | case={case} | | + +--------+----------------------------+------------------------------------------+ + | GET | /api/v1/results? | Get the test results of build_tag | + | | build_tag={tag} | {tag}. | + +--------+----------------------------+------------------------------------------+ + | GET | /api/v1/results? | Get last {N} records of test results | + | | last={N} | | + +--------+----------------------------+------------------------------------------+ + | GET | /api/v1/results? | Get the test results of scenario | + | | scenario={scenario} | {scenario}. | + +--------+----------------------------+------------------------------------------+ + | GET | /api/v1/results? | Get the test results of trust_indicator | + | | trust_indicator={ind} | {ind}. | + +--------+----------------------------+------------------------------------------+ + | GET | /api/v1/results? | Get the test results of last days | + | | period={period} | {period}. | + +--------+----------------------------+------------------------------------------+ + | GET | /api/v1/results? | Get the test results of {project} | + | | project={project} | | + +--------+----------------------------+------------------------------------------+ + | GET | /api/v1/results? | Get the test results of version | + | | version={version} | {version}. | + +--------+----------------------------+------------------------------------------+ + | GET | /api/v1/results? | Get the test results of criteria | + | | criteria={criteria} | {criteria}. | + +--------+----------------------------+------------------------------------------+ + | GET | /api/v1/results? | get the results on pod {pod} | + | | pod={pod} | | + +--------+----------------------------+------------------------------------------+ + | GET | /api/v1/results? | Get the test results of installer {inst} | + | | installer={inst} | | + +--------+----------------------------+------------------------------------------+ + | GET | /api/v1/results? | Get the results according to combined | + | | | query conditions supported above | + +--------+----------------------------+------------------------------------------+ + | GET | /api/v1/results/{result_id}| Get the test result by result_id | + +--------+----------------------------+------------------------------------------+ + +Scenarios: + + +--------+----------------------------+-----------------------------------------+ + | Method | Path | Description | + +========+============================+=========================================+ + | GET | /api/v1/scenarios | Get the list of declared scenarios | + +--------+----------------------------+-----------------------------------------+ + | POST | /api/v1/scenario | Declare a new scenario | + +--------+----------------------------+-----------------------------------------+ + | GET | /api/v1/scenario? | Get a declared scenario | + | | | | + +--------+----------------------------+-----------------------------------------+ + + +The code of the API is hosted in the releng repository `[6]`_. +The static documentation of the API can be found at `[17]`_. +The test API has been dockerized and may be installed locally in your +lab. See `[15]`_ for details. + +The deployment of the test API has been automated. +A jenkins job manages: + * the unit tests of the test api + * the creation of a new docker file + * the deployment of the new test api + * the archive of the old test api + * the backup of the Mongo DB + +Test API Authorization +~~~~~~~~~~~~~~~~~~~~~~ + +PUT/DELETE/POST operations of the testapi now require token based authorization. The token needs +to be added in the request using a header 'X-Auth-Token' for access to the database. + +e.g:: + headers['X-Auth-Token'] + +The value of the header i.e the token can be accessed in the jenkins environment variable +*TestApiToken*. The token value is added as a masked password. + +.. code-block:: python + + headers['X-Auth-Token'] = os.environ.get('TestApiToken') + +The above example is in Python. Token based authentication has been added so that only ci pods +jenkins job can have access to the database. + +Please note that currently token authorization is implemented but is not yet enabled. + + Automatic reporting + =================== + + An automatic reporting page has been created in order to provide a + consistant view of the scenarios. + In this page, each scenario is evaluated according to test criteria. + The code for the automatic reporting is available at `[8]`_. + + The results are collected from the centralized database every day and, + per scenario. A score is calculated based on the results from the last + 10 days. This score is the addition of single test scores. Each test + case has a success criteria reflected in the criteria field from the + results. + + Considering an instance of a scenario os-odl_l2-nofeature-ha, the + scoring is the addition of the scores of all the runnable tests from the + categories (tiers healthcheck, smoke and features) + corresponding to this scenario. + + + +---------------------+---------+---------+---------+---------+ + | Test | Apex | Compass | Fuel | Joid | + +=====================+=========+=========+=========+=========+ + | vPing_ssh | X | X | X | X | + +---------------------+---------+---------+---------+---------+ + | vPing_userdata | X | X | X | X | + +---------------------+---------+---------+---------+---------+ + | tempest_smoke_serial| X | X | X | X | + +---------------------+---------+---------+---------+---------+ + | rally_sanity | X | X | X | X | + +---------------------+---------+---------+---------+---------+ + | odl | X | X | X | X | + +---------------------+---------+---------+---------+---------+ + | promise | | | X | X | + +---------------------+---------+---------+---------+---------+ + | doctor | X | | X | | + +---------------------+---------+---------+---------+---------+ + | security_scan | X | | | | + +---------------------+---------+---------+---------+---------+ + | parser | | | X | | + +---------------------+---------+---------+---------+---------+ + | copper | X | | | X | + +---------------------+---------+---------+---------+---------+ + + All the testcases listed in the table are runnable on + os-odl_l2-nofeature scenarios. + If no result is available or if all the results are failed, the test + case get 0 point. + If it was succesfull at least once but not anymore during the 4 runs, + the case get 1 point (it worked once). + If at least 3 of the last 4 runs were successful, the case get 2 points. + If the last 4 runs of the test are successful, the test get 3 points. + + In the example above, the target score for fuel/os-odl_l2-nofeature-ha + is 3x6 = 18 points. + + The scenario is validated per installer when we got 3 points for all + individual test cases (e.g 18/18). + Please note that complex or long duration tests are not considered for + the scoring. The success criteria are not always easy to define and may + require specific hardware configuration. These results however provide + a good level of trust on the scenario. + + A web page is automatically generated every day to display the status. + This page can be found at `[9]`_. For the status, click on Status menu, + you may also get feedback for vims and tempest_smoke_serial test cases. + + Any validated scenario is stored in a local file on the web server. In + fact as we are using a sliding windows to get results, it may happen + that a successful scenarios is no more run (because considered as + stable) and then the number of iterations (4 needed) would not be + sufficient to get the green status. + + Please note that other test cases (e.g. sfc_odl, bgpvpn) need also + ODL configuration addons and as a consequence specific scenario. + There are not considered as runnable on the generic odl_l2 scenario. + +Dashboard +========= + +Dashboard is used to provide a consistant view of the results collected +in CI. +The results showed on the dashboard are post processed from the Database, +which only contains raw results. + +In Brahmaputra, we created a basic dashboard. +Since Colorado, it was decided to adopt ELK framework. Mongo DB results +are extracted to feed Elasticsearch database (`[7]`_). + +A script was developed to build elasticsearch data set. This +script can be found in `[16]`_. + +For next versions, it was decided to integrated bitergia dashboard. +Bitergia already provides a dashboard for code and infrastructure. +A new Test tab will be added. The dataset will be built by consuming +the test API. + + +======= +How TOs +======= + +How Functest works? +=================== + +The installation and configuration of the Functest docker image is +described in `[1]`_. + +The procedure to start tests is described in `[2]`_ + + +How can I contribute to Functest? +================================= + +If you are already a contributor of any OPNFV project, you can +contribute to functest. If you are totally new to OPNFV, you must first +create your Linux Foundation account, then contact us in order to +declare you in the repository database. + +We distinguish 2 levels of contributors: + + * the standard contributor can push patch and vote +1/0/-1 on any Functest patch + * The commitor can vote -2/-1/0/+1/+2 and merge + +Functest commitors are promoted by the Functest contributors. + + +Where can I find some help to start? +==================================== + +This guide is made for you. You can also have a look at the project wiki +page `[10]`_. +There are references on documentation, video tutorials, tips... + +You can also directly contact us by mail with [Functest] prefix in the +title at opnfv-tech-discuss@lists.opnfv.org or on the IRC chan +#opnfv-functest. + + +What kind of testing do you do in Functest? +=========================================== + +Functest is focusing on Functional testing. The results must be PASS or +FAIL. We do not deal with performance and/or qualification tests. +We consider OPNFV as a black box and execute our tests from the jumphost +according to Pharos reference technical architecture. + +Upstream test suites are integrated (Rally/Tempest/ODL/ONOS,...). +If needed Functest may bootstrap temporarily testing activities if they +are identified but not covered yet by an existing testing project (e.g +security_scan before the creation of the security repository) + + +How test constraints are defined? +================================= + +Test constraints are defined according to 2 paramaters: + + * The scenario (DEPLOY_SCENARIO env variable) + * The installer (INSTALLER_TYPE env variable) + +A scenario is a formal description of the system under test. +The rules to define a scenario are described in `[4]`_ + +These 2 constraints are considered to determinate if the test is runnable +or not (e.g. no need to run onos suite on odl scenario). + +In the test declaration for CI, the test owner shall indicate these 2 +constraints. The file testcases.yaml `[5]`_ must be patched in git to +include new test cases. A more elaborated system based on template is +planned for next releases + +For each dependency, it is possible to define a regex:: + + name: promise + criteria: 'success_rate == 100%' + description: >- + Test suite from Promise project. + dependencies: + installer: '(fuel)|(joid)' + scenario: '' + +In the example above, it means that promise test will be runnable only +with joid or fuel installers on any scenario. + +The vims criteria means any installer and exclude onos and odl with +bgpvpn scenarios:: + + name: vims + criteria: 'status == "PASS"' + description: >- + This test case deploys an OpenSource vIMS solution from Clearwater + using the Cloudify orchestrator. It also runs some signaling traffic. + dependencies: + installer: '' + scenario: '(ocl)|(nosdn)|^(os-odl)((?!bgpvpn).)*$' + + +How to write and check constaint regex? +======================================= + +Regex are standard regex. You can have a look at `[11]`_ + +You can also easily test your regex via an online regex checker such as `[12]`_. +Put your scenario in the TEST STRING window (e.g. os-odl_l3-ovs-ha), put +your regex in the REGULAR EXPRESSION window, then you can test your rule +. + + +How to know which test I can run? +================================= + +You can use the API `[13]`_. The static declaration is in git `[5]`_ + +If you are in a Functest docker container (assuming that the +environement has been prepared): just use the CLI. + +You can get the list per Test cases or by Tier:: + + # functest testcase list + healthcheck + vping_ssh + vping_userdata + tempest_smoke_serial + rally_sanity + odl + doctor + security_scan + tempest_full_parallel + rally_full + vims + # functest tier list + - 0. healthcheck: + ['healthcheck'] + - 1. smoke: + ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity'] + - 2. sdn_suites: + ['odl'] + - 3. features: + ['doctor', 'security_scan'] + - 4. openstack: + ['tempest_full_parallel', 'rally_full'] + - 5. vnf: + ['vims'] + + +How to manually start Functest tests? +===================================== + +Assuming that you are connected on the jumphost and that the system is +"Pharos compliant", i.e the technical architecture is compatible with +the one defined in the Pharos project:: + + # docker pull opnfv/functest:latest + # envs="-e INSTALLER_TYPE=fuel -e INSTALLER_IP=10.20.0.2 -e DEPLOY_SCENARIO=os-odl_l2-nofeature-ha -e CI_DEBUG=true" + # sudo docker run --privileged=true -id ${envs} opnfv/functest:latest /bin/bash + + +Then you must connect to the docker container and source the +credentials:: + + # docker ps (copy the id) + # docker exec -ti bash + # source $creds + + +You must first check if the environment is ready:: + + # functest env status + Functest environment ready to run tests. + + +If not ready, prepare the env by launching:: + + # functest env prepare + Functest environment ready to run tests. + +Once the Functest env is ready, you can use the CLI to start tests. + +You can run test cases per test case or per tier: + # functest testcase run or # functest tier run + + +e.g:: + + # functest testcase run tempest_smoke_serial + # functest tier run features + + +If you want to run all the tests you can type:: + + # functest testcase run all + + +If you want to run all the tiers (same at the end that running all the +test cases) you can type:: + + # functest tier run all + + +How to declare my tests in Functest? +==================================== + +If you want to add new internal test cases, you can submit patch under +the testcases directory of Functest repository. + +For feature test integration, the code can be kept into your own +repository. The Functest files to be modified are: + + * functest/docker/Dockerfile: get your code in Functest container + * functest/ci/testcases.yaml: reference your test and its associated constraints + + +Dockerfile +---------- + +This file lists the repositories (internal or external) to be cloned in +the Functest container. You can also add external packages:: + + RUN git clone https://gerrit.opnfv.org/gerrit/ ${REPOS_DIR}/ + +testcases.yaml +-------------- + +All the test cases that must be run from CI / CLI must be declared in +ci/testcases.yaml. + +This file is used to get the constraints related to the test:: + + name: + criteria: 'PASS', 'rate > 90%' + description: >- + + dependencies: + installer: regex related to installer e.g. 'fuel', '(apex)||(joid)' + scenario: regex related to the scenario e.g. 'ovs*no-ha' + + +You must declare your test case in one of the category (tier). + +If you are integrating test suites from a feature project, the default +category is **features**. + + +How to select my list of tests for CI? +====================================== + +Functest can be run automatically from CI, a jenkins job is usually +called after an OPNFV fresh installation. +By default we try to run all the possible tests (see `[14]` called from +Functest jenkins job):: + + cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t all ${flags}" + + +Each case can be configured as daily and/or weekly task. +Weekly tasks are used for long duration or experimental tests. +Daily tasks correspond to the minimum set of test suites to validate a scenario. + +When executing run_tests.py, a check based on the jenkins build tag will +be considered to detect whether it is a daily and/or a weekly test. + +in your CI you can customize the list of test you want to run by case or +by tier, just change the line:: + + cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t ${flags}" + +e.g.:: + + cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t healthcheck,smoke ${flags}" + +This command will run all the test cases of the first 2 tiers, i.e. +healthcheck, connection_check, api_check, vping_ssh, vping_userdata, +snaps_somke, tempest_smoke_serial and rally_sanity. + + +How to push your results into the Test Database +=============================================== + +The test database is used to collect test results. By default it is +enabled only for CI tests from Production CI pods. + +The architecture and associated API is described in previous chapter. +If you want to push your results from CI, you just have to call the API +at the end of your script. + +You can also reuse a python function defined in functest_utils.py:: + + def push_results_to_db(db_url, case_name, logger, pod_name,version, payload): + """ + POST results to the Result target DB + """ + url = db_url + "/results" + installer = get_installer_type(logger) + params = {"project_name": "functest", "case_name": case_name, + "pod_name": pod_name, "installer": installer, + "version": version, "details": payload} + + headers = {'Content-Type': 'application/json'} + try: + r = requests.post(url, data=json.dumps(params), headers=headers) + if logger: + logger.debug(r) + return True + except Exception, e: + print "Error [push_results_to_db('%s', '%s', '%s', '%s', '%s')]:" \ + % (db_url, case_name, pod_name, version, payload), e + return False + + +========== +References +========== + +_`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest configuration guide + +_`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide + +_`[3]`: https://wiki.opnfv.org/opnfv_test_dashboard Brahmaputra dashboard + +_`[4]`: https://wiki.opnfv.org/display/INF/CI+Scenario+Naming + +_`[5]`: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml + +_`[6]`: https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api + +_`[7]`: https://git.opnfv.org/cgit/releng/tree/utils/test/scripts + +_`[8]`: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting/functest + +_`[9]`: http://testresults.opnfv.org/reporting/ + +_`[10]`: https://wiki.opnfv.org/opnfv_functional_testing + +_`[11]`: https://docs.python.org/2/howto/regex.html + +_`[12]`: https://regex101.com/ + +_`[13]`: http://testresults.opnfv.org/test/api/v1/projects/functest/cases + +_`[14]`: https://git.opnfv.org/cgit/releng/tree/jjb/functest/functest-daily.sh + +_`[15]`: https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api/README.rst + +_`[16]`: https://git.opnfv.org/cgit/releng/tree/utils/test/scripts/mongo_to_elasticsearch.py + +_`[17]`: http://artifacts.opnfv.org/releng/docs/testapi.html + +OPNFV main site: http://www.opnfv.org + +OPNFV functional test page: https://wiki.opnfv.org/opnfv_functional_testing + +IRC support chan: #opnfv-functest + +_`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html + +_`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html + +_`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml diff --git a/docs/testing/developer/internship/security_group/index.rst b/docs/testing/developer/internship/security_group/index.rst new file mode 100644 index 000000000..d1cdbdd8f --- /dev/null +++ b/docs/testing/developer/internship/security_group/index.rst @@ -0,0 +1,70 @@ +======= +License +======= + +Functest Docs are licensed under a Creative Commons Attribution 4.0 +International License. +You should have received a copy of the license along with this. +If not, see . + +================================== +Functest Security group test cases +================================== + +Author: Girish Sukhatankar +mentors: D.Blaisonneau, J.Lausuch, M.Richomme + +Abstract +======== + + +Version history +=============== + ++------------+----------+------------------+------------------------+ +| **Date** | **Ver.** | **Author** | **Comment** | +| | | | | ++------------+----------+------------------+------------------------+ +| 2016-??-?? | 0.0.1 | Morgan Richomme | Beginning of the | +| | | (Orange) | Internship | ++------------+----------+------------------+------------------------+ + + +Overview: +========= + + + + +Problem Statement: +------------------ + + + +Curation Phase +-------------- + + + + + +Schedule: +========= + + + ++--------------------------+------------------------------------------+ +| **Date** | **Comment** | +| | | ++--------------------------+------------------------------------------+ +| December - January | ........ | ++--------------------------+------------------------------------------+ +| January - february | ........ | ++--------------------------+------------------------------------------+ + + +References: +=========== + +.. _`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Security+groups+test+case+in+Functest + diff --git a/docs/testing/developer/internship/testapi_evolution/index.rst b/docs/testing/developer/internship/testapi_evolution/index.rst new file mode 100644 index 000000000..6a1cde7df --- /dev/null +++ b/docs/testing/developer/internship/testapi_evolution/index.rst @@ -0,0 +1,237 @@ +======= +License +======= + +Functest Docs are licensed under a Creative Commons Attribution 4.0 +International License. +You should have received a copy of the license along with this. +If not, see . + +================== +Test API evolution +================== + +Author: Sakala Venkata Krishna Rohit +Mentors: S. Feng, J.Lausuch, M.Richomme + +Abstract +======== + +The testapi is used by all the test opnfv projects to report results. +It is also used to declare projects, test cases and labs. A major refactoring +has been done in Colorado with the introduction of swagger. The testapi is defined in Functest +developer guide. The purpose of this project is to add more features to the testapi that automate +the tasks that are done manually now, though there are tasks other than automation. + +Version history +=============== + ++------------+----------+------------------+------------------------+ +| **Date** | **Ver.** | **Author** | **Comment** | +| | | | | ++------------+----------+------------------+------------------------+ +| 2016-11-14 | 0.0.1 | Morgan Richomme | Beginning of the | +| | | (Orange) | Internship | ++------------+----------+------------------+------------------------+ +| 2017-02-17 | 0.0.2 | S.V.K Rohit | End of the Internship | +| | | (IIIT Hyderabad) | | ++------------+----------+------------------+------------------------+ + +Overview: +========= + +The internhip time period was from Nov 14th to Feb 17th. The project prosposal page is here `[1]`_. +The intern project was assigned to Svk Rohit and was mentored by S. Feng, J.Lausuch, M.Richomme. +The link to the patches submitted is `[2]`_. The internship was successfully completed and the +documentation is as follows. + +Problem Statement: +------------------ + +The problem statement could be divided into pending features that needed to be added into testapi +repo. The following were to be accomplished within the internship time frame. + +* **Add verification jenkins job for the testapi code** + The purpose of this job is to verify whehter the unit tests are successful or not with the + inclusion of the patchset submitted. + +* **Automatic update of opnfv/testapi docker image** + The docker image of testapi is hosted in the opnfv docker hub. To ensure that the testapi image + is always updated with the repository, automatic updation of the image is necessary and a job + is triggered whenever a new patch gets merged. + +* **Automation deployment of testresults.opnfv.org/test/ website** + In the same manner as the docker image of testapi is updated, the testapi website needs to be + in sync with the repository code. So, a job has been added to the opnfv jenkins ci for the + updation of the testresults website. + +* **Generate static documentation of testapi calls** + The purpose of this is to give an static/offline view of testapi. If someone wants to have a + look at the Restful apis of testapi, he/she does't need to go to the website, he can download + a html page and view it anytime. + +* **Backup MongoDB of testapi** + The mongoDB needs to be backed up every week. Till now it was done manually, but due to this + internship, it is now automated using a jenkins job. + +* **Add token based authorization to the testapi calls** + The token based authorization was implemented to ensure that only ci_pods could access the + database. Authentication has been added to only delete/put/post requests. + +Curation Phase: +--------------- + +The curation phase was the first 3 to 4 weeks of the internship. This phase was to get familiar +with the testapi code and functionality and propose the solutions/tools for the tasks mentioned +above. Swagger codegen was choosen out of the four tools proposed `[3]`_ for generating static +documentaion. + +Also, specific amount of time was spent on the script flow of the jenkins jobs. The automatic +deployment task involves accessing a remote server from inside the jenkins build. The deployment +had to be done only after the docker image update is done. For these constraints to satisfy, a +multijob jenkins job was choosen instead of a freestyle job. + +Important Links: +---------------- + +* MongoDB Backup Link - `[4]`_ +* Static Documentation - `[5]`_ +* TestAPI Token addition to ci_pods - `[6]`_ + +Schedule: +========= + +The progress and completion of the tasks is described in the below table. + ++--------------------------+------------------------------------------+ +| **Date** | **Comment** | +| | | ++--------------------------+------------------------------------------+ +| Nov 14th - Dec 31st | Understand Testapi code and the | +| | requirements. | ++--------------------------+------------------------------------------+ +| Jan 1st - Jan 7th | Add jenkins job to create static | +| | documentation and write build scripts. | ++--------------------------+------------------------------------------+ +| Jan 8th - Jan 21st | Add verification jenkins job for unit | +| | tests. | ++--------------------------+------------------------------------------+ +| Jan 22nd - Jan 28th | Add jenkins job for mongodb backup | +| | | ++--------------------------+------------------------------------------+ +| Jan 29th - Feb 11th | Enable automatic deployment of | +| | testresults.opnfv.org/test/ | ++--------------------------+------------------------------------------+ +| Feb 12th - Feb 17th | Add token based authentication | +| | | ++--------------------------+------------------------------------------+ + +FAQ's +===== + +This section lists the problems that I have faced and the understanding that I have acquired during +the internship. This section may help other developers in solving any errors casused because of the +code written as a part of this internship. + + +Test Api +-------- + +What is the difference between defining data_file as "/etc/.." and "etc/.." in setup.cfg ? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If in the setup.cfg, it is defined as + +[files] +data_files = +etc/a.conf = etc/a.conf.sample + +then it ends up installed in the /usr/etc/. With this configuration, it would be installed +correctly within a venv. but when it is defined as + +[files] +data_files = +/etc/a.conf = etc/a.conf.sample + +then it ends up installed on the root of the filesystem instead of properly be installed within the +venv. + +Which attribute does swagger-codegen uses as the title in the generation of document generation ? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It uses the nickname of the api call in swagger as the title in the generation of the document +generation. + +Does swagger-codegen take more than one yaml file as input ? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +No, swagger-codegen only takes one yaml file as input to its jar file. If there more than one yaml +file, one needs to merge them and give it as an input keeping mind the swagger specs. + + +Jenkins & JJB +------------- + +Which scm macro is used for verification jenkins jobs ? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are two macros for scm one is git-scm and other git-scm-gerrit. git-scm-gerrit is used for +verification jenkins job. + +Does the virtualenv created in one build script exists in other build scripts too ? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +No, the virtualenv created in one build script only exists in that build script/shell. + +What parameters are needed for the scm macros ? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Project and Branch are the two parameters needed for scm macros. + +What is the directory inside the jenkins build ? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The directory of the jenkins build is the directory of the repo. `ls $WORKSPACE` command will give +you all the contents of the directory. + +How to include a bash script in jenkins job yaml file ? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +An example might be apt here as an answer. + +builders: + - shell: + !include-raw: include-raw001-hello-world.sh + + +How do you make a build server run on a specific machine ? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It can be done by defining a label parameter 'SLAVE_LABEL' or in OPNFV , there are macros for each +server, one can use those parameter macros. +Ex: opnfv-build-defaults. Note, if we use macro, then no need to define GIT_BASE, but if one uses +SLAVE_LABEL, one needs to define a parameter GIT_BASE. This is because macro already has GIT_BASE +defined. + +What job style should be used when there is a situation like one build should trigger other builds +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +or when different build scripts need to be run on different machines ? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MultiJob style should be used as it has phases where each phase can be taken as a build scipt and +can have its own parameters by which one can define the SLAVE_LABEL parameter. + +References: +=========== + +_`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+testapi+evolution + +_`[2]` : https://gerrit.opnfv.org/gerrit/#/q/status:merged+owner:%22Rohit+Sakala+%253Crohitsakala%2540gmail.com%253E%22 + +_`[3]` : https://docs.google.com/document/d/1jWwVZ1ZpKgKcOS_zSz2KzX1nwg4BXxzBxcwkesl7krw/edit?usp=sharing + +_`[4]` : http://artifacts.opnfv.org/testapibackup.html + +_`[5]` : http://artifacts.opnfv.org/releng/docs/testapi.html + +_`[6]` : http://artifacts.opnfv.org/functest/docs/devguide/index.html#test-api-authorization diff --git a/docs/testing/developer/internship/unit_tests/index.rst b/docs/testing/developer/internship/unit_tests/index.rst new file mode 100644 index 000000000..f969aa72d --- /dev/null +++ b/docs/testing/developer/internship/unit_tests/index.rst @@ -0,0 +1,70 @@ +======= +License +======= + +Functest Docs are licensed under a Creative Commons Attribution 4.0 +International License. +You should have received a copy of the license along with this. +If not, see . + +=================== +Functest Unit tests +=================== + +Author: Ashish Kumar +Mentors: H.Yao, J.Lausuch, M.Richomme + +Abstract +======== + + +Version history +=============== + ++------------+----------+------------------+------------------------+ +| **Date** | **Ver.** | **Author** | **Comment** | +| | | | | ++------------+----------+------------------+------------------------+ +| 2016-??-?? | 0.0.1 | Morgan Richomme | Beginning of the | +| | | (Orange) | Internship | ++------------+----------+------------------+------------------------+ + + +Overview: +========= + + + + +Problem Statement: +------------------ + + + +Curation Phase +-------------- + + + + + +Schedule: +========= + + + ++--------------------------+------------------------------------------+ +| **Date** | **Comment** | +| | | ++--------------------------+------------------------------------------+ +| December - January | ........ | ++--------------------------+------------------------------------------+ +| January - february | ........ | ++--------------------------+------------------------------------------+ + + +References: +=========== + +.. _`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Functest+unit+tests + diff --git a/docs/testing/developer/internship/vnf_catalog/index.rst b/docs/testing/developer/internship/vnf_catalog/index.rst new file mode 100644 index 000000000..df7633391 --- /dev/null +++ b/docs/testing/developer/internship/vnf_catalog/index.rst @@ -0,0 +1,170 @@ +======= +License +======= + +Functest Docs are licensed under a Creative Commons Attribution 4.0 +International License. +You should have received a copy of the license along with this. +If not, see . + +======================= +Open Source VNF Catalog +======================= + +Author: Kumar Rishabh +Mentors: B.Souville, M.Richomme, J.Lausuch + +Abstract +======== + + + +Version hissory +=============== + ++------------+----------+------------------+------------------------+ +| **Date** | **Ver.** | **Author** | **Comment** | +| | | | | ++------------+----------+------------------+------------------------+ +| 2016-12-12 | 0.0.1 | Morgan Richomme | Beginning of the | +| | | (Orange) | Internship | ++------------+----------+------------------+------------------------+ + + +Overview: +========= + + +This project aims to create an Open Source catalog for reference and +classification of Virtual Network Functions (VNFs)s available on +Internet. The classification method proposed will be in sync with the +requirements of Telcos active in NFV landscape. The project aims to have +running web platform similar to [1] by the mid of internship (2nd week +of March). By the penultimate month of internship I aim to have fully +functional implementation of an Open Source VNF in functest. + + +Problem Statement: +------------------ + +OPNFV aims to be the reference platform for development, +standardization and integration of Open Source NFV components across +various Open Source Platforms. It mainly deals with the infrastructure +through the Network Function Virtualization Infrastructure (NFVI) and +Virtual Infrastructure manager (VIM). The MANO (Management and +orchestration) stacks have been introduced recently. VNFs are not +directly in OPNFV scope, however VNFs are needed to test and qualify the +infrastructure. In this regard having a common curated Open Source +Reference VNF catalog would be of immense importance to community. + +Since major focus of OPNFV is Telcos, a curated platform targeted from +industry point of view would be very useful. We plan to divide the +entire project into three major phases(with some iterative improvements +and overlaps) + + +Curation Phase +-------------- +This phase pertains to studying various Open Source VNFs available and +classification of them based on certain parameters. The parameters that +I currently have in mind are: + * Developer Metrics: These pertain to repo characteristics of VNF under + study + * Usage Statistics - Activity, Number of Commits, stars + * Maturity Statistics - For instance if an NFV community decides code + coverage is important for them, it shows the NFV community is serious + about taking the project forward + * Technical Tagging: These are the tags that pertain to technical + characteristics of a VNF + * Broad Use Cases - Whether the VNF fits strictly in IaaS, PaaS or + SaaS layer or is an hybrid of two/all. + * Generic Use Cases - This in my opinion is the broadest + classification category. For instance a VNF could be built with a + broad idea of powering IOT devices at home or from usage perspective + of Telco Operators (vFW, vEPC, vIMS, vCDN, vAAA, vCPE,...).`[2]`_ + * Fields of Application + * Library Status - Whether APIs are standardized, support RESTful + services. + * Dependency Forwarding Graph - This is pretty complex tagging + mechanism. It essentially tries to establish a graph relationship + between the VNFs (elementary VNFs are used in Service Function + Chaining chains such as Firewall, DPI, content enrichment,..). In my + opinion this is useful immensely. This will allow users to go to + platform and ask a question like - “I have this X tech stack to + support, Y and Z are my use cases, which NFVs should I use to support + this. + * Visitor Score - Based on `[1]`_ I plan to evolve a visitor score for + the platform. This will allow users to score an NFV on certain + parameters, may be post comments. + +**I plan to use the above three scores and evolve cumulative score which +will be displayed next to each of the NFV on the platform.** + + * Platform building phase - This will involve erecting a Web Platform + which will be similar to this `[1]`_. I am decently familiar with + Django and hence I will write the platform in Django. There are two + action plans that I have in mind right now. Either I can start writing + the platform simultaneously which will help keep track of my progress + or I can write the platform after 1.5 - 2 months into the internship. + Either way I aim to have the Web Platform ready by March 12. + + * Functest VNF implementation phase - This is the last phase that will + involve writing a fully functional implementation of an Open Source VNF + into Functest. I will undertake this after I am 3 months into the + internship. I have a decent familiarity with python and hence I think + it shouldn’t be too difficult. I need to decide how complex the VNFI + should undertake this exercise for (e.g. AAA such as free radius sounds + relatively easy, vCDN is much more challenging). + This will be decided in consent with my mentors. + + + + +Schedule: +========= +I plan to take this project in 6 months time frame as I want to use it +as a chance to read more about NFVs in particular and SDN in general + + ++--------------------------+------------------------------------------+ +| **Date** | **Comment** | +| | | ++--------------------------+------------------------------------------+ +| December 12 - January 12 | Study the above mentioned metrics | +| | Decide which of them are important for | +| | community (and which are not). | ++--------------------------+------------------------------------------+ +| January 12 - January 27 | Make a database for the above studied | +| | metrics and evolve it further based on | +| | Mentors’ input. + associated API | ++--------------------------+------------------------------------------+ +| January 27 - February 5 | Compile the data collected above and make| +| | it public. Although I can keep everything| +| | public from the beginning too. My | +| | rationale of not making the entire data | +| | public in initial stage as the errors | +| | caused by me could be misleading for | +| | developers. | ++--------------------------+------------------------------------------+ +| February 5 - March 5 | Erect the Web Platform and release it | +| | for restricted group for alpha testing. | ++--------------------------+------------------------------------------+ +| March 5 - March 12 | Make it public. Release it to public for | +| | beta testing. Fix Bugs. | ++--------------------------+------------------------------------------+ +| March 12 - April 12 | Start working on implementation of an | +| | Open Source VNF in Functest. | ++--------------------------+------------------------------------------+ +| April 12 - May 12 | I will decided what to do here based on | +| | discussion with mentors. | ++--------------------------+------------------------------------------+ + + +References: +=========== + +.. _`[1]` : Openhub: https://www.openhub.net/explore/projects + +.. _`[2]` : ETSI NFV White Paper: https://portal.etsi.org/Portals/0/TBpages/NFV/Docs/NFV_White_Paper3.pdf + +.. _`[3]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Open+Source+VNF+catalog diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst new file mode 100644 index 000000000..08e089c2b --- /dev/null +++ b/docs/testing/user/configguide/configguide.rst @@ -0,0 +1,821 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + + +Pulling the Docker image +------------------------ +Pull the Functest Docker image ('opnfv/functest') from the public +dockerhub registry under the OPNFV account: [dockerhub_], with the +following docker command:: + + docker pull opnfv/functest: + +where identifies a release of the Functest docker +container image in the public dockerhub registry. There are many tags +created automatically by the CI mechanisms, and you must ensure you +pull an image with the **correct tag** to match the OPNFV software +release installed in your environment. All available tagged images can +be seen from location [FunctestDockerTags_]. For example, when running +on the first official release of the OPNFV Colorado system platform, +tag "colorado.1.0" is needed. Pulling other tags might cause some +problems while running the tests. +Docker images pulled without a tag specifier bear the implicitly +assigned label "latest". If you need to specifically pull the latest +Functest docker image, then omit the tag argument:: + + docker pull opnfv/functest + +After pulling the Docker image, check that it is available with the +following docker command:: + + [functester@jumphost ~]$ docker images + REPOSITORY TAG IMAGE ID CREATED SIZE + opnfv/functest latest 8cd6683c32ae 2 weeks ago 1.321 GB + opnfv/functest danube.1.0 13fa54a1b238 4 weeks ago 1.29 GB + opnfv/functest colorado.1.0 94b78faa94f7 9 weeks ago 968 MB + +The Functest docker container environment can -in principle- be also +used with non-OPNFV official installers (e.g. 'devstack'), with the +**disclaimer** that support for such environments is outside of the +scope and responsibility of the OPNFV project. + +Accessing the Openstack credentials +----------------------------------- +OpenStack credentials are mandatory and can be retrieved in different +ways. From inside the running Functest docker container the +"functest env prepare" command will automatically look for the +Openstack credentials file "/home/opnfv/functest/conf/openstack.creds" +and retrieve it unless the file already exists. This Functest +environment preparation step is described later in this document. + +WARNING: When the installer type is "joid" you have to have the +credentials file inside the running container **before** initiating the +functest environment preparation. For that reason you have to choose +either one of the options below, since the automated copying does not +work for "joid". + +You can also specifically pass in the needed file prior to running the +environment preparation either: + + * by using the -v option when creating the Docker container. This is + referred to in docker documentation as "Bind Mounting". See the + usage of this parameter in the following chapter. + * or creating a local file '/home/opnfv/functest/conf/openstack.creds' + inside the running container with the credentials in it. Consult + your installer guide for further details. This is however not + instructed in this document. + +NOTE: When the installer type is "fuel" and virtualized deployment +is used, there you have to explicitly fetch the credentials file +executing the following sequence + + #. Create a container as described in next chapter but do not + "Bind Mount" the credentials + #. Log in to container and execute the following command. Replace + the IP with installer address after the "-a" parameter:: + + $REPOS_DIR/releng/utils/fetch_os_creds.sh \ + -d /home/opnfv/functest/conf/openstack.creds \ + -i fuel \ + -a 10.20.0.2 \ + -v + ( -d specifies the full path to the Openstack credential file + -i specifies the INSTALLER_TYPE + -a specifies the INSTALLER_IP + -v indicates a virtualized environment and takes no arguments ) + + #. Continue with your testing, initiate functest environment + preparation, run tests etc. + +In proxified environment you may need to change the credentials file. +There are some tips in chapter: `Proxy support`_ + +Functest Docker parameters +-------------------------- +This chapter explains how to run a container for executing functest +test suites. Numbered list below explains some details of the +recommended parameters for invoking docker container + + #. It is a good practice to assign a precise container name through + the **--name** option. + + #. Assign parameter for installer type:: + + -e "INSTALLER_TYPE=" + # Use one of following apex, compass, fuel or joid + + #. Functest needs to know the IP of some installers:: + + -e "INSTALLER_IP=" + + This IP is needed to fetch RC file from deployment, fetch logs, ... + If not provided, there is no way to fetch the RC file. It must be + provided manually as a volume + + #. Credentials for accessing the Openstack. + Most convenient way of passing them to container is by having a + local copy of the credentials file in Jumphost and then using the + **-v** option. In the example we have local file by the name of + "overcloudrc" and we are using that as an argument:: + + -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds + + The credentials file needs to exist in the Docker container + under the path: '/home/opnfv/functest/conf/openstack.creds'. + + **WARNING:** If you are using the Joid installer, you must pass the + credentials using the **-v** option: + -v /var/lib/jenkins/admin-openrc:/home/opnfv/functest/conf/openstack.creds. + See the section `Accessing the Openstack credentials`_ above. + + #. Passing deployment scenario + When running Functest against any of the supported OPNFV scenarios, + it is recommended to include also the environment variable + **DEPLOY_SCENARIO**. The **DEPLOY_SCENARIO** environment variable + is passed with the format:: + + -e "DEPLOY_SCENARIO=os---" + where: + os = OpenStack (No other VIM choices currently available) + controller is one of ( nosdn | odl_l2 | odl_l3 | onos | ocl) + nfv_feature is one or more of ( ovs | kvm | sfc | bgpvpn | nofeature ) + If several features are pertinent then use the underscore + character '_' to separate each feature (e.g. ovs_kvm) + 'nofeature' indicates no NFV feature is deployed + ha_mode (high availability) is one of ( ha | noha ) + + **NOTE:** Not all possible combinations of "DEPLOY_SCENARIO" are + supported. The name passed in to the Functest Docker container + must match the scenario used when the actual OPNFV platform was + deployed. See release note to see the list of supported scenarios. + + **NOTE:** The scenario name is mainly used to automatically detect + if a test suite is runnable or not (e.g. it will prevent ONOS test suite + to be run on ODL scenarios). If not set, Functest will try to run the + default test cases that might not include SDN controller or a specific + feature + + **NOTE:** A HA scenario means that 3 OpenStack controller nodes are + deployed. It does not necessarily mean that the whole system is HA. See + installer release notes for details. + + +Putting all above together, when using installer 'fuel' and an invented +INSTALLER_IP of '10.20.0.2', the recommended command to create the +Functest Docker container is as follows:: + + docker run --name "FunctestContainer" -it \ + -e "INSTALLER_IP=10.20.0.2" \ + -e "INSTALLER_TYPE=fuel" \ + -e "DEPLOY_SCENARIO=os-odl_l2-ovs_kvm-ha" \ + -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds \ + opnfv/functest /bin/bash + +After the *run* command, a new prompt appears which means that we are inside +the container and ready to move to the next step. + +For tips on how to set up container with installer Apex, see chapter +`Apex Installer Tips`_. + +Finally, three additional environment variables can also be passed in +to the Functest Docker Container, using the -e +"=" mechanism. The first two of these are +only relevant to Jenkins CI invoked testing and **should not be used** +when performing manual test scenarios:: + + -e "NODE_NAME=" \ + -e "BUILD_TAG=" \ + -e "CI_DEBUG=" + where: + = Symbolic name of the POD where the tests are run. + Visible in test results files, which are stored + to the database. This option is only used when + tests are activated under Jenkins CI control. + It indicates the POD/hardware where the test has + been run. If not specified, then the POD name is + defined as "Unknown" by default. + DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS. + = Symbolic name of the Jenkins Build Job. + Visible in test results files, which are stored + to the database. This option is only set when + tests are activated under Jenkins CI control. + It enables the correlation of test results, + which + are independently pushed to the results datbase + from different Jenkins jobs. + DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS. + = "true" or "false" + Default = "false", if not specified + If "true" is specified, then additional debug trace + text can be sent to the test results file / log files + and also to the standard console output. + +Apex Installer Tips +------------------- +Some specific tips are useful for the Apex Installer case. If not using +Apex Installer; ignore this section. + +In case of Triple-O based installer (like Apex) the docker container +needs to connect to the installer VM, so it is then required that some +known SSH keys are present in docker container. Since the Jumphost root +SSH keys are already known, easiest way is to use those using the +'Bind mount' method. See below for sample parameter:: + + -v /root/.ssh/id_rsa:/root/.ssh/id_rsa + + NOTE: You need the "sudo" when creating the container to access root + users ssh credentials even the docker command itself might not + require that. + +HINT! In case of Triple-O installers you can find value for the +INSTALLER_IP parameter by executing command and note the returned IP +address:: + + inst=$(sudo virsh list | grep -iEo "undercloud|instack") + sudo virsh domifaddr ${inst} + + NOTE: In releases prior to Colorado, the name 'instack' was + used. Currently the name 'undercloud' is used. + +You can copy the credentials file from the "stack" users home directory +in installer VM to Jumphost. Please check the correct IP from the +command above. In the example below we are using invented IP address +"192.168.122.89":: + + scp stack@192.168.122.89:overcloudrc . + +Here is an example of the full docker command invocation for an Apex +installed system, using latest Functest docker container, for +illustration purposes:: + + sudo docker run -it --name "ApexFuncTestODL" \ + -e "INSTALLER_IP=192.168.122.89" \ + -e "INSTALLER_TYPE=apex" \ + -e "DEPLOY_SCENARIO=os-odl_l2-nofeature-ha" \ + -v /root/.ssh/id_rsa:/root/.ssh/id_rsa \ + -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds \ + opnfv/functest /bin/bash + +Compass installer local development env usage Tips +-------------------------------------------------- +In the compass-functest local test case check and development environment, +in order to get openstack service inside the functest container, some +parameters should be configured during container creation, which are +hard to guess for freshman. This section will provide the guideline, the +parameters values are defaults here, which should be adjusted according +to the settings, the complete steps are given here so as not to appear +too abruptly. + +1, Pull Functest docker image from public dockerhub:: + + docker pull opnfv/functest: + + here can be "brahmaputra.1.0", "colorado.1.0", etc. +Tag omitted means the latest docker image:: + + docker pull opnfv/functest + +2, Functest Docker container creation + +To make a file used for the environment, such as 'functest-docker-env':: + + OS_AUTH_URL=http://172.16.1.222:35357/v2.0 + OS_USERNAME=admin + OS_PASSWORD=console + OS_TENANT_NAME=admin + OS_VOLUME_API_VERSION=2 + OS_PROJECT_NAME=admin + INSTALLER_TYPE=compass + INSTALLER_IP=192.168.200.2 + EXTERNAL_NETWORK=ext-net + +Note: please adjust the content according to the environment, such as +'TENANT_ID' maybe used for some special cases. + +Then to create the Functest docker:: + + docker run --privileged=true --rm -t \ + --env-file functest-docker-env \ + --name \ + opnfv/functest: /bin/bash + +3, To attach Functest container + +Before trying to attach the Functest container, the status can be checked by:: + + docker ps -a + +to attach the 'Up' status Functest container and start bash mode:: + + docker exec -it bash + +4, Functest environemnt preparation and check + +To see the Section below `Preparing the Functest environment`_. + +Functest docker container directory structure +--------------------------------------------- +Inside the Functest docker container, the following directory structure +should now be in place:: + + `-- home + `-- opnfv + |-- functest + | |-- conf + | |-- data + | `-- results + `-- repos + |-- bgpvpn + |-- copper + |-- doctor + |-- domino + |-- functest + |-- kingbird + |-- odl_test + |-- onos + |-- ovno + |-- parser + |-- promise + |-- rally + |-- releng + |-- sdnvpn + |-- securityscanning + |-- sfc + |-- tempest + |-- vims_test + `-- vnfs + +Underneath the '/home/opnfv/' directory, the Functest docker container +includes two main directories: + + * The **functest** directory stores configuration files (e.g. the + OpenStack creds are stored in path '/home/opnfv/functest/conf/openstack.creds'), + the **data** directory stores a 'cirros' test image used in some + functional tests and the **results** directory stores some temporary + result log files + * The **repos** directory holds various repositories. The directory + '/home/opnfv/repos/functest' is used to prepare the needed Functest + environment and to run the tests. The other repository directories + are used for the installation of the needed tooling (e.g. rally) or + for the retrieval of feature projects scenarios (e.g. promise) + +The structure under the **functest** repository can be described as +follows:: + + . |-- INFO + |-- LICENSE + |-- requirements.txt + |-- run_unit_tests.sh + |-- setup.py + |-- test-requirements.txt + |-- commons + | |-- ims + | |-- mobile + | `--traffic-profile-guidelines.rst + |-- docker + | |-- Dockerfile + | |-- config_install_env.sh + | `-- docker_remote_api + |-- docs + | |-- com + | |-- configguide + | |-- devguide + | |-- images + | |-- internship + | |-- release-notes + | |-- results + | `--userguide + |-- functest + |-- __init__.py + |-- ci + | |-- __init__.py + | |-- check_os.sh + | |-- config_functest.yaml + | |-- config_patch.yaml + | |-- exec_test.sh + | |-- generate_report.py + | |-- prepare_env.py + | |-- run_tests.py + | |-- testcases.yaml + | |-- tier_builder.py + | `-- tier_handler.py + |-- cli + | |-- __init__.py + | |-- cli_base.py + | |-- commands + | |-- functest-complete.sh + | `-- setup.py + |-- core + | |-- __init__.py + | |-- feature_base.py + | |-- pytest_suite_runner.py + | |-- testcase_base.py + | |-- vnf_base.py + |-- opnfv_tests + | |-- __init__.py + | |-- features + | |-- mano + | |-- openstack + | |-- sdn + | |-- security_scan + | `-- vnf + |-- tests + | |-- __init__.py + | `-- unit + `-- utils + |-- __init__.py + |-- config.py + |-- constants.py + |-- env.py + |-- functest_constants.py + |-- functest_logger.py + |-- functest_utils.py + |-- openstack + |-- openstack_clean.py + |-- openstack_snapshot.py + |-- openstack_tacker.py + `-- openstack_utils.py + + + (Note: All *.pyc files removed from above list for brevity...) + +We may distinguish several directories, the first level has 4 directories: + +* **commons**: This directory is dedicated for storage of traffic + profile or any other test inputs that could be reused by any test + project. +* **docker**: This directory includes the needed files and tools to + build the Funtest Docker image. +* **docs**: This directory includes documentation: Release Notes, + User Guide, Configuration Guide and Developer Guide. Test results + are also located in a sub--directory called 'results'. +* **functest**: This directory contains all the code needed to run + functest internal cases and OPNFV onboarded feature or VNF test cases. + +Functest directory has 6 directories: + * **ci**: This directory contains test structure definition files + (e.g .yaml) and bash shell/python scripts used to + configure and execute Functional tests. The test execution script + can be executed under the control of Jenkins CI jobs. + * **cli**: This directory holds the python based Functest CLI utility + source code, which is based on the Python 'click' framework. + * **core**: This directory holds the python based Functest core + source code. Three abstraction classes have been created to ease + the integration of internal, feature or vnf cases. + * **opnfv_tests**: This directory includes the scripts required by + Functest internal test cases and other feature projects test cases. + * **tests**: This directory includes the functest unit tests + * **utils**: this directory holds Python source code for some general + purpose helper utilities, which testers can also re-use in their + own test code. See for an example the Openstack helper utility: + 'openstack_utils.py'. + +Useful Docker commands +---------------------- +When typing **exit** in the container prompt, this will cause exiting +the container and probably stopping it. When stopping a running Docker +container all the changes will be lost, there is a keyboard shortcut +to quit the container without stopping it: -P + -Q. To +reconnect to the running container **DO NOT** use the *run* command +again (since it will create a new container), use the *exec* or *attach* +command instead:: + + docker ps # + docker exec -ti /bin/bash + +There are other useful Docker commands that might be needed to manage possible +issues with the containers. + +List the running containers:: + + docker ps + +List all the containers including the stopped ones:: + + docker ps -a + +Start a stopped container named "FunTest":: + + docker start FunTest + +Attach to a running container named "StrikeTwo":: + + docker attach StrikeTwo + +It is useful sometimes to remove a container if there are some problems:: + + docker rm + +Use the *-f* option if the container is still running, it will force to +destroy it:: + + docker rm -f + +Check the Docker documentation dockerdocs_ for more information. + +Preparing the Functest environment +---------------------------------- +Once the Functest docker container is up and running, the required +Functest environment needs to be prepared. A custom built **functest** +CLI utility is available to perform the needed environment preparation +action. Once the environment is prepared, the **functest** CLI utility +can be used to run different functional tests. The usage of the +**functest** CLI utility to run tests is described further in the +Functest User Guide `OPNFV_FuncTestUserGuide`_ + +Prior to commencing the Functest environment preparation, we can check +the initial status of the environment. Issue the **functest env status** +command at the prompt:: + + functest env status + Functest environment is not installed. + + Note: When the Functest environment is prepared, the command will + return the status: "Functest environment ready to run tests." + +To prepare the Functest docker container for test case execution, issue +the **functest env prepare** command at the prompt:: + + functest env prepare + +This script will make sure that the requirements to run the tests are +met and will install the needed libraries and tools by all Functest +test cases. It should be run only once every time the Functest docker +container is started from scratch. If you try to run this command, on +an already prepared enviroment, you will be prompted whether you really +want to continue or not:: + + functest env prepare + It seems that the environment has been already prepared. + Do you want to do it again? [y|n] + + (Type 'n' to abort the request, or 'y' to repeat the + environment preparation) + + +To list some basic information about an already prepared Functest +docker container environment, issue the **functest env show** at the +prompt:: + + functest env show + +======================================================+ + | Functest Environment info | + +======================================================+ + | INSTALLER: apex, 192.168.122.89 | + | SCENARIO: os-odl_l2-nofeature-ha | + | POD: localhost | + | GIT BRANCH: master | + | GIT HASH: 5bf1647dec6860464eeb082b2875798f0759aa91 | + | DEBUG FLAG: false | + +------------------------------------------------------+ + | STATUS: ready | + +------------------------------------------------------+ + + Where: + + INSTALLER: Displays the INSTALLER_TYPE value + - here = "apex" + and the INSTALLER_IP value + - here = "192.168.122.89" + SCENARIO: Displays the DEPLOY_SCENARIO value + - here = "os-odl_l2-nofeature-ha" + POD: Displays the value passed in NODE_NAME + - here = "localhost" + GIT BRANCH: Displays the git branch of the OPNFV Functest + project repository included in the Functest + Docker Container. + - here = "master" + (In first official colorado release + would be "colorado.1.0") + GIT HASH: Displays the git hash of the OPNFV Functest + project repository included in the Functest + Docker Container. + - here = "5bf1647dec6860464eeb082b2875798f0759aa91" + DEBUG FLAG: Displays the CI_DEBUG value + - here = "false" + + NOTE: In Jenkins CI runs, an additional item "BUILD TAG" + would also be listed. The valaue is set by Jenkins CI. + +Finally, the **functest** CLI has a **--help** options: + +Some examples:: + + functest --help Usage: functest [OPTIONS] COMMAND [ARGS]... + + Options: + --version Show the version and exit. + -h, --help Show this message and exit. + + Commands: + env + openstack + testcase + tier + + functest env --help + Usage: functest env [OPTIONS] COMMAND [ARGS]... + + Options: + -h, --help Show this message and exit. + + Commands: + prepare Prepares the Functest environment. + show Shows information about the current... + status Checks if the Functest environment is ready... + +Checking Openstack and credentials +---------------------------------- +It is recommended and fairly straightforward to check that Openstack +and credentials are working as expected. + +Once the credentials are there inside the container, they should be +sourced before running any Openstack commands:: + + source /home/opnfv/functest/conf/openstack.creds + +After this, try to run any OpenStack command to see if you get any +output, for instance:: + + openstack user list + +This will return a list of the actual users in the OpenStack +deployment. In any other case, check that the credentials are sourced:: + + env|grep OS_ + +This command must show a set of environment variables starting with +*OS_*, for example:: + + OS_REGION_NAME=RegionOne + OS_DEFAULT_DOMAIN=default + OS_PROJECT_NAME=admin + OS_PASSWORD=admin + OS_AUTH_STRATEGY=keystone + OS_AUTH_URL=http://172.30.10.3:5000/v2.0 + OS_USERNAME=admin + OS_TENANT_NAME=admin + OS_ENDPOINT_TYPE=internalURL + OS_NO_CACHE=true + +If the OpenStack command still does not show anything or complains +about connectivity issues, it could be due to an incorrect url given to +the OS_AUTH_URL environment variable. Check the deployment settings. + +SSL Support +----------- +If you need to connect to a server that is TLS-enabled (the auth URL +begins with "https") and it uses a certificate from a private CA or a +self-signed certificate, then you will need to specify the path to an +appropriate CA certificate to use, to validate the server certificate +with the environment variable OS_CACERT:: + + echo $OS_CACERT + /etc/ssl/certs/ca.crt + +However, this certificate does not exist in the container by default. +It has to be copied manually from the OpenStack deployment. This can be +done in 2 ways: + + #. Create manually that file and copy the contents from the OpenStack + controller. + #. (Recommended) Add the file using a Docker volume when starting the + container:: + + -v :/etc/ssl/certs/ca.cert + +You might need to export OS_CACERT environment variable inside the +container:: + + export OS_CACERT=/etc/ssl/certs/ca.crt + +Certificate verification can be turned off using OS_INSECURE=true. For +example, Fuel uses self-signed cacerts by default, so an pre step would +be:: + + export OS_INSECURE=true + +Proxy support +------------- +If your Jumphost node is operating behind a http proxy, then there are +2 places where some special actions may be needed to make operations +succeed: + + #. Initial installation of docker engine First, try following the + official Docker documentation for Proxy settings. Some issues were + experienced on CentOS 7 based Jumphost. Some tips are documented + in section: `Docker Installation on CentOS behind http proxy`_ + below. + + #. Execution of the Functest environment preparation inside the + created docker container Functest needs internet access to + download some resources for some test cases. This might not + work properly if the Jumphost is connecting to internet + through a http Proxy. + +If that is the case, make sure the resolv.conf and the needed +http_proxy and https_proxy environment variables, as well as the +'no_proxy' environment variable are set correctly:: + + # Make double sure that the 'no_proxy=...' line in the + # 'openstack.creds' file is commented out first. Otherwise, the + # values set into the 'no_proxy' environment variable below will + # be ovewrwritten, each time the command + # 'source ~/functest/conf/openstack.creds' is issued. + + cd ~/functest/conf/ + sed -i 's/export no_proxy/#export no_proxy/' openstack.creds + source ./openstack.creds + + # Next calculate some IP addresses for which http_proxy + # usage should be excluded: + + publicURL_IP=$(echo $OS_AUTH_URL | grep -Eo "([0-9]+\.){3}[0-9]+") + + adminURL_IP=$(openstack catalog show identity | \ + grep adminURL | grep -Eo "([0-9]+\.){3}[0-9]+") + + export http_proxy="" + export https_proxy="" + export no_proxy="127.0.0.1,localhost,$publicURL_IP,$adminURL_IP" + + # Ensure that "git" uses the http_proxy + # This may be needed if your firewall forbids SSL based git fetch + git config --global http.sslVerify True + git config --global http.proxy + +Validation check: Before running **'functest env prepare'** CLI command, +make sure you can reach http and https sites from inside the Functest +docker container. + +For example, try to use the **nc** command from inside the functest +docker container:: + + nc -v opnfv.org 80 + Connection to opnfv.org 80 port [tcp/http] succeeded! + + nc -v opnfv.org 443 + Connection to opnfv.org 443 port [tcp/https] succeeded! + +Note: In a Jumphost node based on the CentOS family OS, the **nc** +commands might not work. You can use the **curl** command instead. + + curl http://www.opnfv.org:80 + + + curl https://www.opnfv.org:443 + + + (Ignore the content. If command returns a valid HTML page, it proves + the connection.) + +Docker Installation on CentOS behind http proxy +----------------------------------------------- +This section is applicable for CentOS family OS on Jumphost which +itself is behind a proxy server. In that case, the instructions below +should be followed **before** installing the docker engine:: + + 1) # Make a directory '/etc/systemd/system/docker.service.d' + # if it does not exist + sudo mkdir /etc/systemd/system/docker.service.d + + 2) # Create a file called 'env.conf' in that directory with + # the following contents: + [Service] + EnvironmentFile=-/etc/sysconfig/docker + + 3) # Set up a file called 'docker' in directory '/etc/sysconfig' + # with the following contents: + HTTP_PROXY="" + HTTPS_PROXY="" + http_proxy="${HTTP_PROXY}" + https_proxy="${HTTPS_PROXY}" + + 4) # Reload the daemon + systemctl daemon-reload + + 5) # Sanity check - check the following docker settings: + systemctl show docker | grep -i env + + Expected result: + ---------------- + EnvironmentFile=/etc/sysconfig/docker (ignore_errors=yes) + DropInPaths=/etc/systemd/system/docker.service.d/env.conf + +Now follow the instructions in [`InstallDockerCentOS`_] to download +and install the **docker-engine**. The instructions conclude with a +"test pull" of a sample "Hello World" docker container. This should now +work with the above pre-requisite actions. + +.. _dockerdocs: https://docs.docker.com/ +.. _dockerhub: https://hub.docker.com/r/opnfv/functest/ +.. _Proxy: https://docs.docker.com/engine/admin/systemd/#http-proxy +.. _FunctestDockerTags: https://hub.docker.com/r/opnfv/functest/tags/ +.. _InstallDockerCentOS: https://docs.docker.com/engine/installation/linux/centos/ +.. _OPNFV_FuncTestUserGuide: http://artifacts.opnfv.org/functest/docs/userguide/index.html diff --git a/docs/testing/user/configguide/index.rst b/docs/testing/user/configguide/index.rst new file mode 100644 index 000000000..f12739e31 --- /dev/null +++ b/docs/testing/user/configguide/index.rst @@ -0,0 +1,296 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +================================== +OPNFV FUNCTEST Configuration Guide +================================== + +.. toctree:: + :numbered: + :maxdepth: 2 + +Version history +=============== + ++------------+----------+------------------+----------------------------------+ +| **Date** | **Ver.** | **Author** | **Comment** | +| | | | | ++------------+----------+------------------+----------------------------------+ +| 2016-08-17 | 1.0.0 | Juha Haapavirta | Colorado release | +| | | Column Gaynor | | ++------------+----------+------------------+----------------------------------+ +| 2017-01-19 | 1.0.1 | Morgan Richomme | Adaptations for Danube | +| | | | * update testcase list | +| | | | * update docker command | ++------------+----------+------------------+----------------------------------+ + +Introduction +============ +This document describes how to install and configure Functest in OPNFV. +The Functest CLI is utilized during the Functest environment preparation +step. The given example commands should work in both virtual and bare +metal cases alike. + +High level architecture +----------------------- + +The high level architecture of Functest within OPNFV can be described as +follows:: + + CIMC/Lights+out management Admin Mgmt/API Public Storage Private + PXE + + + + + + + + | | | | | | + | +----------------------------+ | | | | | + | | | | | | | | + +-----+ Jumphost | | | | | | + | | +--------+ | | | | + | | | | | | | | + | | +--------------------+ | | | | | | + | | | | | | | | | | + | | | Tools | +----------------+ | | | + | | | - Rally | | | | | | | + | | | - Robot | | | | | | | + | | | - TestON | | | | | | | + | | | | |-------------------------+ | | + | | | Testcases | | | | | | | + | | | - VIM | | | | | | | + | | | -- healthcheck | | | | | | | + | | | -- vPing_ssh | | | | | | | + | | | -- vPing_userdata | | | | | | | + | | | -- SNAPS_cases | | | | | | | + | | | -- Tempest_smoke | | | | | | | + | | | -- Rally_sanity | | | | | | | + | | | -- Tempest_full | | | | | | | + | | | -- Rally_full | | | | | | | + | | | | | | | | | | + | | | - SDN Controller | | | | | | | + | | | -- odl | | | | | | | + | | | -- onos | | | | | | | + | | | | | | | | | | + | | | - Features | | | | | | | + | | | | | | | | | | + | | | - VNF | | | | | | | + | | | | | | | | | | + | | +--------------------+ | | | | | | + | | Functest Docker + | | | | | + | | | | | | | | + | | | | | | | | + | | | | | | | | + | +----------------------------+ | | | | | + | | | | | | + | +----------------+ | | | | | + | | 1 | | | | | | + +----+ +--------------+-+ | | | | | + | | | 2 | | | | | | + | | | +--------------+-+ | | | | | + | | | | 3 | | | | | | + | | | | +--------------+-+ | | | | | + | | | | | 4 | | | | | | + | +-+ | | +--------------+-+ | | | | | + | | | | | 5 +-------------+ | | | | + | +-+ | | nodes for | | | | | | + | | | | deploying +---------------------+ | | | + | +-+ | OPNFV | | | | | | + | | | +------------------------------+ | | + | +-+ SUT | | | | | | + | | +--------------------------------------+ | + | | | | | | | | + | | +----------------------------------------------+ + | +----------------+ | | | | | + | | | | | | + + + + + + + + SUT = System Under Test + +All the libraries and dependencies needed by all of the Functest tools +are pre-installed into the Docker image. This allows running Functest +on any platform on any Operating System. + +The automated mechanisms inside the Functest Docker container will: + + * Retrieve OpenStack credentials + * Prepare the environment according to the SUT + * Perform the appropriate functional tests + * Push the test results into the OPNFV test result database + +This Docker image can be integrated into CI or deployed independently. + +Please note that the Functest Docker container has been designed for +OPNFV, however, it would be possible to adapt it to any VIM + controller +environment, since most of the test cases are integrated from upstream +communities. + +The functional test cases are described in the Functest User Guide `[2]`_ + + +Prerequisites +============= +The OPNFV deployment is out of the scope of this document but it can be +found in http://docs.opnfv.org. +The OPNFV platform is considered as the System Under Test (SUT) in this +document. + +Several prerequisites are needed for Functest: + + #. A Jumphost to run Functest on + #. A Docker daemon shall be installed on the Jumphost + #. A public/external network created on the SUT + #. An admin/management network created on the SUT + #. Connectivity from the Jumphost to the SUT public/external network + #. Connectivity from the Jumphost to the SUT admin/management network + +WARNING: Connectivity from Jumphost is essential and it is of paramount +importance to make sure it is working before even considering to install +and run Functest. Make also sure you understand how your networking is +designed to work. + +NOTE: **Jumphost** refers to any server which meets the previous +requirements. Normally it is the same server from where the OPNFV +deployment has been triggered previously. + +NOTE: If your Jumphost is operating behind a company http proxy and/or +Firewall, please consult first the section `Proxy Support`_, towards +the end of this document. The section details some tips/tricks which +*may* be of help in a proxified environment. + +Docker installation +------------------- +Docker installation and configuration is only needed to be done once +through the life cycle of Jumphost. + +If your Jumphost is based on Ubuntu, RHEL or CentOS linux, please +consult the references below for more detailed instructions. The +commands below are offered as a short reference. + +*Tip:* For running docker containers behind the proxy, you need first +some extra configuration which is described in section +`Docker Installation on CentOS behind http proxy`_. You should follow +that section before installing the docker engine. + +Docker installation needs to be done as root user. You may use other +userid's to create and run the actual containers later if so desired. +Log on to your Jumphost as root user and install the Docker Engine +(e.g. for CentOS family):: + + curl -sSL https://get.docker.com/ | sh + systemctl start docker + + *Tip:* If you are working through proxy, please set the https_proxy + environment variable first before executing the curl command. + +Add your user to docker group to be able to run commands without sudo:: + + sudo usermod -aG docker + +A reconnect is needed. There are 2 ways for this: + #. Re-login to your account + #. su - + +References - Installing Docker Engine on different Linux Operating Systems: + * Ubuntu_ + * RHEL_ + * CentOS_ + +.. _Ubuntu: https://docs.docker.com/engine/installation/linux/ubuntulinux/ +.. _RHEL: https://docs.docker.com/engine/installation/linux/rhel/ +.. _CentOS: https://docs.docker.com/engine/installation/linux/centos/ + +Public/External network on SUT +------------------------------ +Some of the tests against the VIM (Virtual Infrastructure Manager) need +connectivity through an existing public/external network in order to +succeed. This is needed, for example, to create floating IPs to access +VM instances through the public/external network (i.e. from the Docker +container). + +By default, the four OPNFV installers provide a fresh installation with +a public/external network created along with a router. Make sure that +the public/external subnet is reachable from the Jumphost. + +*Hint:* For the given OPNFV Installer in use, the IP sub-net address +used for the public/external network is usually a planning item and +should thus be known. Consult the OPNFV Configuration guide `[4]`_, and +ensure you can reach each node in the SUT, from the Jumphost using the +'ping' command using the respective IP address on the public/external +network for each node in the SUT. The details of how to determine the +needed IP addresses for each node in the SUT may vary according to the +used installer and are therefore ommitted here. + +Connectivity to Admin/Management network on SUT +----------------------------------------------- +Some of the Functest tools need to have access to the OpenStack +admin/management network of the controllers `[1]`_. + +For this reason, check the connectivity from the Jumphost to all the +controllers in cluster in the OpenStack admin/management network range. + +Installation and configuration +============================== + +.. include:: ./configguide.rst + +Integration in CI +================= +In CI we use the Docker image and execute the appropriate commands within the +container from Jenkins. + +Docker creation in set-functest-env builder `[3]`_:: + + envs="-e INSTALLER_TYPE=${INSTALLER_TYPE} -e INSTALLER_IP=${INSTALLER_IP} -e NODE_NAME=${NODE_NAME}" + [...] + docker pull opnfv/functest:$DOCKER_TAG >/dev/null + cmd="sudo docker run -id ${envs} ${volumes} ${custom_params} ${TESTCASE_OPTIONS} opnfv/functest:${DOCKER_TAG} /bin/bash" + echo "Functest: Running docker run command: ${cmd}" + ${cmd} >${redirect} + sleep 5 + container_id=$(docker ps | grep "opnfv/functest:${DOCKER_TAG}" | awk '{print $1}' | head -1) + echo "Container ID=${container_id}" + if [ -z ${container_id} ]; then + echo "Cannot find opnfv/functest container ID ${container_id}. Please check if it is existing." + docker ps -a + exit 1 + fi + echo "Starting the container: docker start ${container_id}" + docker start ${container_id} + sleep 5 + docker ps >${redirect} + if [ $(docker ps | grep "opnfv/functest:${DOCKER_TAG}" | wc -l) == 0 ]; then + echo "The container opnfv/functest with ID=${container_id} has not been properly started. Exiting..." + exit 1 + fi + + cmd="python ${FUNCTEST_REPO_DIR}/functest/ci/prepare_env.py start" + echo "Executing command inside the docker: ${cmd}" + docker exec ${container_id} ${cmd} + + +Test execution in functest-all builder `[3]`_:: + + branch=${GIT_BRANCH##*/} + echo "Functest: run $FUNCTEST_SUITE_NAME on branch ${branch}" + cmd="functest testcase run $FUNCTEST_SUITE_NAME" + fi + container_id=$(docker ps -a | grep opnfv/functest | awk '{print $1}' | head -1) + docker exec $container_id $cmd + ret_value=$? + exit $ret_value + +Docker clean in functest-cleanup builder `[3]`_ calling docker rm and docker rmi + + +References +========== +.. _`[1]`: https://ask.openstack.org/en/question/68144/keystone-unable-to-use-the-public-endpoint/ +.. _`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html +.. _`[3]`: https://git.opnfv.org/cgit/releng/tree/jjb/functest/functest-ci-jobs.yml +.. _`[4]`: http://artifacts.opnfv.org/functest/danube/docs/configguide/index.html + + +OPNFV main site: opnfvmain_. + +OPNFV functional test page: opnfvfunctest_. + +IRC support channel: #opnfv-functest + +.. _opnfvmain: http://www.opnfv.org +.. _opnfvfunctest: https://wiki.opnfv.org/functest diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst new file mode 100644 index 000000000..9436de2b9 --- /dev/null +++ b/docs/testing/user/userguide/index.rst @@ -0,0 +1,548 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +========================= +OPNFV FUNCTEST user guide +========================= + +.. toctree:: + :maxdepth: 2 + +Version history +=============== ++------------+----------+------------------+----------------------------------+ +| **Date** | **Ver.** | **Author** | **Comment** | +| | | | | ++------------+----------+------------------+----------------------------------+ +| 2016-08-17 | 1.0.0 | Juha Haapavirta | Colorado release | +| | | Column Gaynor | | ++------------+----------+------------------+----------------------------------+ +| 2017-01-23 | 1.0.1 | Morgan Richomme | Adaptations for Danube | +| | | | | +| | | | | ++------------+----------+------------------+----------------------------------+ + + +Introduction +============ + +The goal of this document is to describe the OPNFV Functest test cases and to +provide a procedure to execute them. In the OPNFV Danube system release, +a Functest CLI utility is introduced for easier execution of test procedures. + +**IMPORTANT**: It is assumed here that the Functest Docker container is already +properly deployed and that all instructions described in this guide are to be +performed from *inside* the deployed Functest Docker container. + +.. include:: ./introduction.rst + +The different test cases are described in the remaining sections of this document. + +VIM (Virtualized Infrastructure Manager) +---------------------------------------- + +Healthcheck +^^^^^^^^^^^ +In Colorado release a new Tier 'healthcheck' with one testcase 'healthcheck' +was introduced. The healthcheck testcase verifies that some basic IP connectivity +and essential operations of OpenStack functionality over the command line are +working correctly. + +In particular, the following verifications are performed: + + * DHCP agent functionality for IP address allocation + * Openstack Authentication management functionality via the Keystone API + * OpenStack Image management functionality via the Glance API + * OpenStack Block Storage management functionality via the Cinder API + * OpenStack Networking management functionality via the Neutron API + * Openstack Compute management functionality via the NOVA API + +Self-obviously, successful completion of the 'healthcheck' testcase is a +necessary pre-requisite for the execution of all other test Tiers. + + +vPing_ssh +^^^^^^^^^ + +Given the script **ping.sh**:: + + #!/bin/sh + while true; do + ping -c 1 $1 2>&1 >/dev/null + RES=$? + if [ "Z$RES" = "Z0" ] ; then + echo 'vPing OK' + break + else + echo 'vPing KO' + fi + sleep 1 + done + + +The goal of this test is to establish an SSH connection using a floating IP +on the Public/External network and verify that 2 instances can talk over a Private +Tenant network:: + + vPing_ssh test case + +-------------+ +-------------+ + | | | | + | | Boot VM1 with IP1 | | + | +------------------->| | + | Tester | | System | + | | Boot VM2 | Under | + | +------------------->| Test | + | | | | + | | Create floating IP | | + | +------------------->| | + | | | | + | | Assign floating IP | | + | | to VM2 | | + | +------------------->| | + | | | | + | | Establish SSH | | + | | connection to VM2 | | + | | through floating IP| | + | +------------------->| | + | | | | + | | SCP ping.sh to VM2 | | + | +------------------->| | + | | | | + | | VM2 executes | | + | | ping.sh to VM1 | | + | +------------------->| | + | | | | + | | If ping: | | + | | exit OK | | + | | else (timeout): | | + | | exit Failed | | + | | | | + +-------------+ +-------------+ + +This test can be considered as an "Hello World" example. +It is the first basic use case which **must** work on any deployment. + +vPing_userdata +^^^^^^^^^^^^^^ + +This test case is similar to vPing_ssh but without the use of Floating IPs +and the Public/External network to transfer the ping script. +Instead, it uses Nova metadata service to pass it to the instance at booting time. +As vPing_ssh, it checks that 2 instances can talk to +each other on a Private Tenant network:: + + vPing_userdata test case + +-------------+ +-------------+ + | | | | + | | Boot VM1 with IP1 | | + | +------------------->| | + | | | | + | | Boot VM2 with | | + | | ping.sh as userdata| | + | | with IP1 as $1. | | + | +------------------->| | + | Tester | | System | + | | VM2 exeutes ping.sh| Under | + | | (ping IP1) | Test | + | +------------------->| | + | | | | + | | Monitor nova | | + | | console-log VM 2 | | + | | If ping: | | + | | exit OK | | + | | else (timeout) | | + | | exit Failed | | + | | | | + +-------------+ +-------------+ + +When the second VM boots it will execute the script passed as userdata +automatically. The ping will be detected by periodically capturing the output +in the console-log of the second VM. + + +Tempest +^^^^^^^ + +Tempest `[2]`_ is the reference OpenStack Integration test suite. +It is a set of integration tests to be run against a live OpenStack cluster. +Tempest has suites of tests for: + + * OpenStack API validation + * Scenarios + * Other specific tests useful in validating an OpenStack deployment + +Functest uses Rally `[3]`_ to run the Tempest suite. +Rally generates automatically the Tempest configuration file **tempest.conf**. +Before running the actual test cases, +Functest creates the needed resources (user, tenant) and +updates the appropriate parameters into the configuration file. + +When the Tempest suite is executed, each test duration is measured and the full +console output is stored to a *log* file for further analysis. + +The Tempest testcases are distributed accross two +Tiers: + + * Smoke Tier - Test Case 'tempest_smoke_serial' + * Components Tier - Test case 'tempest_full_parallel' + +NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke +tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel' +executes all defined Tempest tests using several concurrent threads +(i.e. parallel mode). The number of threads activated corresponds to the number +of available logical CPUs. + +The goal of the Tempest test suite is to check the basic functionalities of the +different OpenStack components on an OPNFV fresh installation, using the +corresponding REST API interfaces. + + +Rally bench test suites +^^^^^^^^^^^^^^^^^^^^^^^ + +Rally `[3]`_ is a benchmarking tool that answers the question: + +*How does OpenStack work at scale?* + +The goal of this test suite is to benchmark all the different OpenStack modules and +get significant figures that could help to define Telco Cloud KPIs. + +The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios: + + * authenticate + * cinder + * glance + * heat + * keystone + * neutron + * nova + * quotas + * requests + +A basic SLA (stop test on errors) has been implemented. + +The Rally testcases are distributed accross two Tiers: + + * Smoke Tier - Test Case 'rally_sanity' + * Components Tier - Test case 'rally_full' + +NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test +cases. Test case 'rally_full' executes the full defined set of Rally tests. + +SNAPS +----- + +SNAPS stands for "SNA/NFV Application development Platform and Stack". +This project seeks to develop baseline OpenStack NFV installations. It has been +developed by Steven Pisarski and provided an object oriented library to perform +functional and performance tests. It has been declined in several test suites in +Functest, 2 are part of healthcheck tier, one belongs to smoke tier. + +connection check +^^^^^^^^^^^^^^^^ +Connection_check consists in 9 test cases (test duration < 5s) checking the +connectivity with Glance, Keystone, Neutron, Nova and the external network. + +api_check +^^^^^^^^^ +This test case verifies the retrieval of OpenStack clients: Keystone, Glance, +Neutron and Nova and may perform some simple queries. When the config value of +snaps.use_keystone is True, functest must have access to the cloud's private +network. +This suite consists in 49 tests (test duration< 2 minutes) + +snaps_smoke +^^^^^^^^^^^ +This test case contains tests that setup and destroy environments with VMs with +and without Floating IPs with a newly created user and project. Set the config +value snaps.use_floating_ips (True|False) to toggle this functionality. When +the config value of snaps.use_keystone is True, functest must have access +the cloud's private network. +This suite consists in 38 tests (test duration < 10 minutes) + +More information on SNAPS can be found in  `[13]`_ + + +SDN Controllers +--------------- + +There are currently 2 available controllers: + + * OpenDaylight (ODL) + * ONOS + +OpenDaylight +^^^^^^^^^^^^ + +The OpenDaylight (ODL) test suite consists of a set of basic tests inherited +from the ODL project using the Robot `[11]`_ framework. +The suite verifies creation and deletion of networks, subnets and ports with +OpenDaylight and Neutron. + +The list of tests can be described as follows: + + * Basic Restconf test cases + * Connect to Restconf URL + * Check the HTTP code status + + * Neutron Reachability test cases + * Get the complete list of neutron resources (networks, subnets, ports) + + * Neutron Network test cases + * Check OpenStack networks + * Check OpenDaylight networks + * Create a new network via OpenStack and check the HTTP status code returned by Neutron + * Check that the network has also been successfully created in OpenDaylight + + * Neutron Subnet test cases + * Check OpenStack subnets + * Check OpenDaylight subnets + * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron + * Check that the subnet has also been successfully created in OpenDaylight + + * Neutron Port test cases + * Check OpenStack Neutron for known ports + * Check OpenDaylight ports + * Create a new port via OpenStack and check the HTTP status code returned by Neutron + * Check that the new port has also been successfully created in OpenDaylight + + * Delete operations + * Delete the port previously created via OpenStack + * Check that the port has been also succesfully deleted in OpenDaylight + * Delete previously subnet created via OpenStack + * Check that the subnet has also been successfully deleted in OpenDaylight + * Delete the network created via OpenStack + * Check that the network has also been succesfully deleted in OpenDaylight + +Note: the checks in OpenDaylight are based on the returned HTTP status +code returned by OpenDaylight. + + +ONOS +^^^^ + +TestON Framework is used to test the ONOS SDN controller functions. +The test cases deal with L2 and L3 functions. +The ONOS test suite can be run on any ONOS compliant scenario. + +The test cases are described as follows: + + * onosfunctest: The main executable file contains the initialization of + the docker environment and functions called by FUNCvirNetNB and + FUNCvirNetNBL3 + + * FUNCvirNetNB + + * Create Network: Post Network data and check it in ONOS + * Update Network: Update the Network and compare it in ONOS + * Delete Network: Delete the Network and check if it's NULL in ONOS or + not + * Create Subnet: Post Subnet data and check it in ONOS + * Update Subnet: Update the Subnet and compare it in ONOS + * Delete Subnet: Delete the Subnet and check if it's NULL in ONOS or not + * Create Port: Post Port data and check it in ONOS + * Update Port: Update the Port and compare it in ONOS + * Delete Port: Delete the Port and check if it's NULL in ONOS or not + + * FUNCvirNetNBL3 + + * Create Router: Post data for create Router and check it in ONOS + * Update Router: Update the Router and compare it in ONOS + * Delete Router: Delete the Router data and check it in ONOS + * Create RouterInterface: Post Router Interface data to an existing Router + and check it in ONOS + * Delete RouterInterface: Delete the RouterInterface and check the Router + * Create FloatingIp: Post data for create FloatingIp and check it in ONOS + * Update FloatingIp: Update the FloatingIp and compare it in ONOS + * Delete FloatingIp: Delete the FloatingIp and check that it is 'NULL' in + ONOS + * Create External Gateway: Post data to create an External Gateway for an + existing Router and check it in ONOS + * Update External Gateway: Update the External Gateway and compare the change + * Delete External Gateway: Delete the External Gateway and check that it is + 'NULL' in ONOS + + +Features +-------- + +Please refer to the dedicated feature user guides for details: + + * bgpvpn: http://artifacts.opnfv.org/sdnvpn/danube/docs/userguide/index.html + * copper: http://artifacts.opnfv.org/copper/danube/docs/userguide/index.html + * doctor: http://artifacts.opnfv.org/doctor/danube/userguide/index.html + * domino: http://artifacts.opnfv.org/domino/docs/userguide-single/index.html + * multisites: http://artifacts.opnfv.org/multisite/docs/userguide/index.html + * onos-sfc: http://artifacts.opnfv.org/onosfw/danube/userguide/index.html + * odl-sfc: http://artifacts.opnfv.org/sfc/danube/userguide/index.html + * promise: http://artifacts.opnfv.org/danube/colorado/docs/userguide/index.html + * security_scan: http://artifacts.opnfv.org/security_scan/colorado/docs/userguide/index.html + * TODO + +security_scan +^^^^^^^^^^^^^ + +Security Scanning, is a project to insure security compliance and vulnerability +checks, as part of an automated CI / CD platform delivery process. + +The project makes use of the existing SCAP format `[6]`_ to perform deep +scanning of NFVI nodes, to insure they are hardened and free of known CVE +reported vulnerabilities. + +The SCAP content itself, is then consumed and run using an upstream opensource tool +known as OpenSCAP `[7]`_. + +The OPNFV Security Group have developed the code that will called by the OPNFV Jenkins +build platform, to perform a complete scan. Resulting reports are then copied to the +OPNFV functest dashboard. + +The current work flow is as follows: + + * Jenkins Build Initiated + * security_scan.py script is called, and a config file is passed to the script as + an argument. + * The IP addresses of each NFVi node (compute / control) are gathered + * A scan profile is matched to the node type. + * The OpenSCAP application is remotely installed onto each target node gathered + on step 3, using upstream packaging (rpm and .deb). + * A scan is made against each node gathered within step 3. + * HTML Reports are downloaded for rendering on a dashboard. + * If the config file value 'clean' is set to 'True' then the application installed in + step 5 is removed, and all reports created at step 6 are deleted. + +Security scan is supported by Apex, TODO.... + + + +VNF +--- + + +vIMS +^^^^ +The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an +architectural framework for delivering IP multimedia services. + +vIMS has been integrated in Functest to demonstrate the capability to deploy a +relatively complex NFV scenario on the OPNFV platform. The deployment of a complete +functional VNF allows the test of most of the essential functions needed for a +NFV platform. + +The goal of this test suite consists of: + + * deploy a VNF orchestrator (Cloudify) + * deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF from this + orchestrator based on a TOSCA blueprint defined in `[5]`_ + * run suite of signaling tests on top of this VNF + +The Clearwater architecture is described as follows: + +.. figure:: ../images/clearwater-architecture.png + :align: center + :alt: vIMS architecture + + +parser +^^^^^^ + +See parser user guide for details: `[12]`_ + + +.. include:: ./runfunctest.rst + + +Test results +============ + +Manual testing +-------------- + +In manual mode test results are displayed in the console and result files +are put in /home/opnfv/functest/results. + +Automated testing +-------------- + +In automated mode, test results are displayed in jenkins logs, a summary is provided +at the end of the job and can be described as follow:: + + +==================================================================================================================================================+ + | FUNCTEST REPORT | + +==================================================================================================================================================+ + | | + | Deployment description: | + | INSTALLER: fuel | + | SCENARIO: os-odl_l2-nofeature-ha | + | BUILD TAG: jenkins-functest-fuel-baremetal-daily-master-324 | + | CI LOOP: daily | + | | + +=========================+===============+============+===============+===========================================================================+ + | TEST CASE | TIER | DURATION | RESULT | URL | + +=========================+===============+============+===============+===========================================================================+ + | healthcheck | healthcheck | 03:07 | PASS | | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | vping_ssh | smoke | 00:56 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac13d79377c54b278bd4c1 | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | vping_userdata | smoke | 00:41 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac14019377c54b278bd4c2 | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | tempest_smoke_serial | smoke | 16:05 | FAIL | http://testresults.opnfv.org/test/api/v1/results/57ac17ca9377c54b278bd4c3 | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | rally_sanity | smoke | 12:19 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1aad9377c54b278bd4cd | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | odl | sdn_suites | 00:24 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1ad09377c54b278bd4ce | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | promise | features | 00:41 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1ae59377c54b278bd4cf | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + +Results are automatically pushed to the test results database, some additional +result files are pushed to OPNFV artifact web sites. + +Based on the results stored in the result database, a `Functest reporting`_ +portal is also automatically updated. This portal provides information on: + + * The overall status per scenario and per installer + * Tempest: Tempest test case including reported errors per scenario and installer + * vIMS: vIMS details per scenario and installer + +.. figure:: ../images/functest-reporting-status.png + :align: center + :alt: Functest reporting portal Fuel status page + + +Test Dashboard +============== + +Based on results collected in CI, a test dashboard is dynamically generated. + + +.. include:: ./troubleshooting.rst + + +References +========== + +.. _`[1]`: http://artifacts.opnfv.org/functest/colorado/docs/configguide/# +.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html +.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html +.. _`[4]`: http://events.linuxfoundation.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf +.. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml +.. _`[6]`: https://scap.nist.gov/ +.. _`[7]`: https://github.com/OpenSCAP/openscap +.. _`[9]`: https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/libraries/os_defaults.yaml +.. _`[11]`: http://robotframework.org/ +.. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html +.. _`[13]`: TODO URL doc SNAPS + +OPNFV main site: opnfvmain_. + +OPNFV functional test page: opnfvfunctest_. + +IRC support chan: #opnfv-testperf + +.. _opnfvmain: http://www.opnfv.org +.. _opnfvfunctest: https://wiki.opnfv.org/opnfv_functional_testing +.. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html +.. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html +.. _`config_test.py` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.py +.. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml +.. _`Functest reporting`: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html diff --git a/docs/testing/user/userguide/introduction.rst b/docs/testing/user/userguide/introduction.rst new file mode 100644 index 000000000..4dfe79375 --- /dev/null +++ b/docs/testing/user/userguide/introduction.rst @@ -0,0 +1,255 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +Overview of the Functest suites +=============================== + +Functest is the OPNFV project primarily targeting function testing. +In the Continuous Integration pipeline, it is launched after an OPNFV fresh +installation to validate and verify the basic functions of the +infrastructure. + +The current list of test suites can be distributed over 5 main domains: VIM +(Virtualised Infrastructure Manager), Controllers (i.e. SDN Controllers), +Features, VNF (Virtual Network Functions) and MANO stacks. + +Functest test suites are also distributed in the OPNFV testing categories: +healthcheck, smoke, features, components, performance, VNF, Stress tests. + +All the Healthcheck and smoke tests of a given scenario must be succesful to +validate the scenario for the release. + ++-------------+---------------+----------------+----------------------------------+ +| Domain | Tier | Test case | Comments | ++=============+===============+================+==================================+ +| VIM | healthcheck | healthcheck | Verify basic operation in VIM | +| | +----------------+----------------------------------+ +| | | connection | Check OpenStack connectivity | +| | | _check | through SNAPS framework | +| | +----------------+----------------------------------+ +| | | api_check | Check OpenStack API through | +| | | | SNAPS framework | +| +---------------+----------------+----------------------------------+ +| | smoke | vPing_SSH | NFV "Hello World" using an SSH | +| | | | connection to a destination VM | +| | | | over a created floating IP | +| | | | address on the SUT Public / | +| | | | External network. Using the SSH | +| | | | connection a test script is then | +| | | | copied to the destination | +| | | | VM and then executed via SSH. | +| | | | The script will ping another | +| | | | VM on a specified IP address over| +| | | | the SUT Private Tenant network. | +| | +----------------+----------------------------------+ +| | | vPing_userdata | Uses Ping with given userdata | +| | | | to test intra-VM connectivity | +| | | | over the SUT Private Tenant | +| | | | network. The correct operation | +| | | | of the NOVA Metadata service is | +| | | | also verified in this test. | +| | +----------------+----------------------------------+ +| | | tempest_smoke | Generate and run a relevant | +| | | \_serial | Tempest Test Suite in smoke mode.| +| | | | The generated test set is | +| | | | dependent on the OpenStack | +| | | | deployment environment. | +| | +----------------+----------------------------------+ +| | | rally_sanity | Run a subset of the OpenStack | +| | | | Rally Test Suite in smoke mode | +| | +----------------+----------------------------------+ +| | | snaps_smoke | Run a subset of the OpenStack | +| | | | Rally Test Suite in smoke mode | +| +---------------+----------------+----------------------------------+ +| | components | tempest_full | Generate and run a full set of | +| | | \_parallel | the OpenStack Tempest Test Suite.| +| | | | See the OpenStack reference test | +| | | | suite `[2]`_. The generated | +| | | | test set is dependent on the | +| | | | OpenStack deployment environment.| +| | +----------------+----------------------------------+ +| | | rally_full | Run the OpenStack testing tool | +| | | | benchmarking OpenStack modules | +| | | | See the Rally documents `[3]`_. | ++-------------+---------------+----------------+----------------------------------+ +| Controllers | smoke | odl | Opendaylight Test suite | +| | | | Limited test suite to check the | +| | | | basic neutron (Layer 2) | +| | | | operations mainly based on | +| | | | upstream testcases. See below | +| | | | for details | +| | +----------------+----------------------------------+ +| | | onos | Test suite of ONOS L2 and L3 | +| | | | functions. | +| | | | See `ONOSFW User Guide`_ for | +| | | | details. | ++-------------+---------------+----------------+----------------------------------+ +| Features | features | promise | Resource reservation and | +| | | | management project to identify | +| | | | NFV related requirements and | +| | | | realize resource reservation for | +| | | | future usage by capacity | +| | | | management of resource pools | +| | | | regarding compute, network and | +| | | | storage. | +| | | | See `Promise User Guide`_ for | +| | | | details. | +| | +----------------+----------------------------------+ +| | | doctor | Doctor platform, as of Colorado | +| | | | release, provides the three | +| | | | features: | +| | | | * Immediate Notification | +| | | | * Consistent resource state | +| | | | awareness for compute host down | +| | | | * Valid compute host status | +| | | | given to VM owner | +| | | | See `Doctor User Guide`_ for | +| | | | details | +| | +----------------+----------------------------------+ +| | | bgpvpn | Implementation of the OpenStack | +| | | | bgpvpn API from the SDNVPN | +| | | | feature project. It allows for | +| | | | the creation of BGP VPNs. | +| | | | See `SDNVPN User Guide`_ for | +| | | | details | +| | +----------------+----------------------------------+ +| | | security_scan | Implementation of a simple | +| | | | security scan. (Currently | +| | | | available only for the Apex | +| | | | installer environment) | +| | +----------------+----------------------------------+ +| | | onos-sfc | SFC testing for onos scenarios | +| | | | See `ONOSFW User Guide`_ for | +| | | | details | +| | +----------------+----------------------------------+ +| | | odl-sfc | SFC testing for odl scenarios | +| | | | See `SFC User Guide`_ for details| +| | +----------------+----------------------------------+ +| | | domino | Domino provides TOSCA template | +| | | | distribution service for network | +| | | | service and VNF descriptors | +| | | | among MANO components e.g., | +| | | | NFVO, VNFM, VIM, SDN-C, etc., | +| | | | as well as OSS/BSS functions. | +| | | | See `Domino User Guide`_ for | +| | | | details | +| | +----------------+----------------------------------+ +| | | copper | Copper develops OPNFV platform | +| | | | support for policy management, | +| | | | using open source projects such | +| | | | as OpenStack Congress, focused | +| | | | on helping ensure that virtual | +| | | | infrastructure and the apps that | +| | | | execute on it comply with the | +| | | | configuration policy intent of | +| | | | service providers, developers, | +| | | | and end users. See more detail | +| | | | in the `Copper User Guide`_. | +| | +----------------+----------------------------------+ +| | | multisites | Multisites | +| | | | See `Multisite User Guide`_ for | +| | | | details | ++-------------+---------------+----------------+----------------------------------+ +| VNF | vnf | cloudify_ims | Example of a real VNF deployment | +| | | | to show the NFV capabilities of | +| | | | the platform. The IP Multimedia | +| | | | Subsytem is a typical Telco test | +| | | | case, referenced by ETSI. | +| | | | It provides a fully functional | +| | | | VoIP System | +| | +----------------+----------------------------------+ +| | | opera_ims | vIMS deployment using openBaton | +| | +----------------+----------------------------------+ +| | | orchestra_ims | vIMS deployment using open-O | ++ +---------------+----------------+----------------------------------+ +| | | parser | Parser is an integration project | +| | | | which aims to provide | +| | | | placement/deployment templates | +| | | | translation for OPNFV platform, | +| | | | including TOSCA -> HOT, POLICY ->| +| | | | TOSCA and YANG -> TOSCA. | +| | | | See `Parser User Guide`_ for | +| | | | details | ++-------------+---------------+----------------+----------------------------------+ + + +As shown in the above table, Functest is structured into different 'domains', +'tiers' and 'test cases'. Each 'test case' usually represents an actual +'Test Suite' comprised -in turn- of several test cases internally. + +Test cases also have an implicit execution order. For example, if the early +'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke' +Tier testcases, there is little point to launch a full testcase execution round. + +In Danube, we merged smoke and sdn controller tiers in smoke tier. + +An overview of the Functest Structural Concept is depicted graphically below: + +.. figure:: ../images/concepts_mapping_final.png + :align: center + :alt: Functest Concepts Structure + +Some of the test cases are developed by Functest team members, whereas others +are integrated from upstream communities or other OPNFV projects. For example, +`Tempest `_ is the +OpenStack integration test suite and Functest is in charge of the selection, +integration and automation of those tests that fit suitably to OPNFV. + +The Tempest test suite is the default OpenStack smoke test suite but no new test +cases have been created in OPNFV Functest. + +The results produced by the tests run from CI are pushed and collected into a +NoSQL database. The goal is to populate the database with results from different +sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot +of a live Functest Dashboard is shown below: + +** TODO ** +.. figure:: ../images/FunctestDashboardDanube.png + :align: center + :alt: Functest Dashboard + + +Basic components (VIM, SDN controllers) are tested through their own suites. +Feature projects also provide their own test suites with different ways of +running their tests. + +The notion of domain has been introduced in the description of the test cases +stored in the Database. +This parameters as well as possible tags can be used for the Test case catalog. + +vIMS test case was integrated to demonstrate the capability to deploy a +relatively complex NFV scenario on top of the OPNFV infrastructure. + +Functest considers OPNFV as a black box. As of Danube release the OPNFV +offers a lot of potential combinations: + + * 3 controllers (OpenDaylight, ONOS, OpenContrail) + * 4 installers (Apex, Compass, Fuel, Joid) + +Most of the tests are runnable by any combination, but some tests might have +restrictions imposed by the utilized installers or due to the available +deployed features. The system uses the environment variables (INSTALLER_IP and +DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given +environment. + +A convenience Functest CLI utility is also available to simplify setting up the +Functest evironment, management of the OpenStack environment (e.g. resource +clean-up) and for executing tests. +The Functest CLI organised the testcase into logical Tiers, which contain in +turn one or more testcases. The CLI allows execution of a single specified +testcase, all test cases in a specified Tier, or the special case of execution +of **ALL** testcases. The Functest CLI is introduced in more detail in the +section `Executing the functest suites`_ of this document. + +.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html +.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html +.. _`Copper User Guide`: http://artifacts.opnfv.org/copper/colorado/docs/userguide/index.html +.. _`Doctor User Guide`: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html +.. _`Promise User Guide`: http://artifacts.opnfv.org/promise/colorado/docs/userguide/index.html +.. _`ONOSFW User Guide`: http://artifacts.opnfv.org/onosfw/colorado/userguide/index.html +.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html +.. _`Domino User Guide`: http://artifacts.opnfv.org/domino/docs/userguide-single/index.html +.. _`Parser User Guide`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html +.. _`Functest Dashboard`: http://testresults.opnfv.org/kibana_dashboards/ +.. _`SFC User Guide`: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html +.. _`Multisite User Guide`: http://artifacts.opnfv.org/multisite/docs/userguide/index.html diff --git a/docs/testing/user/userguide/runfunctest.rst b/docs/testing/user/userguide/runfunctest.rst new file mode 100644 index 000000000..e7ab84b26 --- /dev/null +++ b/docs/testing/user/userguide/runfunctest.rst @@ -0,0 +1,385 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +Executing the functest suites +============================= + +Manual testing +-------------- + +This section assumes the following: + * The Functest Docker container is running + * The docker prompt is shown + * The Functest environment is ready (Functest CLI command 'functest env prepare' + has been executed) + +If any of the above steps are missing please refer to the Functest Config Guide +as they are a prerequisite and all the commands explained in this section **must** be +performed **inside the container**. + +The Functest CLI offers two commands (functest tier ...) and (functest testcase ... ) +for the execution of Test Tiers or Test Cases:: + + root@22e436918db0:~/repos/functest/ci# functest tier --help + Usage: functest tier [OPTIONS] COMMAND [ARGS]... + + Options: + -h, --help Show this message and exit. + + Commands: + get-tests Prints the tests in a tier. + list Lists the available tiers. + run Executes all the tests within a tier. + show Shows information about a tier. + root@22e436918db0:~/repos/functest/ci# functest testcase --help + + Usage: functest testcase [OPTIONS] COMMAND [ARGS]... + + Options: + -h, --help Show this message and exit. + + Commands: + list Lists the available testcases. + run Executes a test case. + show Shows information about a test case. + +More details on the existing Tiers and Test Cases can be seen with the 'list' +command:: + + root@22e436918db0:~/repos/functest/ci# functest tier list + - 0. healthcheck: + ['healthcheck', 'connection_check', 'api_check',] + - 1. smoke: + ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity', 'snaps_smoke', 'odl'] + - 2. features: + ['doctor', 'security_scan'] + - 3. components: + ['tempest_full_parallel', 'rally_full'] + - 4. vnf: + ['cloudify_ims'] + + and + + root@22e436918db0:~/repos/functest/ci# functest testcase list + healthcheck + api_check + connection_check + vping_ssh + vping_userdata + snaps_smoke + tempest_smoke_serial + rally_sanity + odl + doctor + security_scan + tempest_full_parallel + rally_full + cloudify_ims + +More specific details on specific Tiers or Test Cases can be seen wih the +'show' command:: + + root@22e436918db0:~/repos/functest/ci# functest tier show smoke + +======================================================================+ + | Tier: smoke | + +======================================================================+ + | Order: 1 | + | CI Loop: (daily)|(weekly) | + | Description: | + | Set of basic Functional tests to validate the OpenStack | + | deployment. | + | Test cases: | + | - vping_ssh | + | - vping_userdata | + | - tempest_smoke_serial | + | - rally_sanity | + | | + +----------------------------------------------------------------------+ + + and + + root@22e436918db0:~/repos/functest/ci# functest testcase show tempest_smoke_serial + +======================================================================+ + | Testcase: tempest_smoke_serial | + +======================================================================+ + | Description: | + | This test case runs the smoke subset of the OpenStack Tempest | + | suite. The list of test cases is generated by Tempest | + | automatically and depends on the parameters of the OpenStack | + | deplopyment. | + | Dependencies: | + | - Installer: | + | - Scenario : | + | | + +----------------------------------------------------------------------+ + + +To execute a Test Tier or Test Case, the 'run' command is used:: + + root@22e436918db0:~/repos/functest/ci# functest tier run healthcheck + Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t healthcheck' + 2016-06-30 11:44:56,933 - run_tests - INFO - Sourcing the OpenStack RC file... + 2016-06-30 11:44:56,937 - run_tests - INFO - ############################################ + 2016-06-30 11:44:56,938 - run_tests - INFO - Running tier 'healthcheck' + 2016-06-30 11:44:56,938 - run_tests - INFO - ############################################ + 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================ + 2016-06-30 11:44:56,938 - run_tests - INFO - Running test case 'healthcheck'... + 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================ + 2016-06-30 11:44:56,953 - healtcheck - INFO - Testing Keystone API... + 2016-06-30 11:45:05,351 - healtcheck - INFO - ...Keystone OK! + 2016-06-30 11:45:05,354 - healtcheck - INFO - Testing Glance API... + 2016-06-30 11:45:29,746 - healtcheck - INFO - ... Glance OK! + 2016-06-30 11:45:29,749 - healtcheck - INFO - Testing Cinder API... + 2016-06-30 11:45:37,502 - healtcheck - INFO - ...Cinder OK! + 2016-06-30 11:45:37,505 - healtcheck - INFO - Testing Neutron API... + 2016-06-30 11:45:39,664 - healtcheck - INFO - External network found. ccd98ad6-d34a-4768-b03c-e28ecfcd51ca + 2016-06-30 11:45:39,667 - healtcheck - INFO - 1. Create Networks... + 2016-06-30 11:45:44,227 - healtcheck - INFO - 2. Create subnets... + 2016-06-30 11:45:46,805 - healtcheck - INFO - 4. Create Routers... + 2016-06-30 11:45:54,261 - healtcheck - INFO - ...Neutron OK! + 2016-06-30 11:45:54,264 - healtcheck - INFO - Testing Nova API... + 2016-06-30 11:47:12,272 - healtcheck - INFO - ...Nova OK! + 2016-06-30 11:47:12,274 - healtcheck - INFO - Checking if instances get an IP from DHCP... + : + : + 2016-06-30 11:48:17,832 - healtcheck - INFO - ...DHCP OK! + 2016-06-30 11:48:17,835 - healtcheck - INFO - Health check passed! + 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ + 2016-06-30 11:48:17,837 - clean_openstack - INFO - Cleaning OpenStack resources... + 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ + Version 1 is deprecated, use alternative version 2 instead. + WARNING:cinderclient.api_versions:Version 1 is deprecated, use alternative version 2 instead. + 2016-06-30 11:48:18,272 - clean_openstack - INFO - Removing Nova instances... + 2016-06-30 11:48:24,439 - clean_openstack - INFO - ------------------------------------------- + 2016-06-30 11:48:24,440 - clean_openstack - INFO - Removing Glance images... + 2016-06-30 11:48:35,853 - clean_openstack - INFO - ------------------------------------------- + 2016-06-30 11:48:35,854 - clean_openstack - INFO - Removing Cinder volumes... + 2016-06-30 11:48:37,344 - clean_openstack - INFO - ------------------------------------------- + 2016-06-30 11:48:37,344 - clean_openstack - INFO - Removing floating IPs... + 2016-06-30 11:48:37,467 - clean_openstack - INFO - ------------------------------------------- + 2016-06-30 11:48:37,467 - clean_openstack - INFO - Removing Neutron objects + 2016-06-30 11:48:53,633 - clean_openstack - INFO - ------------------------------------------- + 2016-06-30 11:48:53,633 - clean_openstack - INFO - Removing Security groups... + 2016-06-30 11:48:53,689 - clean_openstack - INFO - ------------------------------------------- + 2016-06-30 11:48:53,689 - clean_openstack - INFO - Removing Users... + 2016-06-30 11:48:54,444 - clean_openstack - INFO - ------------------------------------------- + 2016-06-30 11:48:54,444 - clean_openstack - INFO - Removing Tenants... + 2016-06-30 11:48:54,711 - clean_openstack - INFO - ------------------------------------------- + + and + + root@22e436918db0:~/repos/functest/ci# functest testcase run vping_ssh + Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t vping_ssh' + 2016-06-30 11:50:31,861 - run_tests - INFO - Sourcing the OpenStack RC file... + 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================ + 2016-06-30 11:50:31,865 - run_tests - INFO - Running test case 'vping_ssh'... + 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================ + 2016-06-30 11:50:32,977 - vping_ssh - INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.5-x86_64-disk.img'... + 2016-06-30 11:50:45,470 - vping_ssh - INFO - Creating neutron network vping-net... + 2016-06-30 11:50:47,645 - vping_ssh - INFO - Creating security group 'vPing-sg'... + 2016-06-30 11:50:48,843 - vping_ssh - INFO - Using existing Flavor 'm1.small'... + 2016-06-30 11:50:48,927 - vping_ssh - INFO - vPing Start Time:'2016-06-30 11:50:48' + 2016-06-30 11:50:48,927 - vping_ssh - INFO - Creating instance 'opnfv-vping-1'... + 2016-06-30 11:51:34,664 - vping_ssh - INFO - Instance 'opnfv-vping-1' is ACTIVE. + 2016-06-30 11:51:34,818 - vping_ssh - INFO - Adding 'opnfv-vping-1' to security group 'vPing-sg'... + 2016-06-30 11:51:35,209 - vping_ssh - INFO - Creating instance 'opnfv-vping-2'... + 2016-06-30 11:52:01,439 - vping_ssh - INFO - Instance 'opnfv-vping-2' is ACTIVE. + 2016-06-30 11:52:01,439 - vping_ssh - INFO - Adding 'opnfv-vping-2' to security group 'vPing-sg'... + 2016-06-30 11:52:01,754 - vping_ssh - INFO - Creating floating IP for VM 'opnfv-vping-2'... + 2016-06-30 11:52:01,969 - vping_ssh - INFO - Floating IP created: '10.17.94.140' + 2016-06-30 11:52:01,969 - vping_ssh - INFO - Associating floating ip: '10.17.94.140' to VM 'opnfv-vping-2' + 2016-06-30 11:52:02,792 - vping_ssh - INFO - Trying to establish SSH connection to 10.17.94.140... + 2016-06-30 11:52:19,915 - vping_ssh - INFO - Waiting for ping... + 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing detected! + 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing duration:'92.2' s. + 2016-06-30 11:52:21,109 - vping_ssh - INFO - vPing OK + 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ + 2016-06-30 11:52:21,153 - clean_openstack - INFO - Cleaning OpenStack resources... + 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ + Version 1 is deprecated, use alternative version 2 instead. + : + : + etc. + +To list the test cases which are part of a specific Test Tier, the 'get-tests' +command is used with 'functest tier':: + + root@22e436918db0:~/repos/functest/ci# functest tier get-tests healthcheck + Test cases in tier 'healthcheck': + ['healthcheck'] + + +Please note that for some scenarios some test cases might not be launched. +For example, the last example displayed only the 'odl' testcase for the given +environment. In this particular system the deployment does not support the 'onos' SDN +Controller Test Case; for example. + +**Important** If you use the command 'functest tier run ', then the +Functest CLI utility will call **all valid Test Cases**, which belong to the +specified Test Tier, as relevant to scenarios deployed to the SUT environment. +Thus, the Functest CLI utility calculates automatically which tests can be +executed and which cannot, given the environment variable **DEPLOY_SCENARIO**, +which is passed in to the Functest docker container. + +Currently, the Functest CLI command 'functest testcase run ', supports +two possibilities:: + + * Run a single Test Case, specified by a valid choice of + * Run ALL test Test Cases (for all Tiers) by specifying = 'all' + +Functest includes a cleaning mechanism in order to remove all the OpenStack +resources except those present before running any test. The script +*$REPOS_DIR/functest/functest/utils/generate_defaults.py* is called once when setting up +the Functest environment (i.e. CLI command 'functest env prepare') to snapshot +all the OpenStack resources (images, networks, volumes, security groups, tenants, +users) so that an eventual cleanup does not remove any of these defaults. + +The script **clean_openstack.py** which is located in +*$REPOS_DIR/functest/functest/utils/* is normally called after a test execution. It is +in charge of cleaning the OpenStack resources that are not specified in the +defaults file generated previously which is stored in +*/home/opnfv/functest/conf/os_defaults.yaml* in the Functest docker container. + +It is important to mention that if there are new OpenStack resources created +manually after preparing the Functest environment, they will be removed, unless +you use the special method of invoking the test case with specific suppression +of clean up. (See the `Troubleshooting`_ section). + +The reason to include this cleanup meachanism in Functest is because some +test suites such as Tempest or Rally create a lot of resources (users, +tenants, networks, volumes etc.) that are not always properly cleaned, so this +function has been set to keep the system as clean as it was before a +full Functest execution. + +Although the Functest CLI provides an easy way to run any test, it is possible to +do a direct call to the desired test script. For example: + + python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/vPing/vPing_ssh.py -d + + +Automated testing +----------------- + +As mentioned previously, the Functest Docker container preparation as well as +invocation of Test Cases can be called within the container from the Jenkins CI +system. There are 3 jobs that automate the whole process. The first job runs all +the tests referenced in the daily loop (i.e. that must been run daily), the second +job runs the tests referenced in the weekly loop (usually long duration tests run +once a week maximum) and the third job allows testing test suite by test suite specifying +the test suite name. The user may also use either of these Jenkins jobs to execute +the desired test suites. + +One of the most challenging task in the Danube release consists +in dealing with lots of scenarios and installers. Thus, when the tests are +automatically started from CI, a basic algorithm has been created in order to +detect whether a given test is runnable or not on the given scenario. +Some Functest test suites cannot be systematically run (e.g. ODL suite can not +be run on an ONOS scenario). The daily/weekly notion has been introduces in +Colorado in order to save CI time and avoid running systematically +long duration tests. It was not used in Colorado due to CI resource shortage. +The mechanism remains however as part of the CI evolution. + +CI provides some useful information passed to the container as environment +variables: + + * Installer (apex|compass|daisy|fuel|joid), stored in INSTALLER_TYPE + * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP + * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with + + * controller = (odl|onos|ocl|nosdn) + * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|multisites) + * mode = (ha|noha) + +The constraints per test case are defined in the Functest configuration file +*/home/opnfv/repos/functest/functest/ci/testcases.yaml*:: + + tiers: + - + name: healthcheck + order: 0 + ci_loop: '(daily)|(weekly)' + description : >- + First tier to be executed to verify the basic + operations in the VIM. + testcases: + - + name: healthcheck + criteria: 'status == "PASS"' + blocking: true + description: >- + This test case verifies the basic OpenStack services like + Keystone, Glance, Cinder, Neutron and Nova. + + dependencies: + installer: '' + scenario: '' + + - + name: smoke + order: 1 + ci_loop: '(daily)|(weekly)' + description : >- + Set of basic Functional tests to validate the OpenStack deployment. + testcases: + - + name: vping_ssh + criteria: 'status == "PASS"' + blocking: true + description: >- + This test case verifies: 1) SSH to an instance using floating + IPs over the public network. 2) Connectivity between 2 instances + over a private network. + dependencies: + installer: '' + scenario: '^((?!bgpvpn|odl_l3).)*$' + run: + module: 'functest.opnfv_tests.openstack.vping.vping_ssh' + class: 'VPingSSH' + .... + +We may distinguish 2 levels in the test case description: + * Tier level + * Test case level + +At the tier level, we define the following parameters: + + * ci_loop: indicate if in automated mode, the test case must be run in daily and/or weekly jobs + * description: a high level view of the test case + +For a given test case we defined: + * the name of the test case + * the criteria (experimental): a criteria used to declare the test case as PASS or FAIL + * blocking: if set to true, if the test is failed, the execution of the following tests is canceled + * the description of the test case + * the dependencies: a combination of 2 regex on the scenario and the installer name + * run: In Danube we introduced the notion of abstract class in order to harmonize the way to run internal, feature or vnf tests + +For further details on abstraction classes, see developper guide. + +Additional parameters have been added in the desription in the Database. +The target is to use the configuration stored in the Database and consider the +local file as backup if the Database is not reachable. +The additional fields related to a test case are: + * trust: we introduced this notion to put in place a mechanism of scenario promotion. + * Version: it indicates since which version you can run this test + * domains: the main domain covered by the test suite + * tags: a list of tags related to the test suite + +The order of execution is the one defined in the file if all test cases are selected. + +In CI daily job the tests are executed in the following order: + + 1) healthcheck (blocking) + 2) smoke: both vPings are blocking + 3) Feature project tests cases + +In CI weekly job we add 2 tiers: + + 4) VNFs (vIMS) + 5) Components (Rally and Tempest long duration suites) + +As explained before, at the end of an automated execution, the OpenStack resources +might be eventually removed. +Please note that a system snapshot is taken before any test case execution. + +This testcase.yaml file is used for CI, for the CLI and for the automatic reporting. diff --git a/docs/testing/user/userguide/troubleshooting.rst b/docs/testing/user/userguide/troubleshooting.rst new file mode 100644 index 000000000..845501916 --- /dev/null +++ b/docs/testing/user/userguide/troubleshooting.rst @@ -0,0 +1,387 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +Troubleshooting +=============== + +This section gives some guidelines about how to troubleshoot the test cases +owned by Functest. + +**IMPORTANT**: As in the previous section, the steps defined below must be +executed inside the Functest Docker container and after sourcing the OpenStack +credentials:: + + . $creds + +or:: + + source /home/opnfv/functest/conf/openstack.creds + +VIM +--- + +This section covers the test cases related to the VIM (healthcheck, vping_ssh, +vping_userdata, tempest_smoke_serial, tempest_full_parallel, rally_sanity, +rally_full). + +vPing common +^^^^^^^^^^^^ +For both vPing test cases (**vPing_ssh**, and **vPing_userdata**), the first steps are +similar: + + * Create Glance image + * Create Network + * Create Security Group + * Create Instances + +After these actions, the test cases differ and will be explained in their +respective section. + +These test cases can be run inside the container, using new Functest CLI as follows:: + + $ functest testcase run vping_ssh + $ functest testcase run vping_userdata + +The Functest CLI is designed to route a call to the corresponding internal +python scripts, located in paths: +*$REPOS_DIR/functest/functest/opnfv_tests/vPing/CI/libraries/vPing_ssh.py* and +*$REPOS_DIR/functest/functest/opnfv_tests/vPing/CI/libraries/vPing_userdata.py* + +Notes: + + #. There is one difference, between the Functest CLI based test case + execution compared to the earlier used Bash shell script, which is + relevant to point out in troubleshooting scenarios: + + The Functest CLI does **not yet** support the option to suppress + clean-up of the generated OpenStack resources, following the execution + of a test case. + + Explanation: After finishing the test execution, the corresponding + script will remove, by default, all created resources in OpenStack + (image, instances, network and security group). When troubleshooting, + it is advisable sometimes to keep those resources in case the test + fails and a manual testing is needed. + + It is actually still possible to invoke test execution, with suppression + of OpenStack resource cleanup, however this requires invocation of a + **specific Python script:** '/home/opnfv/repos/functest/ci/run_test.py'. + The `OPNFV Functest Developer Guide`_ provides guidance on the use of that + Python script in such troubleshooting cases. + +Some of the common errors that can appear in this test case are:: + + vPing_ssh- ERROR - There has been a problem when creating the neutron network.... + +This means that there has been some problems with Neutron, even before creating the +instances. Try to create manually a Neutron network and a Subnet to see if that works. +The debug messages will also help to see when it failed (subnet and router creation). +Example of Neutron commands (using 10.6.0.0/24 range for example):: + + neutron net-create net-test + neutron subnet-create --name subnet-test --allocation-pool start=10.6.0.2,end=10.6.0.100 \ + --gateway 10.6.0.254 net-test 10.6.0.0/24 + neutron router-create test_router + neutron router-interface-add test_subnet + neutron router-gateway-set + +Another related error can occur while creating the Security Groups for the instances:: + + vPing_ssh- ERROR - Failed to create the security group... + +In this case, proceed to create it manually. These are some hints:: + + neutron security-group-create sg-test + neutron security-group-rule-create sg-test --direction ingress --protocol icmp \ + --remote-ip-prefix 0.0.0.0/0 + neutron security-group-rule-create sg-test --direction ingress --ethertype IPv4 \ + --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0 + neutron security-group-rule-create sg-test --direction egress --ethertype IPv4 \ + --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0 + +The next step is to create the instances. The image used is located in +*/home/opnfv/functest/data/cirros-0.3.5-x86_64-disk.img* and a Glance image is created +with the name **functest-vping**. If booting the instances fails (i.e. the status +is not **ACTIVE**), you can check why it failed by doing:: + + nova list + nova show + +It might show some messages about the booting failure. To try that manually:: + + nova boot --flavor m1.small --image functest-vping --nic net-id= nova-test + +This will spawn a VM using the network created previously manually. +In all the OPNFV tested scenarios from CI, it never has been a problem with the +previous actions. Further possible problems are explained in the following sections. + + +vPing_SSH +^^^^^^^^^ +This test case creates a floating IP on the external network and assigns it to +the second instance **opnfv-vping-2**. The purpose of this is to establish +a SSH connection to that instance and SCP a script that will ping the first +instance. This script is located in the repository under +*$REPOS_DIR/functest/functest/opnfv_tests/OpenStack/vPing/ping.sh* and takes an IP as +a parameter. When the SCP is completed, the test will do an SSH call to that script +inside the second instance. Some problems can happen here:: + + vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting + +If this is displayed, stop the test or wait for it to finish, if you have used +the special method of test invocation with specific supression of OpenStack +resource clean-up, as explained earler. It means that the Container can not +reach the Public/External IP assigned to the instance **opnfv-vping-2**. There +are many possible reasons, and they really depend on the chosen scenario. For +most of the ODL-L3 and ONOS scenarios this has been noticed and it is a known +limitation. + +First, make sure that the instance **opnfv-vping-2** succeeded to get an IP +from the DHCP agent. It can be checked by doing:: + + nova console-log opnfv-vping-2 + +If the message *Sending discover* and *No lease, failing* is shown, it probably +means that the Neutron dhcp-agent failed to assign an IP or even that it was not +responding. At this point it does not make sense to try to ping the floating IP. + +If the instance got an IP properly, try to ping manually the VM from the container:: + + nova list + + ping + +If the ping does not return anything, try to ping from the Host where the Docker +container is running. If that solves the problem, check the iptable rules because +there might be some rules rejecting ICMP or TCP traffic coming/going from/to the +container. + +At this point, if the ping does not work either, try to reproduce the test +manually with the steps described above in the vPing common section with the +addition:: + + neutron floatingip-create + nova floating-ip-associate nova-test + + +Further troubleshooting is out of scope of this document, as it might be due to +problems with the SDN controller. Contact the installer team members or send an +email to the corresponding OPNFV mailing list for more information. + + + +vPing_userdata +^^^^^^^^^^^^^^ +This test case does not create any floating IP neither establishes an SSH +connection. Instead, it uses nova-metadata service when creating an instance +to pass the same script as before (ping.sh) but as 1-line text. This script +will be executed automatically when the second instance **opnfv-vping-2** is booted. + +The only known problem here for this test to fail is mainly the lack of support +of cloud-init (nova-metadata service). Check the console of the instance:: + + nova console-log opnfv-vping-2 + +If this text or similar is shown:: + + checking http://169.254.169.254/2009-04-04/instance-id + failed 1/20: up 1.13. request failed + failed 2/20: up 13.18. request failed + failed 3/20: up 25.20. request failed + failed 4/20: up 37.23. request failed + failed 5/20: up 49.25. request failed + failed 6/20: up 61.27. request failed + failed 7/20: up 73.29. request failed + failed 8/20: up 85.32. request failed + failed 9/20: up 97.34. request failed + failed 10/20: up 109.36. request failed + failed 11/20: up 121.38. request failed + failed 12/20: up 133.40. request failed + failed 13/20: up 145.43. request failed + failed 14/20: up 157.45. request failed + failed 15/20: up 169.48. request failed + failed 16/20: up 181.50. request failed + failed 17/20: up 193.52. request failed + failed 18/20: up 205.54. request failed + failed 19/20: up 217.56. request failed + failed 20/20: up 229.58. request failed + failed to read iid from metadata. tried 20 + +it means that the instance failed to read from the metadata service. Contact +the Functest or installer teams for more information. + +NOTE: Cloud-init in not supported on scenarios dealing with ONOS and the tests +have been excluded from CI in those scenarios. + + +Tempest +^^^^^^^ + +In the upstream OpenStack CI all the Tempest test cases are supposed to pass. +If some test cases fail in an OPNFV deployment, the reason is very probably one +of the following + ++-----------------------------+-----------------------------------------------------+ +| Error | Details | ++=============================+=====================================================+ +| Resources required for test | Such resources could be e.g. an external network | +| case execution are missing | and access to the management subnet (adminURL) from | +| | the Functest docker container. | ++-----------------------------+-----------------------------------------------------+ +| OpenStack components or | Check running services in the controller and compute| +| services are missing or not | nodes (e.g. with "systemctl" or "service" commands).| +| configured properly | Configuration parameters can be verified from the | +| | related .conf files located under '/etc/'| +| | directories. | ++-----------------------------+-----------------------------------------------------+ +| Some resources required for | The tempest.conf file, automatically generated by | +| execution test cases are | Rally in Functest, does not contain all the needed | +| missing | parameters or some parameters are not set properly. | +| | The tempest.conf file is located in directory | +| | '/home/opnfv/.rally/tempest/for-deployment-' | +| | in the Functest Docker container. Use the "rally | +| | deployment list" command in order to check the UUID | +| | the UUID of the current deployment. | ++-----------------------------+-----------------------------------------------------+ + + +When some Tempest test case fails, captured traceback and possibly also the +related REST API requests/responses are output to the console. More detailed debug +information can be found from tempest.log file stored into related Rally deployment +folder. + + +Rally +^^^^^ + +The same error causes which were mentioned above for Tempest test cases, may also +lead to errors in Rally as well. + +It is possible to run only one Rally scenario, instead of the whole suite. +To do that, call the alternative python script as follows:: + + python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/rally/run_rally-cert.py -h + usage: run_rally-cert.py [-h] [-d] [-r] [-s] [-v] [-n] test_name + + positional arguments: + test_name Module name to be tested. Possible values are : [ + authenticate | glance | cinder | heat | keystone | neutron | + nova | quotas | requests | vm | all ] The 'all' value + performs all possible test scenarios + + optional arguments: + -h, --help show this help message and exit + -d, --debug Debug mode + -r, --report Create json result file + -s, --smoke Smoke test mode + -v, --verbose Print verbose info about the progress + -n, --noclean Don't clean the created resources for this test. + +For example, to run the Glance scenario with debug information:: + + python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/rally/run_rally-cert.py -d glance + +Possible scenarios are: + * authenticate + * glance + * cinder + * heat + * keystone + * neutron + * nova + * quotas + * requests + * vm + +To know more about what those scenarios are doing, they are defined in directory: +*$REPOS_DIR/functest/functest/opnfv_tests/OpenStack/rally/scenario* +For more info about Rally scenario definition please refer to the Rally official +documentation. `[3]`_ + +If the flag *all* is specified, it will run all the scenarios one by one. Please +note that this might take some time (~1,5hr), taking around 1 hour alone to +complete the Nova scenario. + +To check any possible problems with Rally, the logs are stored under +*/home/opnfv/functest/results/rally/* in the Functest Docker container. + + +Controllers +----------- + +Opendaylight +^^^^^^^^^^^^ + +If the Basic Restconf test suite fails, check that the ODL controller is +reachable and its Restconf module has been installed. + +If the Neutron Reachability test fails, verify that the modules +implementing Neutron requirements have been properly installed. + +If any of the other test cases fails, check that Neutron and ODL have +been correctly configured to work together. Check Neutron configuration +files, accounts, IP addresses etc.). + + +ONOS +^^^^ + +Please refer to the ONOS documentation. `ONOSFW User Guide`_ . + + +Features +-------- + +Please refer to the dedicated feature user guides for details. + + +security_scan +^^^^^^^^^^^^^ + +See OpenSCAP web site: https://www.open-scap.org/ + + + +NFV +--- + +cloudify_ims +^^^^^^^^^^^^ +vIMS deployment may fail for several reasons, the most frequent ones are +described in the following table: + ++-----------------------------------+------------------------------------+ +| Error | Comments | ++===================================+====================================+ +| Keystone admin API not reachable | Impossible to create vIMS user and | +| | tenant | ++-----------------------------------+------------------------------------+ +| Impossible to retrieve admin role | Impossible to create vIMS user and | +| id | tenant | ++-----------------------------------+------------------------------------+ +| Error when uploading image from | impossible to deploy VNF | +| OpenStack to glance | | ++-----------------------------------+------------------------------------+ +| Cinder quota cannot be updated | Default quotas not sufficient, they| +| | are adapted in the script | ++-----------------------------------+------------------------------------+ +| Impossible to create a volume | VNF cannot be deployed | ++-----------------------------------+------------------------------------+ +| SSH connection issue between the | if vPing test fails, vIMS test will| +| Test Docker container and the VM | fail... | ++-----------------------------------+------------------------------------+ +| No Internet access from the VM | the VMs of the VNF must have an | +| | external access to Internet | ++-----------------------------------+------------------------------------+ +| No access to OpenStack API from | Orchestrator can be installed but | +| the VM | the vIMS VNF installation fails | ++-----------------------------------+------------------------------------+ + + +parser +^^^^^^ + +For now log info is the only way to do trouble shooting + + +.. _`OPNFV Functest Developer Guide`: http://artifacts.opnfv.org/functest/docs/devguide/# diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst deleted file mode 100644 index 9436de2b9..000000000 --- a/docs/userguide/index.rst +++ /dev/null @@ -1,548 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - -========================= -OPNFV FUNCTEST user guide -========================= - -.. toctree:: - :maxdepth: 2 - -Version history -=============== -+------------+----------+------------------+----------------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+----------------------------------+ -| 2016-08-17 | 1.0.0 | Juha Haapavirta | Colorado release | -| | | Column Gaynor | | -+------------+----------+------------------+----------------------------------+ -| 2017-01-23 | 1.0.1 | Morgan Richomme | Adaptations for Danube | -| | | | | -| | | | | -+------------+----------+------------------+----------------------------------+ - - -Introduction -============ - -The goal of this document is to describe the OPNFV Functest test cases and to -provide a procedure to execute them. In the OPNFV Danube system release, -a Functest CLI utility is introduced for easier execution of test procedures. - -**IMPORTANT**: It is assumed here that the Functest Docker container is already -properly deployed and that all instructions described in this guide are to be -performed from *inside* the deployed Functest Docker container. - -.. include:: ./introduction.rst - -The different test cases are described in the remaining sections of this document. - -VIM (Virtualized Infrastructure Manager) ----------------------------------------- - -Healthcheck -^^^^^^^^^^^ -In Colorado release a new Tier 'healthcheck' with one testcase 'healthcheck' -was introduced. The healthcheck testcase verifies that some basic IP connectivity -and essential operations of OpenStack functionality over the command line are -working correctly. - -In particular, the following verifications are performed: - - * DHCP agent functionality for IP address allocation - * Openstack Authentication management functionality via the Keystone API - * OpenStack Image management functionality via the Glance API - * OpenStack Block Storage management functionality via the Cinder API - * OpenStack Networking management functionality via the Neutron API - * Openstack Compute management functionality via the NOVA API - -Self-obviously, successful completion of the 'healthcheck' testcase is a -necessary pre-requisite for the execution of all other test Tiers. - - -vPing_ssh -^^^^^^^^^ - -Given the script **ping.sh**:: - - #!/bin/sh - while true; do - ping -c 1 $1 2>&1 >/dev/null - RES=$? - if [ "Z$RES" = "Z0" ] ; then - echo 'vPing OK' - break - else - echo 'vPing KO' - fi - sleep 1 - done - - -The goal of this test is to establish an SSH connection using a floating IP -on the Public/External network and verify that 2 instances can talk over a Private -Tenant network:: - - vPing_ssh test case - +-------------+ +-------------+ - | | | | - | | Boot VM1 with IP1 | | - | +------------------->| | - | Tester | | System | - | | Boot VM2 | Under | - | +------------------->| Test | - | | | | - | | Create floating IP | | - | +------------------->| | - | | | | - | | Assign floating IP | | - | | to VM2 | | - | +------------------->| | - | | | | - | | Establish SSH | | - | | connection to VM2 | | - | | through floating IP| | - | +------------------->| | - | | | | - | | SCP ping.sh to VM2 | | - | +------------------->| | - | | | | - | | VM2 executes | | - | | ping.sh to VM1 | | - | +------------------->| | - | | | | - | | If ping: | | - | | exit OK | | - | | else (timeout): | | - | | exit Failed | | - | | | | - +-------------+ +-------------+ - -This test can be considered as an "Hello World" example. -It is the first basic use case which **must** work on any deployment. - -vPing_userdata -^^^^^^^^^^^^^^ - -This test case is similar to vPing_ssh but without the use of Floating IPs -and the Public/External network to transfer the ping script. -Instead, it uses Nova metadata service to pass it to the instance at booting time. -As vPing_ssh, it checks that 2 instances can talk to -each other on a Private Tenant network:: - - vPing_userdata test case - +-------------+ +-------------+ - | | | | - | | Boot VM1 with IP1 | | - | +------------------->| | - | | | | - | | Boot VM2 with | | - | | ping.sh as userdata| | - | | with IP1 as $1. | | - | +------------------->| | - | Tester | | System | - | | VM2 exeutes ping.sh| Under | - | | (ping IP1) | Test | - | +------------------->| | - | | | | - | | Monitor nova | | - | | console-log VM 2 | | - | | If ping: | | - | | exit OK | | - | | else (timeout) | | - | | exit Failed | | - | | | | - +-------------+ +-------------+ - -When the second VM boots it will execute the script passed as userdata -automatically. The ping will be detected by periodically capturing the output -in the console-log of the second VM. - - -Tempest -^^^^^^^ - -Tempest `[2]`_ is the reference OpenStack Integration test suite. -It is a set of integration tests to be run against a live OpenStack cluster. -Tempest has suites of tests for: - - * OpenStack API validation - * Scenarios - * Other specific tests useful in validating an OpenStack deployment - -Functest uses Rally `[3]`_ to run the Tempest suite. -Rally generates automatically the Tempest configuration file **tempest.conf**. -Before running the actual test cases, -Functest creates the needed resources (user, tenant) and -updates the appropriate parameters into the configuration file. - -When the Tempest suite is executed, each test duration is measured and the full -console output is stored to a *log* file for further analysis. - -The Tempest testcases are distributed accross two -Tiers: - - * Smoke Tier - Test Case 'tempest_smoke_serial' - * Components Tier - Test case 'tempest_full_parallel' - -NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke -tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel' -executes all defined Tempest tests using several concurrent threads -(i.e. parallel mode). The number of threads activated corresponds to the number -of available logical CPUs. - -The goal of the Tempest test suite is to check the basic functionalities of the -different OpenStack components on an OPNFV fresh installation, using the -corresponding REST API interfaces. - - -Rally bench test suites -^^^^^^^^^^^^^^^^^^^^^^^ - -Rally `[3]`_ is a benchmarking tool that answers the question: - -*How does OpenStack work at scale?* - -The goal of this test suite is to benchmark all the different OpenStack modules and -get significant figures that could help to define Telco Cloud KPIs. - -The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios: - - * authenticate - * cinder - * glance - * heat - * keystone - * neutron - * nova - * quotas - * requests - -A basic SLA (stop test on errors) has been implemented. - -The Rally testcases are distributed accross two Tiers: - - * Smoke Tier - Test Case 'rally_sanity' - * Components Tier - Test case 'rally_full' - -NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test -cases. Test case 'rally_full' executes the full defined set of Rally tests. - -SNAPS ------ - -SNAPS stands for "SNA/NFV Application development Platform and Stack". -This project seeks to develop baseline OpenStack NFV installations. It has been -developed by Steven Pisarski and provided an object oriented library to perform -functional and performance tests. It has been declined in several test suites in -Functest, 2 are part of healthcheck tier, one belongs to smoke tier. - -connection check -^^^^^^^^^^^^^^^^ -Connection_check consists in 9 test cases (test duration < 5s) checking the -connectivity with Glance, Keystone, Neutron, Nova and the external network. - -api_check -^^^^^^^^^ -This test case verifies the retrieval of OpenStack clients: Keystone, Glance, -Neutron and Nova and may perform some simple queries. When the config value of -snaps.use_keystone is True, functest must have access to the cloud's private -network. -This suite consists in 49 tests (test duration< 2 minutes) - -snaps_smoke -^^^^^^^^^^^ -This test case contains tests that setup and destroy environments with VMs with -and without Floating IPs with a newly created user and project. Set the config -value snaps.use_floating_ips (True|False) to toggle this functionality. When -the config value of snaps.use_keystone is True, functest must have access -the cloud's private network. -This suite consists in 38 tests (test duration < 10 minutes) - -More information on SNAPS can be found in  `[13]`_ - - -SDN Controllers ---------------- - -There are currently 2 available controllers: - - * OpenDaylight (ODL) - * ONOS - -OpenDaylight -^^^^^^^^^^^^ - -The OpenDaylight (ODL) test suite consists of a set of basic tests inherited -from the ODL project using the Robot `[11]`_ framework. -The suite verifies creation and deletion of networks, subnets and ports with -OpenDaylight and Neutron. - -The list of tests can be described as follows: - - * Basic Restconf test cases - * Connect to Restconf URL - * Check the HTTP code status - - * Neutron Reachability test cases - * Get the complete list of neutron resources (networks, subnets, ports) - - * Neutron Network test cases - * Check OpenStack networks - * Check OpenDaylight networks - * Create a new network via OpenStack and check the HTTP status code returned by Neutron - * Check that the network has also been successfully created in OpenDaylight - - * Neutron Subnet test cases - * Check OpenStack subnets - * Check OpenDaylight subnets - * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron - * Check that the subnet has also been successfully created in OpenDaylight - - * Neutron Port test cases - * Check OpenStack Neutron for known ports - * Check OpenDaylight ports - * Create a new port via OpenStack and check the HTTP status code returned by Neutron - * Check that the new port has also been successfully created in OpenDaylight - - * Delete operations - * Delete the port previously created via OpenStack - * Check that the port has been also succesfully deleted in OpenDaylight - * Delete previously subnet created via OpenStack - * Check that the subnet has also been successfully deleted in OpenDaylight - * Delete the network created via OpenStack - * Check that the network has also been succesfully deleted in OpenDaylight - -Note: the checks in OpenDaylight are based on the returned HTTP status -code returned by OpenDaylight. - - -ONOS -^^^^ - -TestON Framework is used to test the ONOS SDN controller functions. -The test cases deal with L2 and L3 functions. -The ONOS test suite can be run on any ONOS compliant scenario. - -The test cases are described as follows: - - * onosfunctest: The main executable file contains the initialization of - the docker environment and functions called by FUNCvirNetNB and - FUNCvirNetNBL3 - - * FUNCvirNetNB - - * Create Network: Post Network data and check it in ONOS - * Update Network: Update the Network and compare it in ONOS - * Delete Network: Delete the Network and check if it's NULL in ONOS or - not - * Create Subnet: Post Subnet data and check it in ONOS - * Update Subnet: Update the Subnet and compare it in ONOS - * Delete Subnet: Delete the Subnet and check if it's NULL in ONOS or not - * Create Port: Post Port data and check it in ONOS - * Update Port: Update the Port and compare it in ONOS - * Delete Port: Delete the Port and check if it's NULL in ONOS or not - - * FUNCvirNetNBL3 - - * Create Router: Post data for create Router and check it in ONOS - * Update Router: Update the Router and compare it in ONOS - * Delete Router: Delete the Router data and check it in ONOS - * Create RouterInterface: Post Router Interface data to an existing Router - and check it in ONOS - * Delete RouterInterface: Delete the RouterInterface and check the Router - * Create FloatingIp: Post data for create FloatingIp and check it in ONOS - * Update FloatingIp: Update the FloatingIp and compare it in ONOS - * Delete FloatingIp: Delete the FloatingIp and check that it is 'NULL' in - ONOS - * Create External Gateway: Post data to create an External Gateway for an - existing Router and check it in ONOS - * Update External Gateway: Update the External Gateway and compare the change - * Delete External Gateway: Delete the External Gateway and check that it is - 'NULL' in ONOS - - -Features --------- - -Please refer to the dedicated feature user guides for details: - - * bgpvpn: http://artifacts.opnfv.org/sdnvpn/danube/docs/userguide/index.html - * copper: http://artifacts.opnfv.org/copper/danube/docs/userguide/index.html - * doctor: http://artifacts.opnfv.org/doctor/danube/userguide/index.html - * domino: http://artifacts.opnfv.org/domino/docs/userguide-single/index.html - * multisites: http://artifacts.opnfv.org/multisite/docs/userguide/index.html - * onos-sfc: http://artifacts.opnfv.org/onosfw/danube/userguide/index.html - * odl-sfc: http://artifacts.opnfv.org/sfc/danube/userguide/index.html - * promise: http://artifacts.opnfv.org/danube/colorado/docs/userguide/index.html - * security_scan: http://artifacts.opnfv.org/security_scan/colorado/docs/userguide/index.html - * TODO - -security_scan -^^^^^^^^^^^^^ - -Security Scanning, is a project to insure security compliance and vulnerability -checks, as part of an automated CI / CD platform delivery process. - -The project makes use of the existing SCAP format `[6]`_ to perform deep -scanning of NFVI nodes, to insure they are hardened and free of known CVE -reported vulnerabilities. - -The SCAP content itself, is then consumed and run using an upstream opensource tool -known as OpenSCAP `[7]`_. - -The OPNFV Security Group have developed the code that will called by the OPNFV Jenkins -build platform, to perform a complete scan. Resulting reports are then copied to the -OPNFV functest dashboard. - -The current work flow is as follows: - - * Jenkins Build Initiated - * security_scan.py script is called, and a config file is passed to the script as - an argument. - * The IP addresses of each NFVi node (compute / control) are gathered - * A scan profile is matched to the node type. - * The OpenSCAP application is remotely installed onto each target node gathered - on step 3, using upstream packaging (rpm and .deb). - * A scan is made against each node gathered within step 3. - * HTML Reports are downloaded for rendering on a dashboard. - * If the config file value 'clean' is set to 'True' then the application installed in - step 5 is removed, and all reports created at step 6 are deleted. - -Security scan is supported by Apex, TODO.... - - - -VNF ---- - - -vIMS -^^^^ -The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an -architectural framework for delivering IP multimedia services. - -vIMS has been integrated in Functest to demonstrate the capability to deploy a -relatively complex NFV scenario on the OPNFV platform. The deployment of a complete -functional VNF allows the test of most of the essential functions needed for a -NFV platform. - -The goal of this test suite consists of: - - * deploy a VNF orchestrator (Cloudify) - * deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF from this - orchestrator based on a TOSCA blueprint defined in `[5]`_ - * run suite of signaling tests on top of this VNF - -The Clearwater architecture is described as follows: - -.. figure:: ../images/clearwater-architecture.png - :align: center - :alt: vIMS architecture - - -parser -^^^^^^ - -See parser user guide for details: `[12]`_ - - -.. include:: ./runfunctest.rst - - -Test results -============ - -Manual testing --------------- - -In manual mode test results are displayed in the console and result files -are put in /home/opnfv/functest/results. - -Automated testing --------------- - -In automated mode, test results are displayed in jenkins logs, a summary is provided -at the end of the job and can be described as follow:: - - +==================================================================================================================================================+ - | FUNCTEST REPORT | - +==================================================================================================================================================+ - | | - | Deployment description: | - | INSTALLER: fuel | - | SCENARIO: os-odl_l2-nofeature-ha | - | BUILD TAG: jenkins-functest-fuel-baremetal-daily-master-324 | - | CI LOOP: daily | - | | - +=========================+===============+============+===============+===========================================================================+ - | TEST CASE | TIER | DURATION | RESULT | URL | - +=========================+===============+============+===============+===========================================================================+ - | healthcheck | healthcheck | 03:07 | PASS | | - +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ - | vping_ssh | smoke | 00:56 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac13d79377c54b278bd4c1 | - +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ - | vping_userdata | smoke | 00:41 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac14019377c54b278bd4c2 | - +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ - | tempest_smoke_serial | smoke | 16:05 | FAIL | http://testresults.opnfv.org/test/api/v1/results/57ac17ca9377c54b278bd4c3 | - +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ - | rally_sanity | smoke | 12:19 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1aad9377c54b278bd4cd | - +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ - | odl | sdn_suites | 00:24 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1ad09377c54b278bd4ce | - +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ - | promise | features | 00:41 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1ae59377c54b278bd4cf | - +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ - -Results are automatically pushed to the test results database, some additional -result files are pushed to OPNFV artifact web sites. - -Based on the results stored in the result database, a `Functest reporting`_ -portal is also automatically updated. This portal provides information on: - - * The overall status per scenario and per installer - * Tempest: Tempest test case including reported errors per scenario and installer - * vIMS: vIMS details per scenario and installer - -.. figure:: ../images/functest-reporting-status.png - :align: center - :alt: Functest reporting portal Fuel status page - - -Test Dashboard -============== - -Based on results collected in CI, a test dashboard is dynamically generated. - - -.. include:: ./troubleshooting.rst - - -References -========== - -.. _`[1]`: http://artifacts.opnfv.org/functest/colorado/docs/configguide/# -.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html -.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html -.. _`[4]`: http://events.linuxfoundation.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf -.. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml -.. _`[6]`: https://scap.nist.gov/ -.. _`[7]`: https://github.com/OpenSCAP/openscap -.. _`[9]`: https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/libraries/os_defaults.yaml -.. _`[11]`: http://robotframework.org/ -.. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html -.. _`[13]`: TODO URL doc SNAPS - -OPNFV main site: opnfvmain_. - -OPNFV functional test page: opnfvfunctest_. - -IRC support chan: #opnfv-testperf - -.. _opnfvmain: http://www.opnfv.org -.. _opnfvfunctest: https://wiki.opnfv.org/opnfv_functional_testing -.. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html -.. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html -.. _`config_test.py` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.py -.. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml -.. _`Functest reporting`: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html diff --git a/docs/userguide/introduction.rst b/docs/userguide/introduction.rst deleted file mode 100644 index 4dfe79375..000000000 --- a/docs/userguide/introduction.rst +++ /dev/null @@ -1,255 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - -Overview of the Functest suites -=============================== - -Functest is the OPNFV project primarily targeting function testing. -In the Continuous Integration pipeline, it is launched after an OPNFV fresh -installation to validate and verify the basic functions of the -infrastructure. - -The current list of test suites can be distributed over 5 main domains: VIM -(Virtualised Infrastructure Manager), Controllers (i.e. SDN Controllers), -Features, VNF (Virtual Network Functions) and MANO stacks. - -Functest test suites are also distributed in the OPNFV testing categories: -healthcheck, smoke, features, components, performance, VNF, Stress tests. - -All the Healthcheck and smoke tests of a given scenario must be succesful to -validate the scenario for the release. - -+-------------+---------------+----------------+----------------------------------+ -| Domain | Tier | Test case | Comments | -+=============+===============+================+==================================+ -| VIM | healthcheck | healthcheck | Verify basic operation in VIM | -| | +----------------+----------------------------------+ -| | | connection | Check OpenStack connectivity | -| | | _check | through SNAPS framework | -| | +----------------+----------------------------------+ -| | | api_check | Check OpenStack API through | -| | | | SNAPS framework | -| +---------------+----------------+----------------------------------+ -| | smoke | vPing_SSH | NFV "Hello World" using an SSH | -| | | | connection to a destination VM | -| | | | over a created floating IP | -| | | | address on the SUT Public / | -| | | | External network. Using the SSH | -| | | | connection a test script is then | -| | | | copied to the destination | -| | | | VM and then executed via SSH. | -| | | | The script will ping another | -| | | | VM on a specified IP address over| -| | | | the SUT Private Tenant network. | -| | +----------------+----------------------------------+ -| | | vPing_userdata | Uses Ping with given userdata | -| | | | to test intra-VM connectivity | -| | | | over the SUT Private Tenant | -| | | | network. The correct operation | -| | | | of the NOVA Metadata service is | -| | | | also verified in this test. | -| | +----------------+----------------------------------+ -| | | tempest_smoke | Generate and run a relevant | -| | | \_serial | Tempest Test Suite in smoke mode.| -| | | | The generated test set is | -| | | | dependent on the OpenStack | -| | | | deployment environment. | -| | +----------------+----------------------------------+ -| | | rally_sanity | Run a subset of the OpenStack | -| | | | Rally Test Suite in smoke mode | -| | +----------------+----------------------------------+ -| | | snaps_smoke | Run a subset of the OpenStack | -| | | | Rally Test Suite in smoke mode | -| +---------------+----------------+----------------------------------+ -| | components | tempest_full | Generate and run a full set of | -| | | \_parallel | the OpenStack Tempest Test Suite.| -| | | | See the OpenStack reference test | -| | | | suite `[2]`_. The generated | -| | | | test set is dependent on the | -| | | | OpenStack deployment environment.| -| | +----------------+----------------------------------+ -| | | rally_full | Run the OpenStack testing tool | -| | | | benchmarking OpenStack modules | -| | | | See the Rally documents `[3]`_. | -+-------------+---------------+----------------+----------------------------------+ -| Controllers | smoke | odl | Opendaylight Test suite | -| | | | Limited test suite to check the | -| | | | basic neutron (Layer 2) | -| | | | operations mainly based on | -| | | | upstream testcases. See below | -| | | | for details | -| | +----------------+----------------------------------+ -| | | onos | Test suite of ONOS L2 and L3 | -| | | | functions. | -| | | | See `ONOSFW User Guide`_ for | -| | | | details. | -+-------------+---------------+----------------+----------------------------------+ -| Features | features | promise | Resource reservation and | -| | | | management project to identify | -| | | | NFV related requirements and | -| | | | realize resource reservation for | -| | | | future usage by capacity | -| | | | management of resource pools | -| | | | regarding compute, network and | -| | | | storage. | -| | | | See `Promise User Guide`_ for | -| | | | details. | -| | +----------------+----------------------------------+ -| | | doctor | Doctor platform, as of Colorado | -| | | | release, provides the three | -| | | | features: | -| | | | * Immediate Notification | -| | | | * Consistent resource state | -| | | | awareness for compute host down | -| | | | * Valid compute host status | -| | | | given to VM owner | -| | | | See `Doctor User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | bgpvpn | Implementation of the OpenStack | -| | | | bgpvpn API from the SDNVPN | -| | | | feature project. It allows for | -| | | | the creation of BGP VPNs. | -| | | | See `SDNVPN User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | security_scan | Implementation of a simple | -| | | | security scan. (Currently | -| | | | available only for the Apex | -| | | | installer environment) | -| | +----------------+----------------------------------+ -| | | onos-sfc | SFC testing for onos scenarios | -| | | | See `ONOSFW User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | odl-sfc | SFC testing for odl scenarios | -| | | | See `SFC User Guide`_ for details| -| | +----------------+----------------------------------+ -| | | domino | Domino provides TOSCA template | -| | | | distribution service for network | -| | | | service and VNF descriptors | -| | | | among MANO components e.g., | -| | | | NFVO, VNFM, VIM, SDN-C, etc., | -| | | | as well as OSS/BSS functions. | -| | | | See `Domino User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | copper | Copper develops OPNFV platform | -| | | | support for policy management, | -| | | | using open source projects such | -| | | | as OpenStack Congress, focused | -| | | | on helping ensure that virtual | -| | | | infrastructure and the apps that | -| | | | execute on it comply with the | -| | | | configuration policy intent of | -| | | | service providers, developers, | -| | | | and end users. See more detail | -| | | | in the `Copper User Guide`_. | -| | +----------------+----------------------------------+ -| | | multisites | Multisites | -| | | | See `Multisite User Guide`_ for | -| | | | details | -+-------------+---------------+----------------+----------------------------------+ -| VNF | vnf | cloudify_ims | Example of a real VNF deployment | -| | | | to show the NFV capabilities of | -| | | | the platform. The IP Multimedia | -| | | | Subsytem is a typical Telco test | -| | | | case, referenced by ETSI. | -| | | | It provides a fully functional | -| | | | VoIP System | -| | +----------------+----------------------------------+ -| | | opera_ims | vIMS deployment using openBaton | -| | +----------------+----------------------------------+ -| | | orchestra_ims | vIMS deployment using open-O | -+ +---------------+----------------+----------------------------------+ -| | | parser | Parser is an integration project | -| | | | which aims to provide | -| | | | placement/deployment templates | -| | | | translation for OPNFV platform, | -| | | | including TOSCA -> HOT, POLICY ->| -| | | | TOSCA and YANG -> TOSCA. | -| | | | See `Parser User Guide`_ for | -| | | | details | -+-------------+---------------+----------------+----------------------------------+ - - -As shown in the above table, Functest is structured into different 'domains', -'tiers' and 'test cases'. Each 'test case' usually represents an actual -'Test Suite' comprised -in turn- of several test cases internally. - -Test cases also have an implicit execution order. For example, if the early -'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke' -Tier testcases, there is little point to launch a full testcase execution round. - -In Danube, we merged smoke and sdn controller tiers in smoke tier. - -An overview of the Functest Structural Concept is depicted graphically below: - -.. figure:: ../images/concepts_mapping_final.png - :align: center - :alt: Functest Concepts Structure - -Some of the test cases are developed by Functest team members, whereas others -are integrated from upstream communities or other OPNFV projects. For example, -`Tempest `_ is the -OpenStack integration test suite and Functest is in charge of the selection, -integration and automation of those tests that fit suitably to OPNFV. - -The Tempest test suite is the default OpenStack smoke test suite but no new test -cases have been created in OPNFV Functest. - -The results produced by the tests run from CI are pushed and collected into a -NoSQL database. The goal is to populate the database with results from different -sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot -of a live Functest Dashboard is shown below: - -** TODO ** -.. figure:: ../images/FunctestDashboardDanube.png - :align: center - :alt: Functest Dashboard - - -Basic components (VIM, SDN controllers) are tested through their own suites. -Feature projects also provide their own test suites with different ways of -running their tests. - -The notion of domain has been introduced in the description of the test cases -stored in the Database. -This parameters as well as possible tags can be used for the Test case catalog. - -vIMS test case was integrated to demonstrate the capability to deploy a -relatively complex NFV scenario on top of the OPNFV infrastructure. - -Functest considers OPNFV as a black box. As of Danube release the OPNFV -offers a lot of potential combinations: - - * 3 controllers (OpenDaylight, ONOS, OpenContrail) - * 4 installers (Apex, Compass, Fuel, Joid) - -Most of the tests are runnable by any combination, but some tests might have -restrictions imposed by the utilized installers or due to the available -deployed features. The system uses the environment variables (INSTALLER_IP and -DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given -environment. - -A convenience Functest CLI utility is also available to simplify setting up the -Functest evironment, management of the OpenStack environment (e.g. resource -clean-up) and for executing tests. -The Functest CLI organised the testcase into logical Tiers, which contain in -turn one or more testcases. The CLI allows execution of a single specified -testcase, all test cases in a specified Tier, or the special case of execution -of **ALL** testcases. The Functest CLI is introduced in more detail in the -section `Executing the functest suites`_ of this document. - -.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html -.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html -.. _`Copper User Guide`: http://artifacts.opnfv.org/copper/colorado/docs/userguide/index.html -.. _`Doctor User Guide`: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html -.. _`Promise User Guide`: http://artifacts.opnfv.org/promise/colorado/docs/userguide/index.html -.. _`ONOSFW User Guide`: http://artifacts.opnfv.org/onosfw/colorado/userguide/index.html -.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html -.. _`Domino User Guide`: http://artifacts.opnfv.org/domino/docs/userguide-single/index.html -.. _`Parser User Guide`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html -.. _`Functest Dashboard`: http://testresults.opnfv.org/kibana_dashboards/ -.. _`SFC User Guide`: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html -.. _`Multisite User Guide`: http://artifacts.opnfv.org/multisite/docs/userguide/index.html diff --git a/docs/userguide/runfunctest.rst b/docs/userguide/runfunctest.rst deleted file mode 100644 index e7ab84b26..000000000 --- a/docs/userguide/runfunctest.rst +++ /dev/null @@ -1,385 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - -Executing the functest suites -============================= - -Manual testing --------------- - -This section assumes the following: - * The Functest Docker container is running - * The docker prompt is shown - * The Functest environment is ready (Functest CLI command 'functest env prepare' - has been executed) - -If any of the above steps are missing please refer to the Functest Config Guide -as they are a prerequisite and all the commands explained in this section **must** be -performed **inside the container**. - -The Functest CLI offers two commands (functest tier ...) and (functest testcase ... ) -for the execution of Test Tiers or Test Cases:: - - root@22e436918db0:~/repos/functest/ci# functest tier --help - Usage: functest tier [OPTIONS] COMMAND [ARGS]... - - Options: - -h, --help Show this message and exit. - - Commands: - get-tests Prints the tests in a tier. - list Lists the available tiers. - run Executes all the tests within a tier. - show Shows information about a tier. - root@22e436918db0:~/repos/functest/ci# functest testcase --help - - Usage: functest testcase [OPTIONS] COMMAND [ARGS]... - - Options: - -h, --help Show this message and exit. - - Commands: - list Lists the available testcases. - run Executes a test case. - show Shows information about a test case. - -More details on the existing Tiers and Test Cases can be seen with the 'list' -command:: - - root@22e436918db0:~/repos/functest/ci# functest tier list - - 0. healthcheck: - ['healthcheck', 'connection_check', 'api_check',] - - 1. smoke: - ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity', 'snaps_smoke', 'odl'] - - 2. features: - ['doctor', 'security_scan'] - - 3. components: - ['tempest_full_parallel', 'rally_full'] - - 4. vnf: - ['cloudify_ims'] - - and - - root@22e436918db0:~/repos/functest/ci# functest testcase list - healthcheck - api_check - connection_check - vping_ssh - vping_userdata - snaps_smoke - tempest_smoke_serial - rally_sanity - odl - doctor - security_scan - tempest_full_parallel - rally_full - cloudify_ims - -More specific details on specific Tiers or Test Cases can be seen wih the -'show' command:: - - root@22e436918db0:~/repos/functest/ci# functest tier show smoke - +======================================================================+ - | Tier: smoke | - +======================================================================+ - | Order: 1 | - | CI Loop: (daily)|(weekly) | - | Description: | - | Set of basic Functional tests to validate the OpenStack | - | deployment. | - | Test cases: | - | - vping_ssh | - | - vping_userdata | - | - tempest_smoke_serial | - | - rally_sanity | - | | - +----------------------------------------------------------------------+ - - and - - root@22e436918db0:~/repos/functest/ci# functest testcase show tempest_smoke_serial - +======================================================================+ - | Testcase: tempest_smoke_serial | - +======================================================================+ - | Description: | - | This test case runs the smoke subset of the OpenStack Tempest | - | suite. The list of test cases is generated by Tempest | - | automatically and depends on the parameters of the OpenStack | - | deplopyment. | - | Dependencies: | - | - Installer: | - | - Scenario : | - | | - +----------------------------------------------------------------------+ - - -To execute a Test Tier or Test Case, the 'run' command is used:: - - root@22e436918db0:~/repos/functest/ci# functest tier run healthcheck - Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t healthcheck' - 2016-06-30 11:44:56,933 - run_tests - INFO - Sourcing the OpenStack RC file... - 2016-06-30 11:44:56,937 - run_tests - INFO - ############################################ - 2016-06-30 11:44:56,938 - run_tests - INFO - Running tier 'healthcheck' - 2016-06-30 11:44:56,938 - run_tests - INFO - ############################################ - 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================ - 2016-06-30 11:44:56,938 - run_tests - INFO - Running test case 'healthcheck'... - 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================ - 2016-06-30 11:44:56,953 - healtcheck - INFO - Testing Keystone API... - 2016-06-30 11:45:05,351 - healtcheck - INFO - ...Keystone OK! - 2016-06-30 11:45:05,354 - healtcheck - INFO - Testing Glance API... - 2016-06-30 11:45:29,746 - healtcheck - INFO - ... Glance OK! - 2016-06-30 11:45:29,749 - healtcheck - INFO - Testing Cinder API... - 2016-06-30 11:45:37,502 - healtcheck - INFO - ...Cinder OK! - 2016-06-30 11:45:37,505 - healtcheck - INFO - Testing Neutron API... - 2016-06-30 11:45:39,664 - healtcheck - INFO - External network found. ccd98ad6-d34a-4768-b03c-e28ecfcd51ca - 2016-06-30 11:45:39,667 - healtcheck - INFO - 1. Create Networks... - 2016-06-30 11:45:44,227 - healtcheck - INFO - 2. Create subnets... - 2016-06-30 11:45:46,805 - healtcheck - INFO - 4. Create Routers... - 2016-06-30 11:45:54,261 - healtcheck - INFO - ...Neutron OK! - 2016-06-30 11:45:54,264 - healtcheck - INFO - Testing Nova API... - 2016-06-30 11:47:12,272 - healtcheck - INFO - ...Nova OK! - 2016-06-30 11:47:12,274 - healtcheck - INFO - Checking if instances get an IP from DHCP... - : - : - 2016-06-30 11:48:17,832 - healtcheck - INFO - ...DHCP OK! - 2016-06-30 11:48:17,835 - healtcheck - INFO - Health check passed! - 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ - 2016-06-30 11:48:17,837 - clean_openstack - INFO - Cleaning OpenStack resources... - 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ - Version 1 is deprecated, use alternative version 2 instead. - WARNING:cinderclient.api_versions:Version 1 is deprecated, use alternative version 2 instead. - 2016-06-30 11:48:18,272 - clean_openstack - INFO - Removing Nova instances... - 2016-06-30 11:48:24,439 - clean_openstack - INFO - ------------------------------------------- - 2016-06-30 11:48:24,440 - clean_openstack - INFO - Removing Glance images... - 2016-06-30 11:48:35,853 - clean_openstack - INFO - ------------------------------------------- - 2016-06-30 11:48:35,854 - clean_openstack - INFO - Removing Cinder volumes... - 2016-06-30 11:48:37,344 - clean_openstack - INFO - ------------------------------------------- - 2016-06-30 11:48:37,344 - clean_openstack - INFO - Removing floating IPs... - 2016-06-30 11:48:37,467 - clean_openstack - INFO - ------------------------------------------- - 2016-06-30 11:48:37,467 - clean_openstack - INFO - Removing Neutron objects - 2016-06-30 11:48:53,633 - clean_openstack - INFO - ------------------------------------------- - 2016-06-30 11:48:53,633 - clean_openstack - INFO - Removing Security groups... - 2016-06-30 11:48:53,689 - clean_openstack - INFO - ------------------------------------------- - 2016-06-30 11:48:53,689 - clean_openstack - INFO - Removing Users... - 2016-06-30 11:48:54,444 - clean_openstack - INFO - ------------------------------------------- - 2016-06-30 11:48:54,444 - clean_openstack - INFO - Removing Tenants... - 2016-06-30 11:48:54,711 - clean_openstack - INFO - ------------------------------------------- - - and - - root@22e436918db0:~/repos/functest/ci# functest testcase run vping_ssh - Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t vping_ssh' - 2016-06-30 11:50:31,861 - run_tests - INFO - Sourcing the OpenStack RC file... - 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================ - 2016-06-30 11:50:31,865 - run_tests - INFO - Running test case 'vping_ssh'... - 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================ - 2016-06-30 11:50:32,977 - vping_ssh - INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.5-x86_64-disk.img'... - 2016-06-30 11:50:45,470 - vping_ssh - INFO - Creating neutron network vping-net... - 2016-06-30 11:50:47,645 - vping_ssh - INFO - Creating security group 'vPing-sg'... - 2016-06-30 11:50:48,843 - vping_ssh - INFO - Using existing Flavor 'm1.small'... - 2016-06-30 11:50:48,927 - vping_ssh - INFO - vPing Start Time:'2016-06-30 11:50:48' - 2016-06-30 11:50:48,927 - vping_ssh - INFO - Creating instance 'opnfv-vping-1'... - 2016-06-30 11:51:34,664 - vping_ssh - INFO - Instance 'opnfv-vping-1' is ACTIVE. - 2016-06-30 11:51:34,818 - vping_ssh - INFO - Adding 'opnfv-vping-1' to security group 'vPing-sg'... - 2016-06-30 11:51:35,209 - vping_ssh - INFO - Creating instance 'opnfv-vping-2'... - 2016-06-30 11:52:01,439 - vping_ssh - INFO - Instance 'opnfv-vping-2' is ACTIVE. - 2016-06-30 11:52:01,439 - vping_ssh - INFO - Adding 'opnfv-vping-2' to security group 'vPing-sg'... - 2016-06-30 11:52:01,754 - vping_ssh - INFO - Creating floating IP for VM 'opnfv-vping-2'... - 2016-06-30 11:52:01,969 - vping_ssh - INFO - Floating IP created: '10.17.94.140' - 2016-06-30 11:52:01,969 - vping_ssh - INFO - Associating floating ip: '10.17.94.140' to VM 'opnfv-vping-2' - 2016-06-30 11:52:02,792 - vping_ssh - INFO - Trying to establish SSH connection to 10.17.94.140... - 2016-06-30 11:52:19,915 - vping_ssh - INFO - Waiting for ping... - 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing detected! - 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing duration:'92.2' s. - 2016-06-30 11:52:21,109 - vping_ssh - INFO - vPing OK - 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ - 2016-06-30 11:52:21,153 - clean_openstack - INFO - Cleaning OpenStack resources... - 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ - Version 1 is deprecated, use alternative version 2 instead. - : - : - etc. - -To list the test cases which are part of a specific Test Tier, the 'get-tests' -command is used with 'functest tier':: - - root@22e436918db0:~/repos/functest/ci# functest tier get-tests healthcheck - Test cases in tier 'healthcheck': - ['healthcheck'] - - -Please note that for some scenarios some test cases might not be launched. -For example, the last example displayed only the 'odl' testcase for the given -environment. In this particular system the deployment does not support the 'onos' SDN -Controller Test Case; for example. - -**Important** If you use the command 'functest tier run ', then the -Functest CLI utility will call **all valid Test Cases**, which belong to the -specified Test Tier, as relevant to scenarios deployed to the SUT environment. -Thus, the Functest CLI utility calculates automatically which tests can be -executed and which cannot, given the environment variable **DEPLOY_SCENARIO**, -which is passed in to the Functest docker container. - -Currently, the Functest CLI command 'functest testcase run ', supports -two possibilities:: - - * Run a single Test Case, specified by a valid choice of - * Run ALL test Test Cases (for all Tiers) by specifying = 'all' - -Functest includes a cleaning mechanism in order to remove all the OpenStack -resources except those present before running any test. The script -*$REPOS_DIR/functest/functest/utils/generate_defaults.py* is called once when setting up -the Functest environment (i.e. CLI command 'functest env prepare') to snapshot -all the OpenStack resources (images, networks, volumes, security groups, tenants, -users) so that an eventual cleanup does not remove any of these defaults. - -The script **clean_openstack.py** which is located in -*$REPOS_DIR/functest/functest/utils/* is normally called after a test execution. It is -in charge of cleaning the OpenStack resources that are not specified in the -defaults file generated previously which is stored in -*/home/opnfv/functest/conf/os_defaults.yaml* in the Functest docker container. - -It is important to mention that if there are new OpenStack resources created -manually after preparing the Functest environment, they will be removed, unless -you use the special method of invoking the test case with specific suppression -of clean up. (See the `Troubleshooting`_ section). - -The reason to include this cleanup meachanism in Functest is because some -test suites such as Tempest or Rally create a lot of resources (users, -tenants, networks, volumes etc.) that are not always properly cleaned, so this -function has been set to keep the system as clean as it was before a -full Functest execution. - -Although the Functest CLI provides an easy way to run any test, it is possible to -do a direct call to the desired test script. For example: - - python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/vPing/vPing_ssh.py -d - - -Automated testing ------------------ - -As mentioned previously, the Functest Docker container preparation as well as -invocation of Test Cases can be called within the container from the Jenkins CI -system. There are 3 jobs that automate the whole process. The first job runs all -the tests referenced in the daily loop (i.e. that must been run daily), the second -job runs the tests referenced in the weekly loop (usually long duration tests run -once a week maximum) and the third job allows testing test suite by test suite specifying -the test suite name. The user may also use either of these Jenkins jobs to execute -the desired test suites. - -One of the most challenging task in the Danube release consists -in dealing with lots of scenarios and installers. Thus, when the tests are -automatically started from CI, a basic algorithm has been created in order to -detect whether a given test is runnable or not on the given scenario. -Some Functest test suites cannot be systematically run (e.g. ODL suite can not -be run on an ONOS scenario). The daily/weekly notion has been introduces in -Colorado in order to save CI time and avoid running systematically -long duration tests. It was not used in Colorado due to CI resource shortage. -The mechanism remains however as part of the CI evolution. - -CI provides some useful information passed to the container as environment -variables: - - * Installer (apex|compass|daisy|fuel|joid), stored in INSTALLER_TYPE - * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP - * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with - - * controller = (odl|onos|ocl|nosdn) - * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|multisites) - * mode = (ha|noha) - -The constraints per test case are defined in the Functest configuration file -*/home/opnfv/repos/functest/functest/ci/testcases.yaml*:: - - tiers: - - - name: healthcheck - order: 0 - ci_loop: '(daily)|(weekly)' - description : >- - First tier to be executed to verify the basic - operations in the VIM. - testcases: - - - name: healthcheck - criteria: 'status == "PASS"' - blocking: true - description: >- - This test case verifies the basic OpenStack services like - Keystone, Glance, Cinder, Neutron and Nova. - - dependencies: - installer: '' - scenario: '' - - - - name: smoke - order: 1 - ci_loop: '(daily)|(weekly)' - description : >- - Set of basic Functional tests to validate the OpenStack deployment. - testcases: - - - name: vping_ssh - criteria: 'status == "PASS"' - blocking: true - description: >- - This test case verifies: 1) SSH to an instance using floating - IPs over the public network. 2) Connectivity between 2 instances - over a private network. - dependencies: - installer: '' - scenario: '^((?!bgpvpn|odl_l3).)*$' - run: - module: 'functest.opnfv_tests.openstack.vping.vping_ssh' - class: 'VPingSSH' - .... - -We may distinguish 2 levels in the test case description: - * Tier level - * Test case level - -At the tier level, we define the following parameters: - - * ci_loop: indicate if in automated mode, the test case must be run in daily and/or weekly jobs - * description: a high level view of the test case - -For a given test case we defined: - * the name of the test case - * the criteria (experimental): a criteria used to declare the test case as PASS or FAIL - * blocking: if set to true, if the test is failed, the execution of the following tests is canceled - * the description of the test case - * the dependencies: a combination of 2 regex on the scenario and the installer name - * run: In Danube we introduced the notion of abstract class in order to harmonize the way to run internal, feature or vnf tests - -For further details on abstraction classes, see developper guide. - -Additional parameters have been added in the desription in the Database. -The target is to use the configuration stored in the Database and consider the -local file as backup if the Database is not reachable. -The additional fields related to a test case are: - * trust: we introduced this notion to put in place a mechanism of scenario promotion. - * Version: it indicates since which version you can run this test - * domains: the main domain covered by the test suite - * tags: a list of tags related to the test suite - -The order of execution is the one defined in the file if all test cases are selected. - -In CI daily job the tests are executed in the following order: - - 1) healthcheck (blocking) - 2) smoke: both vPings are blocking - 3) Feature project tests cases - -In CI weekly job we add 2 tiers: - - 4) VNFs (vIMS) - 5) Components (Rally and Tempest long duration suites) - -As explained before, at the end of an automated execution, the OpenStack resources -might be eventually removed. -Please note that a system snapshot is taken before any test case execution. - -This testcase.yaml file is used for CI, for the CLI and for the automatic reporting. diff --git a/docs/userguide/troubleshooting.rst b/docs/userguide/troubleshooting.rst deleted file mode 100644 index 845501916..000000000 --- a/docs/userguide/troubleshooting.rst +++ /dev/null @@ -1,387 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - -Troubleshooting -=============== - -This section gives some guidelines about how to troubleshoot the test cases -owned by Functest. - -**IMPORTANT**: As in the previous section, the steps defined below must be -executed inside the Functest Docker container and after sourcing the OpenStack -credentials:: - - . $creds - -or:: - - source /home/opnfv/functest/conf/openstack.creds - -VIM ---- - -This section covers the test cases related to the VIM (healthcheck, vping_ssh, -vping_userdata, tempest_smoke_serial, tempest_full_parallel, rally_sanity, -rally_full). - -vPing common -^^^^^^^^^^^^ -For both vPing test cases (**vPing_ssh**, and **vPing_userdata**), the first steps are -similar: - - * Create Glance image - * Create Network - * Create Security Group - * Create Instances - -After these actions, the test cases differ and will be explained in their -respective section. - -These test cases can be run inside the container, using new Functest CLI as follows:: - - $ functest testcase run vping_ssh - $ functest testcase run vping_userdata - -The Functest CLI is designed to route a call to the corresponding internal -python scripts, located in paths: -*$REPOS_DIR/functest/functest/opnfv_tests/vPing/CI/libraries/vPing_ssh.py* and -*$REPOS_DIR/functest/functest/opnfv_tests/vPing/CI/libraries/vPing_userdata.py* - -Notes: - - #. There is one difference, between the Functest CLI based test case - execution compared to the earlier used Bash shell script, which is - relevant to point out in troubleshooting scenarios: - - The Functest CLI does **not yet** support the option to suppress - clean-up of the generated OpenStack resources, following the execution - of a test case. - - Explanation: After finishing the test execution, the corresponding - script will remove, by default, all created resources in OpenStack - (image, instances, network and security group). When troubleshooting, - it is advisable sometimes to keep those resources in case the test - fails and a manual testing is needed. - - It is actually still possible to invoke test execution, with suppression - of OpenStack resource cleanup, however this requires invocation of a - **specific Python script:** '/home/opnfv/repos/functest/ci/run_test.py'. - The `OPNFV Functest Developer Guide`_ provides guidance on the use of that - Python script in such troubleshooting cases. - -Some of the common errors that can appear in this test case are:: - - vPing_ssh- ERROR - There has been a problem when creating the neutron network.... - -This means that there has been some problems with Neutron, even before creating the -instances. Try to create manually a Neutron network and a Subnet to see if that works. -The debug messages will also help to see when it failed (subnet and router creation). -Example of Neutron commands (using 10.6.0.0/24 range for example):: - - neutron net-create net-test - neutron subnet-create --name subnet-test --allocation-pool start=10.6.0.2,end=10.6.0.100 \ - --gateway 10.6.0.254 net-test 10.6.0.0/24 - neutron router-create test_router - neutron router-interface-add test_subnet - neutron router-gateway-set - -Another related error can occur while creating the Security Groups for the instances:: - - vPing_ssh- ERROR - Failed to create the security group... - -In this case, proceed to create it manually. These are some hints:: - - neutron security-group-create sg-test - neutron security-group-rule-create sg-test --direction ingress --protocol icmp \ - --remote-ip-prefix 0.0.0.0/0 - neutron security-group-rule-create sg-test --direction ingress --ethertype IPv4 \ - --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0 - neutron security-group-rule-create sg-test --direction egress --ethertype IPv4 \ - --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0 - -The next step is to create the instances. The image used is located in -*/home/opnfv/functest/data/cirros-0.3.5-x86_64-disk.img* and a Glance image is created -with the name **functest-vping**. If booting the instances fails (i.e. the status -is not **ACTIVE**), you can check why it failed by doing:: - - nova list - nova show - -It might show some messages about the booting failure. To try that manually:: - - nova boot --flavor m1.small --image functest-vping --nic net-id= nova-test - -This will spawn a VM using the network created previously manually. -In all the OPNFV tested scenarios from CI, it never has been a problem with the -previous actions. Further possible problems are explained in the following sections. - - -vPing_SSH -^^^^^^^^^ -This test case creates a floating IP on the external network and assigns it to -the second instance **opnfv-vping-2**. The purpose of this is to establish -a SSH connection to that instance and SCP a script that will ping the first -instance. This script is located in the repository under -*$REPOS_DIR/functest/functest/opnfv_tests/OpenStack/vPing/ping.sh* and takes an IP as -a parameter. When the SCP is completed, the test will do an SSH call to that script -inside the second instance. Some problems can happen here:: - - vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting - -If this is displayed, stop the test or wait for it to finish, if you have used -the special method of test invocation with specific supression of OpenStack -resource clean-up, as explained earler. It means that the Container can not -reach the Public/External IP assigned to the instance **opnfv-vping-2**. There -are many possible reasons, and they really depend on the chosen scenario. For -most of the ODL-L3 and ONOS scenarios this has been noticed and it is a known -limitation. - -First, make sure that the instance **opnfv-vping-2** succeeded to get an IP -from the DHCP agent. It can be checked by doing:: - - nova console-log opnfv-vping-2 - -If the message *Sending discover* and *No lease, failing* is shown, it probably -means that the Neutron dhcp-agent failed to assign an IP or even that it was not -responding. At this point it does not make sense to try to ping the floating IP. - -If the instance got an IP properly, try to ping manually the VM from the container:: - - nova list - - ping - -If the ping does not return anything, try to ping from the Host where the Docker -container is running. If that solves the problem, check the iptable rules because -there might be some rules rejecting ICMP or TCP traffic coming/going from/to the -container. - -At this point, if the ping does not work either, try to reproduce the test -manually with the steps described above in the vPing common section with the -addition:: - - neutron floatingip-create - nova floating-ip-associate nova-test - - -Further troubleshooting is out of scope of this document, as it might be due to -problems with the SDN controller. Contact the installer team members or send an -email to the corresponding OPNFV mailing list for more information. - - - -vPing_userdata -^^^^^^^^^^^^^^ -This test case does not create any floating IP neither establishes an SSH -connection. Instead, it uses nova-metadata service when creating an instance -to pass the same script as before (ping.sh) but as 1-line text. This script -will be executed automatically when the second instance **opnfv-vping-2** is booted. - -The only known problem here for this test to fail is mainly the lack of support -of cloud-init (nova-metadata service). Check the console of the instance:: - - nova console-log opnfv-vping-2 - -If this text or similar is shown:: - - checking http://169.254.169.254/2009-04-04/instance-id - failed 1/20: up 1.13. request failed - failed 2/20: up 13.18. request failed - failed 3/20: up 25.20. request failed - failed 4/20: up 37.23. request failed - failed 5/20: up 49.25. request failed - failed 6/20: up 61.27. request failed - failed 7/20: up 73.29. request failed - failed 8/20: up 85.32. request failed - failed 9/20: up 97.34. request failed - failed 10/20: up 109.36. request failed - failed 11/20: up 121.38. request failed - failed 12/20: up 133.40. request failed - failed 13/20: up 145.43. request failed - failed 14/20: up 157.45. request failed - failed 15/20: up 169.48. request failed - failed 16/20: up 181.50. request failed - failed 17/20: up 193.52. request failed - failed 18/20: up 205.54. request failed - failed 19/20: up 217.56. request failed - failed 20/20: up 229.58. request failed - failed to read iid from metadata. tried 20 - -it means that the instance failed to read from the metadata service. Contact -the Functest or installer teams for more information. - -NOTE: Cloud-init in not supported on scenarios dealing with ONOS and the tests -have been excluded from CI in those scenarios. - - -Tempest -^^^^^^^ - -In the upstream OpenStack CI all the Tempest test cases are supposed to pass. -If some test cases fail in an OPNFV deployment, the reason is very probably one -of the following - -+-----------------------------+-----------------------------------------------------+ -| Error | Details | -+=============================+=====================================================+ -| Resources required for test | Such resources could be e.g. an external network | -| case execution are missing | and access to the management subnet (adminURL) from | -| | the Functest docker container. | -+-----------------------------+-----------------------------------------------------+ -| OpenStack components or | Check running services in the controller and compute| -| services are missing or not | nodes (e.g. with "systemctl" or "service" commands).| -| configured properly | Configuration parameters can be verified from the | -| | related .conf files located under '/etc/'| -| | directories. | -+-----------------------------+-----------------------------------------------------+ -| Some resources required for | The tempest.conf file, automatically generated by | -| execution test cases are | Rally in Functest, does not contain all the needed | -| missing | parameters or some parameters are not set properly. | -| | The tempest.conf file is located in directory | -| | '/home/opnfv/.rally/tempest/for-deployment-' | -| | in the Functest Docker container. Use the "rally | -| | deployment list" command in order to check the UUID | -| | the UUID of the current deployment. | -+-----------------------------+-----------------------------------------------------+ - - -When some Tempest test case fails, captured traceback and possibly also the -related REST API requests/responses are output to the console. More detailed debug -information can be found from tempest.log file stored into related Rally deployment -folder. - - -Rally -^^^^^ - -The same error causes which were mentioned above for Tempest test cases, may also -lead to errors in Rally as well. - -It is possible to run only one Rally scenario, instead of the whole suite. -To do that, call the alternative python script as follows:: - - python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/rally/run_rally-cert.py -h - usage: run_rally-cert.py [-h] [-d] [-r] [-s] [-v] [-n] test_name - - positional arguments: - test_name Module name to be tested. Possible values are : [ - authenticate | glance | cinder | heat | keystone | neutron | - nova | quotas | requests | vm | all ] The 'all' value - performs all possible test scenarios - - optional arguments: - -h, --help show this help message and exit - -d, --debug Debug mode - -r, --report Create json result file - -s, --smoke Smoke test mode - -v, --verbose Print verbose info about the progress - -n, --noclean Don't clean the created resources for this test. - -For example, to run the Glance scenario with debug information:: - - python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/rally/run_rally-cert.py -d glance - -Possible scenarios are: - * authenticate - * glance - * cinder - * heat - * keystone - * neutron - * nova - * quotas - * requests - * vm - -To know more about what those scenarios are doing, they are defined in directory: -*$REPOS_DIR/functest/functest/opnfv_tests/OpenStack/rally/scenario* -For more info about Rally scenario definition please refer to the Rally official -documentation. `[3]`_ - -If the flag *all* is specified, it will run all the scenarios one by one. Please -note that this might take some time (~1,5hr), taking around 1 hour alone to -complete the Nova scenario. - -To check any possible problems with Rally, the logs are stored under -*/home/opnfv/functest/results/rally/* in the Functest Docker container. - - -Controllers ------------ - -Opendaylight -^^^^^^^^^^^^ - -If the Basic Restconf test suite fails, check that the ODL controller is -reachable and its Restconf module has been installed. - -If the Neutron Reachability test fails, verify that the modules -implementing Neutron requirements have been properly installed. - -If any of the other test cases fails, check that Neutron and ODL have -been correctly configured to work together. Check Neutron configuration -files, accounts, IP addresses etc.). - - -ONOS -^^^^ - -Please refer to the ONOS documentation. `ONOSFW User Guide`_ . - - -Features --------- - -Please refer to the dedicated feature user guides for details. - - -security_scan -^^^^^^^^^^^^^ - -See OpenSCAP web site: https://www.open-scap.org/ - - - -NFV ---- - -cloudify_ims -^^^^^^^^^^^^ -vIMS deployment may fail for several reasons, the most frequent ones are -described in the following table: - -+-----------------------------------+------------------------------------+ -| Error | Comments | -+===================================+====================================+ -| Keystone admin API not reachable | Impossible to create vIMS user and | -| | tenant | -+-----------------------------------+------------------------------------+ -| Impossible to retrieve admin role | Impossible to create vIMS user and | -| id | tenant | -+-----------------------------------+------------------------------------+ -| Error when uploading image from | impossible to deploy VNF | -| OpenStack to glance | | -+-----------------------------------+------------------------------------+ -| Cinder quota cannot be updated | Default quotas not sufficient, they| -| | are adapted in the script | -+-----------------------------------+------------------------------------+ -| Impossible to create a volume | VNF cannot be deployed | -+-----------------------------------+------------------------------------+ -| SSH connection issue between the | if vPing test fails, vIMS test will| -| Test Docker container and the VM | fail... | -+-----------------------------------+------------------------------------+ -| No Internet access from the VM | the VMs of the VNF must have an | -| | external access to Internet | -+-----------------------------------+------------------------------------+ -| No access to OpenStack API from | Orchestrator can be installed but | -| the VM | the vIMS VNF installation fails | -+-----------------------------------+------------------------------------+ - - -parser -^^^^^^ - -For now log info is the only way to do trouble shooting - - -.. _`OPNFV Functest Developer Guide`: http://artifacts.opnfv.org/functest/docs/devguide/# -- cgit 1.2.3-korg