diff options
| -rw-r--r-- | docker/requirements.pip | 1 | ||||
| -rw-r--r-- | docs/configguide/configguide.rst | 701 | ||||
| -rw-r--r-- | docs/configguide/index.rst | 249 | ||||
| -rw-r--r-- | docs/release-notes/functest-release.rst | 321 | ||||
| -rw-r--r-- | docs/userguide/index.rst | 3 | ||||
| -rw-r--r-- | docs/userguide/introduction.rst | 11 | ||||
| -rw-r--r-- | docs/userguide/troubleshooting.rst | 2 |
7 files changed, 641 insertions, 647 deletions
diff --git a/docker/requirements.pip b/docker/requirements.pip index 52d0a9cc7..25ec795f3 100644 --- a/docker/requirements.pip +++ b/docker/requirements.pip @@ -26,3 +26,4 @@ python-congressclient==1.3.0 subprocess32 shyaml dnspython +Pillow==3.3.0 diff --git a/docs/configguide/configguide.rst b/docs/configguide/configguide.rst index 9f6693980..c1c717b13 100644 --- a/docs/configguide/configguide.rst +++ b/docs/configguide/configguide.rst @@ -2,142 +2,175 @@ .. http://creativecommons.org/licenses/by/4.0 -Preparing the Docker container ------------------------------- - -Pull the Functest Docker image ('opnfv/functest') from the public dockerhub -registry under the OPNFV account: [dockerhub_], with the following docker -command:: +Pulling the Docker image +------------------------ +Pull the Functest Docker image ('opnfv/functest') from the public +dockerhub registry under the OPNFV account: [dockerhub_], with the +following docker command:: docker pull opnfv/functest:<TagIdentifier> -where <TagIdentifier> identifies a specifically tagged release of the Functest -docker container image in the public dockerhub registry. There are many -different tags created automatically by the CI mechanisms, but you must ensure -you pull an image with the **correct tag** to match the OPNFV software release -installed in your environment. All available tagged images can be seen from -location [FunctestDockerTags_]. For example, when running on the first official -release of the OPNFV Colorado system platform, tag "colorado.1.0" is needed. -Pulling other tags might cause some problems while running the tests. If you -need to specifically pull the latest Functest docker image, then omit the tag -argument:: - +where <TagIdentifier> identifies a release of the Functest docker +container image in the public dockerhub registry. There are many tags +created automatically by the CI mechanisms, and you must ensure you +pull an image with the **correct tag** to match the OPNFV software +release installed in your environment. All available tagged images can +be seen from location [FunctestDockerTags_]. For example, when running +on the first official release of the OPNFV Colorado system platform, +tag "colorado.1.0" is needed. Pulling other tags might cause some +problems while running the tests. If you need to specifically pull the +latest Functest docker image, then omit the tag argument:: docker pull opnfv/functest -After pulling the Docker image, check that the pulled image is available with -the following docker command:: +After pulling the Docker image, check that it is available with the +following docker command:: [functester@jumphost ~]$ docker images - REPOSITORY TAG IMAGE ID CREATED SIZE opnfv/functest latest 8cd6683c32ae 2 weeks ago 1.611 GB opnfv/functest brahmaputra.3.0 94b78faa94f7 4 weeks ago 874.9 MB hello-world latest 94df4f0ce8a4 7 weeks ago 967 B - (Docker images pulled without a tag specifier bear the implicitly - assigned label "latest", as seen above.) - -The Functest docker container environment can -in principle- be also used with -non-OPNFV official installers (e.g. 'devstack), with the **disclaimer** that -support for such environments is outside of the scope of responsibility of the -OPNFV project. - -The minimum command to create the Functest Docker container can be described as -follows:: - - docker run -it opnfv/functest:<TagIdentifier> /bin/bash - -For OPNFV official installers, it is recommended (although no longer mandatory) -to provide two additional environment variables, in the 'docker run ...' -command nvocation: - - * **INSTALLER_TYPE** : possible values are **apex**, **compass**, **fuel** or - **joid**. - * **INSTALLER_IP** : IP of the installer node/VM. - -Functest may need to know the IP of the installer to retrieve automatically the -credentials from the installer node/VM or even from the actual controllers. - -Thus, the recommended minimum command to create the Functest Docker container -for OPNFV installer can be described (using installer 'fuel', and an invented -INSTALLER_IP of '10.20.0.2', for example), as follows:: - - docker run -it \ - -e "INSTALLER_IP=10.20.0.2" \ - -e "INSTALLER_TYPE=fuel" \ - opnfv/functest:<TagIdentifier> /bin/bash - -Optionally, it is possible to assign precisely a container name through the -**--name** option:: - - docker run --name "CONTAINER_NAME" -it \ - -e "INSTALLER_IP=10.20.0.2" \ - -e "INSTALLER_TYPE=fuel" \ - opnfv/functest:<TagIdentifier> /bin/bash +Docker images pulled without a tag specifier bear the implicitly +assigned label "latest", as seen above. + +The Functest docker container environment can -in principle- be also +used with non-OPNFV official installers (e.g. 'devstack'), with the +**disclaimer** that support for such environments is outside of the +scope and responsibility of the OPNFV project. + +Accessing the Openstack credentials +----------------------------------- +OpenStack credentials are mandatory and can be retrieved in different +ways. From inside the running Functest docker container the +"functest env prepare" command will automatically look for the +Openstack credentials file "/home/opnfv/functest/conf/openstack.creds" +and retrieve it unless the file already exists. This Functest +environment preparation step is described later in this document. + +WARNING: When the installer type is "joid" you have to have the +credentials file inside the running container **before** initiating the +functest environment preparation. For that reason you have to choose +either one of the options below, since the automated copying does not +work for "joid". + +You can also specifically pass in the needed file prior to running the +environment preparation either: + + * by using the -v option when creating the Docker container. This is + referred to in docker documentation as "Bind Mounting". See the + usage of this parameter in the following chapter. + * or creating a local file '/home/opnfv/functest/conf/openstack.creds' + inside the running container with the credentials in it. Consult + your installer guide for further details. This is however not + instructed in this document. + +NOTE: When the installer type is "fuel" and virtualized deployment +is used, there you have to explicitly fetch the credentials file +executing the following sequence + + #. Create a container as described in next chapter but do not + "Bind Mount" the credentials + #. Log in to container and execute the following command. Replace + the IP with installer address after the "-a" parameter:: + + $repos_dir/releng/utils/fetch_os_creds.sh \ + -d /home/opnfv/functest/conf/openstack.creds \ + -i fuel \ + -a 10.20.0.2 \ + -v + ( -d specifies the full path to the Openstack credential file + -i specifies the INSTALLER_TYPE + -a specifies the INSTALLER_IP + -v indicates a virtualized environment and takes no arguments ) -It is also possible to to indicate the path of the OpenStack credentials using a -**-v** option:: + #. Continue with your testing, initiate functest environment + preparation, run tests etc. - docker run -it \ - -e "INSTALLER_IP=10.20.0.2" \ - -e "INSTALLER_TYPE=fuel" \ - -v <path_to_your_local_creds_file>:/home/opnfv/functest/conf/openstack.creds \ - opnfv/functest:<TagIdentifier> /bin/bash +In proxified environment you may need to change the credentials file. +There are some tips in chapter: `Proxy support`_ - NOTE: Make sure you have placed the needed credential file into the - Jumphost local path <path_to_your_local_cred_file>. For the - Apex Installer you will need to pre-copy the required OpenStack - credentials file from the Instack/Undercloud Virtual Machine. - See the section 'Apex Installer Tips' later in this document. +Functest Docker parameters +-------------------------- +This chapter explains how to run a container for executing functest +test suites. Numbered list below explains some details of the +recommended parameters for invoking docker container - Warning - ------- - If you are using the Joid installer, you must use the method above - to provide the required OpenStack credentials. See the section - 'Focus on the OpenStack credentials' later in this document. + #. It is a good practice to assign a precise container name through + the **--name** option. + #. Assign parameter for installer type:: -The local openstack credential file will be mounted in the Docker container -under the path: '/home/opnfv/functest/conf/openstack.creds' + -e "INSTALLER_TYPE=<type>" + # Use one of following apex, compass, fuel or joid -If the intention is to run Functest against any of the supported OPNFV -scenarios, it is recommended to include also the environment variable -**DEPLOY_SCENARIO**. The **DEPLOY_SCENARIO** environment variable is passed with the format:: + #. Functest needs to know the IP of the installer:: - -e "DEPLOY_SCENARIO=os-<controller>-<nfv_feature>-<ha_mode>" + -e "INSTALLER_IP=<Specific IP Address>" - where: - os = OpenStack (No other VIM choices currently available) - controller is one of ( nosdn | odl_l2 | odl_l3 | onos | ocl ) - nfv_feature is one or more of ( ovs | kvm | sfc | bgpvpn | nofeature ) - If several features are pertinent then use the underscore - character '_' to separate each feature (e.g. ovs_kvm) - 'nofeature' indicates no NFV feature is deployed - ha_mode is one of ( ha | noha ) + #. Credentials for accessing the Openstack. + Most convenient way of passing them to container is by having a + local copy of the credentials file in Jumphost and then using the + **-v** option. In the example we have local file by the name of + "overcloudrc" and we are using that as an argument:: -For example:: + -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds - docker run -it \ + The credentials file needs to exist in the Docker container + under the path: '/home/opnfv/functest/conf/openstack.creds'. + + WARNING: If you are using the Joid installer, you must pass the + credentials using the **-v** option. See the section + `Accessing the Openstack credentials`_ above. + + #. Passing deployment scenario + When running Functest against any of the supported OPNFV scenarios, + it is recommended to include also the environment variable + **DEPLOY_SCENARIO**. The **DEPLOY_SCENARIO** environment variable + is passed with the format:: + + -e "DEPLOY_SCENARIO=os-<controller>-<nfv_feature>-<ha_mode>" + where: + os = OpenStack (No other VIM choices currently available) + controller is one of ( nosdn | odl_l2 | odl_l3 | onos ) + nfv_feature is one or more of ( ovs | kvm | sfc | bgpvpn | nofeature ) + If several features are pertinent then use the underscore + character '_' to separate each feature (e.g. ovs_kvm) + 'nofeature' indicates no NFV feature is deployed + ha_mode is one of ( ha | noha ) + + **NOTE:** Not all possible combinations of "DEPLOY_SCENARIO" are + supported. The name passed in to the Functest Docker container + must match the scenario used when the actual OPNFV platform was + deployed. + +Putting all above together, when using installer 'fuel' and an invented +INSTALLER_IP of '10.20.0.2', the recommended command to create the +Functest Docker container is as follows:: + + docker run --name "FunctestContainer" -it \ -e "INSTALLER_IP=10.20.0.2" \ -e "INSTALLER_TYPE=fuel" \ -e "DEPLOY_SCENARIO=os-odl_l2-ovs_kvm-ha" \ - opnfv/functest:<TagIdentifier> /bin/bash + -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds \ + opnfv/functest /bin/bash -**NOTE:** Not all possible combinations of "DEPLOY_SCENARIO" are supported. -The scenario name passed in to the Functest Docker container must match the -scenario used with the selected installer to create the actual OPNFV platform -deployment. +After the *run* command, a new prompt appears which means that we are inside +the container and ready to move to the next step. -Finally, three additional environment variables can also be passed in to the -Functest Docker Container, using the -e "<EnvironmentVariableName>=<Value>" -mechanism. The first two of these are only relevant to Jenkins CI invoked -testing and **should not be used** when performing manual test scenarios:: +For tips on how to set up container with installer Apex, see chapter +`Apex Installer Tips`_. + +Finally, three additional environment variables can also be passed in +to the Functest Docker Container, using the -e +"<EnvironmentVariable>=<Value>" mechanism. The first two of these are +only relevant to Jenkins CI invoked testing and **should not be used** +when performing manual test scenarios:: -e "NODE_NAME=<Test POD Name>" \ -e "BUILD_TAG=<Jenkins Build Tag>" \ -e "CI_DEBUG=<DebugTraceValue>" - where: <Test POD Name> = Symbolic name of the POD where the tests are run. Visible in test results files, which are stored @@ -147,16 +180,15 @@ testing and **should not be used** when performing manual test scenarios:: been run. If not specified, then the POD name is defined as "Unknown" by default. DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS. - <Jenkins Build tag> = Symbolic name of the Jenkins Build Job. Visible in test results files, which are stored to the database. This option is only set when tests are activated under Jenkins CI control. - It enables the correlation of test results, which + It enables the correlation of test results, + which are independently pushed to the results datbase from different Jenkins jobs. DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS. - <DebugTraceValue> = "true" or "false" Default = "false", if not specified If "true" is specified, then additional debug trace @@ -165,86 +197,54 @@ testing and **should not be used** when performing manual test scenarios:: Apex Installer Tips ------------------- -Some specific tips are useful for the Apex Installer case. If not using Apex -Installer; ignore this section. +Some specific tips are useful for the Apex Installer case. If not using +Apex Installer; ignore this section. - #. The "INSTALLER_IP" environment variable should be set equal to the IP - address of the so-called "Instack/undercloud Virtual Machine". +In case of Triple-O based installer (like Apex) the docker container +needs to connect to the installer VM, so it is then required that some +known SSH keys are present in docker container. Since the Jumphost root +SSH keys are already known, easiest way is to use those using the +'Bind mount' method. See below for sample parameter:: - In the Jumphost, execute the following command and note the returned - IP address:: + -v /root/.ssh/id_rsa:/root/.ssh/id_rsa - sudo virsh domifaddr undercloud | grep -Eo "[0-9.]+{4}" - - NOTE: In releases prior to Colorado, the name 'instack' was - used. From Colorado onward, the name 'undercloud' is used. - If in doubt, then execute -from the Jumphost- the command - "virsh list" to see which name is in use for the Installer - Virtual Machine. - - You can now enter the <Specific IP Address> as learned in the above step in the - -e option specification:: - - -e "INSTALLER_IP=<Specific IP Address>" + NOTE: You need the "sudo" when creating the container to access root + users ssh credentials even the docker command itself might not + require that. - #. If you want to 'Bind mount' a local Openstack credentials file ("overcloudrc") - to the Docker container, then you may need to first pre-copy that file from the - 'Instack/Undercloud VM' to the Jump host. +HINT! In case of Triple-O installers you can find value for the +INSTALLER_IP parameter by executing command and note the returned IP +address:: - As before, in the Jumphost, execute the following command and note the - returned IP address:: + inst=$(sudo virsh list | grep -iEo "undercloud|instack") + sudo virsh domifaddr ${inst} - sudo virsh domifaddr undercloud | grep -Eo "[0-9.]+{4}" + NOTE: In releases prior to Colorado, the name 'instack' was + used. Currently the name 'undercloud' is used. - Using the <Specific IP Address> just learned above, execute the following - shell commands **in the Jumphost**, before issuing the 'docker run ...' command - invocation:: +You can copy the credentials file from the "stack" users home directory +in installer VM to Jumphost. Please check the correct IP from the +command above. In the example below we are using invented IP address +"192.168.122.89":: - scp stack@<Specific IP Address>:overcloudrc . - sed -i 's/export no_proxy/#export no_proxy/' overcloudrc - # The above 'sed' command is needed *only* in cases where - # the Jumphost is operating behind a http proxy. - # See the 'Proxy Support' section later on in this document + scp stack@192.168.122.89:overcloudrc . - NOTE: There are two Openstack credential files present in the - Instack/Undercloud VM: 'overcloudrc' and 'stackrc'. - Don't mix these up! The file 'stackrc' is intended for use with - 'Triple O Undercloud'; only. The SUT always requires OpenStack - Overcloud Credentials. +Here is an example of the full docker command invocation for an Apex +installed system, using latest Functest docker container, for +illustration purposes:: - The file located at Jumphost path: '~/overcloudrc' is now 'Bind mounted' - to the Docker path '/home/opnfv/functest/conf/openstack.creds' - by specifying a **-v** option:: - - -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds - - in the argument list of the 'docker run ...' command invocation. In the - Apex installer case, the Openstack Credential file has the name - 'overcloudrc' and is located in the home directory of the 'stack' user - ( '/home/stack/' or '~/'] ) in the 'Instack/Undercloud VM'. - - #. In order that the docker container can access the Instack/Undercloud VM, - even with 'stack' user, the SSH keys of the Jumphost root user **must be** - 'Bind mounted' to the docker container by the following **-v** option in - the 'docker run ...' command invocation:: - - -v /root/.ssh/id_rsa:/root/.ssh/id_rsa - - #. Here is an example of the docker command invocation for an Apex installed - system, using latest Funtest docker container, for illustration purposes:: - - docker run -it --name "ApexFuncTstODL" \ - -e "INSTALLER_IP=<Specific IP Address>" \ - -e "INSTALLER_TYPE=apex" \ - -e "DEPLOY_SCENARIO=os-odl_l2-nofeature-ha" \ - -v /root/.ssh/id_rsa:/root/.ssh/id_rsa \ - -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds \ - opnfv/functest /bin/bash + sudo docker run -it --name "ApexFuncTestODL" \ + -e "INSTALLER_IP=192.168.122.89" \ + -e "INSTALLER_TYPE=apex" \ + -e "DEPLOY_SCENARIO=os-odl_l2-nofeature-ha" \ + -v /root/.ssh/id_rsa:/root/.ssh/id_rsa \ + -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds \ + opnfv/functest /bin/bash Functest docker container directory structure --------------------------------------------- -Inside the Functest docker container, the following directory structure should -now be in place:: +Inside the Functest docker container, the following directory structure +should now be in place:: `-- home `-- opnfv @@ -258,30 +258,27 @@ now be in place:: |-- functest |-- odl_integration |-- onos - |-- ovno |-- promise |-- rally |-- releng `-- vims-test - (The sub-directory 'ovno' holds SDN controller functional tests - for the OpenContrail SDN Controller, which should be available - for Colorado release) - Underneath the '/home/opnfv/' directory, the Functest docker container includes two main directories: - * The **functest** directory stores configuration files (e.g. the OpenStack - creds are stored in path '/home/opnfv/functest/conf/openstack.creds'), the - **data** directory stores a 'cirros' test image used in some functional - tests and the **results** directory stores some temporary result log files + * The **functest** directory stores configuration files (e.g. the + OpenStack creds are stored in path '/home/opnfv/functest/conf/openstack.creds'), + the **data** directory stores a 'cirros' test image used in some + functional tests and the **results** directory stores some temporary + result log files * The **repos** directory holds various repositories. The directory '/home/opnfv/repos/functest' is used to prepare the needed Functest - environment and to run the tests. The other repository directories are - used for the installation of the needed tooling (e.g. rally) or for the - retrieval of feature projects scenarios (e.g. promise) + environment and to run the tests. The other repository directories + are used for the installation of the needed tooling (e.g. rally) or + for the retrieval of feature projects scenarios (e.g. promise) -The structure under the **functest** repository can be described as follows:: +The structure under the **functest** repository can be described as +follows:: . |-- INFO |-- LICENSE @@ -337,44 +334,39 @@ The structure under the **functest** repository can be described as follows:: We may distinguish 7 different directories: - * **ci**: This directory contains test structure defintion files - (e.g <filename>.yaml) and bash shell/python scripts used to configure and - execute Functional tests. The test execution script can be executed under - the control of Jenkins CI jobs. - * **cli**: This directory holds the python based Functest CLI utility source - code, which is based on the Python 'click' framework. - * **commons**: This directory is dedicated for storage of traffic profile or - any other test inputs that could be reused by any test project. - * **docker**: This directory includes the needed files and tools to build the - Funtest Docker container image. - * **docs**: This directory includes documentation: Release Notes, User Guide, - Configuration Guide and Developer Guide. Test results are also located in - a sub--directory called 'results'. - * **testcases**: This directory includes the scripts required by Functest - internal test cases and other feature projects test cases. - * **utils**: this directory holds Python source code for some general purpose - helper utilities, which testers can also re-use in their own test code. - See for an example the Openstack helper utility: 'openstack_utils.py'. - -After the *run* command, a new prompt appears which means that we are inside -the container and ready to move to the next step. + * **ci**: This directory contains test structure definition files + (e.g <filename>.yaml) and bash shell/python scripts used to + configure and execute Functional tests. The test execution script + can be executed under the control of Jenkins CI jobs. + * **cli**: This directory holds the python based Functest CLI utility + source code, which is based on the Python 'click' framework. + * **commons**: This directory is dedicated for storage of traffic + profile or any other test inputs that could be reused by any test + project. + * **docker**: This directory includes the needed files and tools to + build the Funtest Docker container image. + * **docs**: This directory includes documentation: Release Notes, + User Guide, Configuration Guide and Developer Guide. Test results + are also located in a sub--directory called 'results'. + * **testcases**: This directory includes the scripts required by + Functest internal test cases and other feature projects test cases. + * **utils**: this directory holds Python source code for some general + purpose helper utilities, which testers can also re-use in their + own test code. See for an example the Openstack helper utility: + 'openstack_utils.py'. Useful Docker commands ---------------------- -When typing **exit** in the container prompt, this will cause exiting the -container and probably stopping it. When stopping a running Docker container -all the changes will be lost, there is a keyboard shortcut to quit the -container without stopping it: CTRL+P+Q. To reconnect to the running container -**DO NOT** use the *run* command again (since it will create a new container), -use the *exec* command instead:: - - docker ps <copy the container ID> docker exec -ti \ - <CONTAINER_ID> /bin/bash +When typing **exit** in the container prompt, this will cause exiting +the container and probably stopping it. When stopping a running Docker +container all the changes will be lost, there is a keyboard shortcut +to quit the container without stopping it: <CTRL>-P + <CTRL>-Q. To +reconnect to the running container **DO NOT** use the *run* command +again (since it will create a new container), use the *exec* or *attach* +command instead:: -or simply:: - - docker exec -ti \ - $(docker ps|grep functest|awk '{print $1}') /bin/bash + docker ps # <check the container ID from the output> + docker exec -ti <CONTAINER_ID> /bin/bash There are other useful Docker commands that might be needed to manage possible issues with the containers. @@ -387,6 +379,14 @@ List all the containers including the stopped ones:: docker ps -a +Start a stopped container named "FunTest":: + + docker start FunTest + +Attach to a running container named "StrikeTwo":: + + docker attach StrikeTwo + It is useful sometimes to remove a container if there are some problems:: docker rm <CONTAINER_ID> @@ -396,43 +396,39 @@ destroy it:: docker -f rm <CONTAINER_ID> -The Docker image is called **opnfv/functest** and it is stored in the public -Docker registry under the OPNFV account: dockerhub_. The are many different -tags that have been created automatically by the CI mechanisms, but the one -that this document refers to is **brahmaputra.1.0**. Pulling other tags might -cause some problems while running the tests. - Check the Docker documentation dockerdocs_ for more information. Preparing the Functest environment ---------------------------------- -Once the Functest docker container is up and running, the required Functest -environment needs to be prepared. A custom built **functest** CLI utility is -availabe to perform the needed environment preparation action. Once the -enviroment is prepared, the **functest** CLI utility can be used to run -different functional tests. The usage of the **functest** CLI utility to run -tests is described further in the Functest User Guide `OPNFV_FuncTestUserGuide`_ - -Prior to commencing the Functest environment preparation, we can check the -initial status of the environment. Issue the **functest env status** command at -the prompt:: +Once the Functest docker container is up and running, the required +Functest environment needs to be prepared. A custom built **functest** +CLI utility is available to perform the needed environment preparation +action. Once the environment is prepared, the **functest** CLI utility +can be used to run different functional tests. The usage of the +**functest** CLI utility to run tests is described further in the +Functest User Guide `OPNFV_FuncTestUserGuide`_ + +Prior to commencing the Functest environment preparation, we can check +the initial status of the environment. Issue the **functest env status** +command at the prompt:: functest env status Functest environment is not installed. - Note: When the Funtest environment is prepared, the command will + Note: When the Functest environment is prepared, the command will return the status: "Functest environment ready to run tests." -To prepare the Functest docker container for test case execution, issue the -**functest env prepare** command at the prompt:: +To prepare the Functest docker container for test case execution, issue +the **functest env prepare** command at the prompt:: functest env prepare -This script will make sure that the requirements to run the tests are met and -will install the needed libraries and tools by all Functest test cases. It -should be run only once every time the Functest docker container is started -from scratch. If you try to run this command, on an already prepared -enviroment, you will be prompted whether you really want to continue or not:: +This script will make sure that the requirements to run the tests are +met and will install the needed libraries and tools by all Functest +test cases. It should be run only once every time the Functest docker +container is started from scratch. If you try to run this command, on +an already prepared enviroment, you will be prompted whether you really +want to continue or not:: functest env prepare It seems that the environment has been already prepared. @@ -442,8 +438,9 @@ enviroment, you will be prompted whether you really want to continue or not:: environment preparation) -To list some basic information about an already prepared Functest docker -container environment, issue the **functest env show** at the prompt:: +To list some basic information about an already prepared Functest +docker container environment, issue the **functest env show** at the +prompt:: functest env show +======================================================+ @@ -467,8 +464,8 @@ container environment, issue the **functest env show** at the prompt:: - here = "192.168.122.89" SCENARIO: Displays the DEPLOY_SCENARIO value - here = "os-odl_l2-nofeature-ha" - POD: Displays the value pass in NODE_NAME - - here = "loclahost" + POD: Displays the value passed in NODE_NAME + - here = "localhost" GIT BRANCH: Displays the git branch of the OPNFV Functest project repository included in the Functest Docker Container. @@ -485,8 +482,7 @@ container environment, issue the **functest env show** at the prompt:: NOTE: In Jenkins CI runs, an additional item "BUILD TAG" would also be listed. The valaue is set by Jenkins CI. -Finally, the **functest** CLI has a basic 'help' system with so called -**--help** options: +Finally, the **functest** CLI has a **--help** options: Some examples:: @@ -513,65 +509,28 @@ Some examples:: show Shows information about the current... status Checks if the Functest environment is ready... -Focus on the OpenStack credentials +Checking Openstack and credentials ---------------------------------- -The OpenStack credentials are needed to run the tests against the VIM. -There are 3 ways to provide them to Functest: - - * using the -v option when running the Docker container - * create an empty file in '/home/opnfv/functest/conf/openstack.creds' and - paste the credentials into it. (Consult your installer guide to know from - where you can retrieve credential files, which are set-up in the Openstack - installation of the SUT) - * automatically retrieved using the following script:: - - $repos_dir/releng/utils/fetch_os_creds.sh \ - -d /home/opnfv/functest/conf/openstack.creds \ - -i fuel \ - -a 10.20.0.2" - - (-d specifies the full destination path where to place the - copied Openstack credential file - -i specifies the INSTALLER_TYPE - -a specifies the INSTALLER_IP - If the installer is of type "fuel" and a Virtualized - deployment is used, then this should be indicated by - adding an option '-v'. The -v option takes no arguments. - It enables some needed special handling in the script.) - - Note: If you omit the -d <full destination path> option in - the command invocation, then the script will create the - credential file with name 'opnfv-openrc.sh' in directory - '/home/opnfv'. In that case, you need to copy/edit the file - into the correct target path: - '/home/opnfv/functest/conf/openstack.creds'. - -**Warning** If you are using the Joid installer, the 'fetch_os_cred-sh' shell -script **should not be used**. Use instead, the **-v** optin to Bind Mount a -suitably prepared local copy of the Openstack credentials for usage by the Functest -docker container - -Once the credentials are there, they should be sourced **before** running the -tests:: +It is recommended and fairly straightforward to check that Openstack +and credentials are working as expected. - source /home/opnfv/functest/conf/openstack.creds - -or simply using the environment variable **creds**:: +Once the credentials are there inside the container, they should be +sourced before running any Openstack commands:: - . $creds + source /home/opnfv/functest/conf/openstack.creds -After this, try to run any OpenStack command to see if you get any output, for -instance:: +After this, try to run any OpenStack command to see if you get any +output, for instance:: openstack user list -This will return a list of the actual users in the OpenStack deployment. In any -other case, check that the credentials are sourced:: +This will return a list of the actual users in the OpenStack +deployment. In any other case, check that the credentials are sourced:: env|grep OS_ -This command must show a set of environment variables starting with *OS_*, for -example:: +This command must show a set of environment variables starting with +*OS_*, for example:: OS_REGION_NAME=RegionOne OS_DEFAULT_DOMAIN=default @@ -584,23 +543,24 @@ example:: OS_ENDPOINT_TYPE=internalURL OS_NO_CACHE=true -If the OpenStack command still does not show anything or complains about -connectivity issues, it could be due to an incorrect url given to the -OS_AUTH_URL environment variable. Check the deployment settings. +If the OpenStack command still does not show anything or complains +about connectivity issues, it could be due to an incorrect url given to +the OS_AUTH_URL environment variable. Check the deployment settings. SSL Support ----------- -If you need to connect to a server that is TLS-enabled (the auth URL begins -with ‘https’) and it uses a certificate from a private CA or a self-signed -certificate, then you will need to specify the path to an appropriate CA -certificate to use, to validate the server certificate with the environment -variable OS_CACERT:: +If you need to connect to a server that is TLS-enabled (the auth URL +begins with "https") and it uses a certificate from a private CA or a +self-signed certificate, then you will need to specify the path to an +appropriate CA certificate to use, to validate the server certificate +with the environment variable OS_CACERT:: echo $OS_CACERT /etc/ssl/certs/ca.crt -However, this certificate does not exist in the container by default. It has -to be copied manually from the OpenStack deployment. This can be done in 2 ways: +However, this certificate does not exist in the container by default. +It has to be copied manually from the OpenStack deployment. This can be +done in 2 ways: #. Create manually that file and copy the contents from the OpenStack controller. @@ -609,33 +569,38 @@ to be copied manually from the OpenStack deployment. This can be done in 2 ways: -v <path_to_your_cert_file>:/etc/ssl/certs/ca.cert -You might need to export OS_CACERT environment variable inside the container:: +You might need to export OS_CACERT environment variable inside the +container:: export OS_CACERT=/etc/ssl/certs/ca.crt -Certificate verification can be turned off using OS_INSECURE=true. For example, -Fuel uses self-signed cacerts by default, so an pre step would be:: +Certificate verification can be turned off using OS_INSECURE=true. For +example, Fuel uses self-signed cacerts by default, so an pre step would +be:: export OS_INSECURE=true Proxy support ------------- -If your Jumphost node is operating behind a http proxy, then there are 2 places -where some special actions may be needed to make operations succeed: - - #. Initial installation of docker engine First, try following the official - Docker documentation for Proxy_ settings. Some issues were experienced on - CentOS 7 based Jumphost. Some tips are documented in section: - `Docker Installation on CentOS 7 behind http proxy`_ below. - - #. Execution of the Functest environment preparation inside the created - docker container Functest needs internet access to download some resources - for some test cases. For example to install the Rally environment. This might - not work properly if the Jumphost is running through a http Proxy. - -If that is the case, make sure the resolv.conf and the needed http_proxy and -https_proxy environment variables, as well as the 'no_proxy' environment -variable are set correctly:: +If your Jumphost node is operating behind a http proxy, then there are +2 places where some special actions may be needed to make operations +succeed: + + #. Initial installation of docker engine First, try following the + official Docker documentation for Proxy settings. Some issues were + experienced on CentOS 7 based Jumphost. Some tips are documented + in section: `Docker Installation on CentOS behind http proxy`_ + below. + + #. Execution of the Functest environment preparation inside the + created docker container Functest needs internet access to + download some resources for some test cases. This might not + work properly if the Jumphost is connecting to internet + through a http Proxy. + +If that is the case, make sure the resolv.conf and the needed +http_proxy and https_proxy environment variables, as well as the +'no_proxy' environment variable are set correctly:: # Make double sure that the 'no_proxy=...' line in the # 'openstack.creds' file is commented out first. Otherwise, the @@ -643,22 +608,20 @@ variable are set correctly:: # be ovewrwritten, each time the command # 'source ~/functest/conf/openstack.creds' is issued. - sed -i 's/export no_proxy/#export no_proxy/' \ - ~/functest/conf/openstack.creds - - source ~/functest/conf/openstack.creds + cd ~/functest/conf/ + sed -i 's/export no_proxy/#export no_proxy/' openstack.creds + source ./openstack.creds # Next calculate some IP addresses for which http_proxy # usage should be excluded: - publicURL_IP=$(echo $OS_AUTH_URL| \ - grep -Eo "([0-9]+\.){3}[0-9]+") + publicURL_IP=$(echo $OS_AUTH_URL | grep -Eo "([0-9]+\.){3}[0-9]+") adminURL_IP=$(openstack catalog show identity | \ grep adminURL | grep -Eo "([0-9]+\.){3}[0-9]+") export http_proxy="<your http proxy settings>" - export https_proxy="<your httpsproxy settings>" + export https_proxy="<your https proxy settings>" export no_proxy="127.0.0.1,localhost,$publicURL_IP,$adminURL_IP" # Ensure that "git" uses the http_proxy @@ -667,11 +630,11 @@ variable are set correctly:: git config --global http.proxy <Your http proxy settings> Validation check: Before running **'functest env prepare'** CLI command, -make sure you can reach http and https sites from inside the Functest docker -container. +make sure you can reach http and https sites from inside the Functest +docker container. -For example, try to use the **nc** command from inside the functest docker -container:: +For example, try to use the **nc** command from inside the functest +docker container:: nc -v google.com 80 Connection to google.com 80 port [tcp/http] succeeded! @@ -679,66 +642,50 @@ container:: nc -v google.com 443 Connection to google.com 443 port [tcp/https] succeeded! -Note: In a Jumphost node based on the CentOS 7, enviroment, it was observed that -the **nc** commands did not function as described in the section above. You can -however try using the **curl** command instead, if you encounter any issues -with the **nc** command:: +Note: In a Jumphost node based on the CentOS family OS, the **nc** +commands might not work. You can use the **curl** command instead. curl http://www.google.com:80 - <HTML><HEAD><meta http-equiv="content-type" - content="text/html;charset=utf-8"> - <TITLE>302 Moved</TITLE> - </HEAD> - <BODY> - <H1>302 Moved</H1> - : - : + . + . </BODY></HTML> curl https://www.google.com:443 - <HTML><HEAD><meta http-equiv="content-type" - content="text/html;charset=utf-8"> - <TITLE>302 Moved</TITLE> - </HEAD> - <BODY> - <H1>302 Moved</H1> - : - : + . + . </BODY></HTML> - (Even Google complained the URL used, it proves the http and https - protocols are working correctly through the http / https proxy.) + (Ignore the content. If command returns a valid HTML page, it proves + the connection.) -Docker Installation on CentOS 7 behind http proxy -------------------------------------------------- -There are good instructions in [`InstallDockerCentOS7`_] for the installation -of **docker** on CentOS 7. However, if your Jumphost is behind a http proxy, -then the following steps are needed **before** following the instructions in -the above reference:: +Docker Installation on CentOS behind http proxy +----------------------------------------------- +This section is applicable for CentOS family OS on Jumphost which +itself is behind a proxy server. In that case, the instructions below +should be followed **before** installing the docker engine:: 1) # Make a directory '/etc/systemd/system/docker.service.d' # if it does not exist sudo mkdir /etc/systemd/system/docker.service.d - # Create a file called 'env.conf' in that directory with + 2) # Create a file called 'env.conf' in that directory with # the following contents: [Service] EnvironmentFile=-/etc/sysconfig/docker - 2) # Set up a file called 'docker' in directory '/etc/sysconfig' + 3) # Set up a file called 'docker' in directory '/etc/sysconfig' # with the following contents: - HTTP_PROXY="<Your http proxy settings>" HTTPS_PROXY="<Your https proxy settings>" http_proxy="${HTTP_PROXY}" https_proxy="${HTTPS_PROXY}" - 3) # Reload the daemon + 4) # Reload the daemon systemctl daemon-reload - 4) # Sanity check - check the following docker settings: + 5) # Sanity check - check the following docker settings: systemctl show docker | grep -i env Expected result: @@ -746,14 +693,14 @@ the above reference:: EnvironmentFile=/etc/sysconfig/docker (ignore_errors=yes) DropInPaths=/etc/systemd/system/docker.service.d/env.conf -Now follow the instructions in [`InstallDockerCentOS7`_] to download and -install the **docker-engine**. The instructions conclude with a "test pull" -of a sample "Hello World" docker container. This should now work with the -above pre-requisite actions. +Now follow the instructions in [`InstallDockerCentOS`_] to download +and install the **docker-engine**. The instructions conclude with a +"test pull" of a sample "Hello World" docker container. This should now +work with the above pre-requisite actions. .. _dockerdocs: https://docs.docker.com/ .. _dockerhub: https://hub.docker.com/r/opnfv/functest/ .. _Proxy: https://docs.docker.com/engine/admin/systemd/#http-proxy .. _FunctestDockerTags: https://hub.docker.com/r/opnfv/functest/tags/ -.. _InstallDockerCentOS7: https://docs.docker.com/engine/installation/linux/centos/ +.. _InstallDockerCentOS: https://docs.docker.com/engine/installation/linux/centos/ .. _OPNFV_FuncTestUserGuide: http://artifacts.opnfv.org/functest/docs/userguide/index.html diff --git a/docs/configguide/index.rst b/docs/configguide/index.rst index 7c8113e16..6b6c62a1b 100644 --- a/docs/configguide/index.rst +++ b/docs/configguide/index.rst @@ -12,7 +12,9 @@ OPNFV FUNCTEST Configuration Guide Introduction ============ This document describes how to install and configure Functest in OPNFV. -The Functest CLI is utilized during the Functest environment preparation step. +The Functest CLI is utilized during the Functest environment preparation +step. The given example commands should work in both virtual and bare +metal cases alike. High level architecture ----------------------- @@ -20,69 +22,66 @@ High level architecture The high level architecture of Functest within OPNFV can be described as follows:: - CIMC/Lights+out management Admin Private Public Storage - PXE - + - + + IP_PRIV/24 | | - | | + + | - | | | IP_PUB/24 | - | +----------------------------+ | | + | - | | | | | | | - +-----+ Jumphost | | | | | - | | +----------+ | | | - | | | | | | | - | | +--------------------+ | | | | | - | | | | | | | | | - | | | Tools | +--------------------+ | | - | | | - Rally | | | | | | - | | | - Robot | | | | | | - | | | - TestON | | | | | | - | | | | | | | | | - | | | Testcases | | | | | | - | | | - VIM | | | | | | - | | | -- vPing | | | | | | - | | | -- vPing_userdata | | | | | | - | | | -- Tempest | | | | | | - | | | -- Rally | | | | | | - | | | - Controller | | | | | | - | | | -- odl | | | | | | - | | | -- onos | | | | | | - | | | -- opencontrail | | | | | | - | | | | | | | | | - | | | Features | | | | | | - | | | - vIMS | | | | | | - | | | | | | | | | - | | +--------------------+ | | | | | - | | +------------------------------+ | - | | Functest Docker | | | | | - | | +---------------------------------------+ - | | | | | | | - | +----------------------------+ | | | | - | | | | | - | +----------------+ | | | | - | | 1 | | | | | - +----+ +--------------+-+ | | | | - | | | 2 | | | | | - | | | +--------------+-+ | | | | - | | | | 3 | | | | | - | | | | +--------------+-+ | | | | - | | | | | 4 | | | | | - | +-+ | | +--------------+-+ | | | | - | | | | | 5 +---------------+ | | | - | +-+ | | nodes for | | | | | - | | | | deploying +-------------------------+ | | - | +-+ | OPNFV | | | | | - | | | +-----------------------------------+ | - | +-+ SUT | | | | | - | | +--------------------------------------------+ - | +----------------+ | | | | - | | | | | - + + + + + + CIMC/Lights+out management Admin Private Public Storage + PXE + + + + + + + | | | | | + | +----------------------------+ | | | | + | | | | | | | + +-----+ Jumphost | | | | | + | | +--------+ | | | + | | | | | | | + | | +--------------------+ | | | | | + | | | | | | | | | + | | | Tools | +----------------+ | | + | | | - Rally | | | | | | + | | | - Robot | | | | | | + | | | - TestON | | | | | | + | | | | | | | | | + | | | Testcases | | | | | | + | | | - VIM | | | | | | + | | | -- vPing | | | | | | + | | | -- vPing_userdata | | | | | | + | | | -- Tempest | | | | | | + | | | -- Rally | | | | | | + | | | - Controller | | | | | | + | | | -- odl | | | | | | + | | | -- onos | | | | | | + | | | | | | | | | + | | | Features | | | | | | + | | | - vIMS | | | | | | + | | | | | | | | | + | | +--------------------+ | | | | | + | | +-------------------------+ | + | | Functest Docker | | | | | + | | +---------------------------------+ + | | | | | | | + | +----------------------------+ | | | | + | | | | | + | +----------------+ | | | | + | | 1 | | | | | + +----+ +--------------+-+ | | | | + | | | 2 | | | | | + | | | +--------------+-+ | | | | + | | | | 3 | | | | | + | | | | +--------------+-+ | | | | + | | | | | 4 | | | | | + | +-+ | | +--------------+-+ | | | | + | | | | | 5 +-------------+ | | | + | +-+ | | nodes for | | | | | + | | | | deploying +---------------------+ | | + | +-+ | OPNFV | | | | | + | | | +------------------------------+ | + | +-+ SUT | | | | | + | | +--------------------------------------+ + | +----------------+ | | | | + | | | | | + + + + + + SUT = System Under Test -All the libraries and dependencies needed by all of the Functest tools are -pre-installed into the Docker image. This allows running Functest on any -platform on any Operating System. +All the libraries and dependencies needed by all of the Functest tools +are pre-installed into the Docker image. This allows running Functest +on any platform on any Operating System. The automated mechanisms inside the Functest Docker container will: @@ -93,18 +92,18 @@ The automated mechanisms inside the Functest Docker container will: This Docker image can be integrated into CI or deployed independently. -Please note that the Functest Docker container has been designed for OPNFV, -however, it would be possible to adapt it to any VIM+controller environment, -since most of the test cases are integrated from upstream communities. - -The test cases are described in the Functest User Guide `[2]`_ +Please note that the Functest Docker container has been designed for +OPNFV, however, it would be possible to adapt it to any VIM + controller +environment, since most of the test cases are integrated from upstream +communities. +The functional test cases are described in the Functest User Guide `[2]`_ Prerequisites ============= -The OPNFV deployment is out of the scope of this document but it can be found -in `[4]`_. The OPNFV platform is considered as the System Under Test (SUT) in -this document. +The OPNFV deployment is out of the scope of this document but it can be +found in `[4]`_. The OPNFV platform is considered as the System Under +Test (SUT) in this document. Several prerequisites are needed for Functest: @@ -115,24 +114,44 @@ Several prerequisites are needed for Functest: #. Connectivity from the Jumphost to the SUT public/external network #. Connectivity from the Jumphost to the SUT admin/management network -NOTE: **Jumphost** refers to any server which meets the previous requirements. -Normally it is the same server from where the OPNFV deployment has been -triggered previously. +WARNING: Connectivity from Jumphost is essential and it is of paramount +importance to make sure it is working before even considering to install +and run Functest. Make also sure you understand how your networking is +designed to work. + +NOTE: **Jumphost** refers to any server which meets the previous +requirements. Normally it is the same server from where the OPNFV +deployment has been triggered previously. + +NOTE: If your Jumphost is operating behind a company http proxy and/or +Firewall, please consult first the section `Proxy Support`_, towards +the end of this document. The section details some tips/tricks which +*may* be of help in a proxified environment. Docker installation ------------------- -.. _Ubuntu: https://docs.docker.com/installation/ubuntulinux/ -.. _RHEL: https://docs.docker.com/installation/rhel/ -.. _CentOS: https://docs.docker.com/engine/installation/linux/centos/ +Docker installation and configuration is only needed to be done once +through the life cycle of Jumphost. + +If your Jumphost is based on Ubuntu, RHEL or CentOS linux, please +consult the references below for more detailed instructions. The +commands below are offered as a short reference. -*Tip:* If your Jumphost is operating behind a company http proxy and/or -Firewall, please consult first the section `Proxy Support`_, towards the end -of this document. The section details some tips/tricks which *may* be of help -in a proxified environment. +*Tip:* For running docker containers behind the proxy, you need first +some extra configuration which is described in section +`Docker Installation on CentOS behind http proxy`_. You should follow +that section before installing the docker engine. -Log on to your Jumphost and install the Docker Engine (e.g. for Ubuntu):: +Docker installation needs to be done as root user. You may use other +userid's to create and run the actual containers later if so desired. +Log on to your Jumphost as root user and install the Docker Engine +(e.g. for CentOS family):: curl -sSL https://get.docker.com/ | sh + systemctl start docker + + *Tip:* If you are working through proxy, please set the https_proxy + environment variable first before executing the curl command. Add your user to docker group to be able to run commands without sudo:: @@ -142,51 +161,43 @@ A reconnect is needed. There are 2 ways for this: #. Re-login to your account #. su - <username> -If your Jumphost is based on Red Hat Enterprise Linux, or CentOS 7 linux, -please consult the references below. - References - Installing Docker Engine on different Linux Operating Systems: - * Ubuntu_ * RHEL_ * CentOS_ +.. _Ubuntu: https://docs.docker.com/engine/installation/linux/ubuntulinux/ +.. _RHEL: https://docs.docker.com/engine/installation/linux/rhel/ +.. _CentOS: https://docs.docker.com/engine/installation/linux/centos/ + Public/External network on SUT ------------------------------ Some of the tests against the VIM (Virtual Infrastructure Manager) need -connectivity through an existing public/external network in order to succeed. -This is needed, for example, to create floating IPs to access VM instances -through the public/external network (i.e. from the Docker container). - -By default, the four OPNFV installers provide a fresh installation with a -public/external network created along with a router. Make sure that the -public/external subnet is reachable from the Jumphost. - -*Hint:* For the given OPNFV Installer in use, the IP sub-net address used for -the public/external network is usually a planning item and should thus be known. -Consult the OPNFV Configuration guide `[4]`_, and ensure you can reach each -node in the SUT, from the Jumphost using the 'ping' command using the -respective IP address on the public/external network for each node in the SUT. -(The details of how to determine the needed IP addresses for each node in the -SUT may vary according to the used installer and are therefore ommitted here.) - -Connectivity to OPNFV admin/management network ----------------------------------------------- -Some of the Functest tools need to have access to the OpenStack admin/management -network of the controllers `[1]`_. - -For this reason, an interface shall be configured in the Jumphost in the OpenStack admin/management network range. - -For example, if the admin/management network is using VLAN 300 and subnet 192.168.1.0/24 and assuming that eth1 is the -physical interface with access to that subnet:: - - ip link add name eth1.300 link eth1 type vlan id 300 - ip link set eth1.300 up - ip addr add 192.168.1.66/24 dev eth1.300 - -This is just an example about how to configure an interface with vlan, but it might differ depending on the deployment -settings on each installer. Check the corresponding installer instructions for more precise instructions. - +connectivity through an existing public/external network in order to +succeed. This is needed, for example, to create floating IPs to access +VM instances through the public/external network (i.e. from the Docker +container). + +By default, the four OPNFV installers provide a fresh installation with +a public/external network created along with a router. Make sure that +the public/external subnet is reachable from the Jumphost. + +*Hint:* For the given OPNFV Installer in use, the IP sub-net address +used for the public/external network is usually a planning item and +should thus be known. Consult the OPNFV Configuration guide `[4]`_, and +ensure you can reach each node in the SUT, from the Jumphost using the +'ping' command using the respective IP address on the public/external +network for each node in the SUT. The details of how to determine the +needed IP addresses for each node in the SUT may vary according to the +used installer and are therefore ommitted here. + +Connectivity to Admin/Management network on SUT +----------------------------------------------- +Some of the Functest tools need to have access to the OpenStack +admin/management network of the controllers `[1]`_. + +For this reason, check the connectivity from the Jumphost to all the +controllers in cluster in the OpenStack admin/management network range. Installation and configuration ============================== @@ -269,7 +280,7 @@ OPNFV main site: opnfvmain_. OPNFV functional test page: opnfvfunctest_. -IRC support channel: #opnfv-testperf +IRC support channel: #opnfv-functest .. _opnfvmain: http://www.opnfv.org .. _opnfvfunctest: https://wiki.opnfv.org/opnfv_functional_testing diff --git a/docs/release-notes/functest-release.rst b/docs/release-notes/functest-release.rst index 65480a23a..9fbc35634 100644 --- a/docs/release-notes/functest-release.rst +++ b/docs/release-notes/functest-release.rst @@ -10,10 +10,10 @@ This document describes the release note of Functest project. License ======= -OPNFV Brahmaputra release note for Functest Docs +OPNFV Colorado release note for Functest Docs (c) by Morgan Richomme (Orange) -OPNFV Brahmaputra release note for Functest Docs +OPNFV Colorado release note for Functest Docs are licensed under a Creative Commons Attribution 4.0 International License. You should have received a copy of the license along with this. If not, see <http://creativecommons.org/licenses/by/4.0/>. @@ -25,14 +25,11 @@ Version history | **Date** | **Ver.** | **Author** | **Comment** | | | | | | +------------+----------+------------------+------------------------+ -| 2016-02-25 | 1.0.0 | Morgan Richomme | Functest for B release | -| | | (Orange) | | -+------------+----------+------------------+------------------------+ -| 2016-04-27 | 3.0.0 | Morgan Richomme | Add scenarios | +| 2016-08-17 | 1.0.0 | Morgan Richomme | Functest for C release | | | | (Orange) | | +------------+----------+------------------+------------------------+ -OPNFV Brahmaputra Release +OPNFV Colorado Release ========================= Functest deals with functional testing of the OPNFV solution. @@ -41,19 +38,36 @@ other OPNFV projects and other upstream communities. The internal test cases are: + * healthcheck * vPing ssh * vPing userdata - * Tempest - * Rally - * vIMS + * Tempest Smoke Serial + * Rally Sanity * ODL + * Tempest full parallel + * Rally full + * vIMS The OPNFV projects integrated into Functest framework for automation are: - * Promise + * bgpvpn + * Copper * Doctor + * Domino + * Moon + * Multisite * ONOSFW - * bgpvpn + * ONOS-sfc + * ODL-sfc + * Parser + * Promise + * Security scan + +The validation of a scenario requires a subset of these tests depending +on the installer and the scenario. + +The 3 last internal test cases (tempest full parallel, Rally full and +vIMS) are not considered for scenario validation. Release Data ============ @@ -62,16 +76,16 @@ Release Data | **Project** | functest | | | | +--------------------------------------+--------------------------------------+ -| **Repo/tag** | brahmaputra.1.0 | +| **Repo/tag** | colorado.1.0 | | | | +--------------------------------------+--------------------------------------+ -| **Release designation** | Brahmaputra base release | +| **Release designation** | Colorado base release | | | | +--------------------------------------+--------------------------------------+ -| **Release date** | February 26 2016 | +| **Release date** | September 22 2016 | | | | +--------------------------------------+--------------------------------------+ -| **Purpose of the delivery** | Brahmaputra base release | +| **Purpose of the delivery** | Colorado base release | | | | +--------------------------------------+--------------------------------------+ @@ -81,18 +95,20 @@ Deliverables Software -------- - - The Functest Docker image + - The Functest Docker image: * TODO link * + + - The testapi Docker image: * TODO link * + Documents --------- - - Installation/configuration guide + - Installation/configuration guide: * TODO link * - - User Guide + - User Guide: * TODO link * - - Developer Guide + - Developer Guide: * TODO link * - - Test results per scenario Version change ============== @@ -100,183 +116,194 @@ Version change Feature evolution ----------------- - - support new scenarios + - refactoring of ODL functional tests (with upstream modifications) + + - refactoring of testapi (update, swagger documentation, dockerization) + + - jenkins logs improvement + + - update integration of Doctor, Promise and SDNVPN projects + + - split Tempest and rally into 2 different tests: smoke and full + + - vIMS test suite integration + + - adoption of Kibana for dashboarding + New features ------------ - - minor bug fixes (formating) + - Functest CLI to prepare and run the tests + + - creation of the healthcheck test case + + - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, multisite) + + - integration of new OPNFV feature projects (copper, domino, multisite, + moon, parser, onos-sfc, odl-sfc, security scan) - - Modification of the configuration to support vPing_userdata on ONOS scenario + - introduction of test tiers in functest framework + + - automatic reporting + + - introduction of a jenkins summary table + + - support of ARM architecture - - Use serial option in Tempest to improve success rate Scenario Matrix =============== -For Brahmaputra 3.0, Functest was succesfully tested on the following scenarios: - -+----------------+---------+---------+---------+---------+ -| Scenario | Apex | Compass | Fuel | Joid | -+================+=========+=========+=========+=========+ -| odl_l2 | X | X | X | X | -+----------------+---------+---------+---------+---------+ -| onos | | X | X | | -+----------------+---------+---------+---------+---------+ -| nosdn | | X | X | | -+----------------+---------+---------+---------+---------+ -| ovs (dpdk) | | | X | | -+----------------+---------+---------+---------+---------+ -| kvm | | | X | | -+----------------+---------+---------+---------+---------+ -| bgpvpn | X | | X | | -+----------------+---------+---------+---------+---------+ -| sfc | | | X | | -+----------------+---------+---------+---------+---------+ - -Functest defines the success criteria when having at least 4 consecutive -successful runs of a given scenario from the Continuous Integration. - -The success criteria is defined as follows: - - * vPing SSH 100% OK - * vPing userdata 100% OK - * Tempest success rate > 90% - * Rally success rate > 90% - * ODL success rate = 100% - * ONOSFW success rate = 100% - * Promise success rate = 100% - * Bgpvpn success rate = 100% - * vIMS: deployement of the orchestrator and the vIMS VNF successful - -Other scenarios are currently available but did not meet success criteria for -the release but might be added in the incremental scenario update of the -release. - -Brahmaputra limitations +For Colorado 1.0, Functest was tested on the following scenarios (if not +precised, the scenario is a na scenario): + ++---------------------+---------+---------+---------+---------+ +| Scenario | Apex | Compass | Fuel | Joid | ++=====================+=========+=========+=========+=========+ +| nosdn | X | X | X | X | ++---------------------+---------+---------+---------+---------+ +| odl_l2 | X | X | X | X | ++---------------------+---------+---------+---------+---------+ +| odl_l3 | X | X | X | | ++---------------------+---------+---------+---------+---------+ +| onos | X | X | X | X | ++---------------------+---------+---------+---------+---------+ +| ocl | | X | | | ++---------------------+---------+---------+---------+---------+ +| ovs-noha (dpdk) | | | X | | ++---------------------+---------+---------+---------+---------+ +| kvm-noha | | | X | | ++---------------------+---------+---------+---------+---------+ +| odl_l2-bgpvpn | X | | X | | ++---------------------+---------+---------+---------+---------+ +| odl_l2-sfc | | | X | | ++---------------------+---------+---------+---------+---------+ +| onos-sfc | X | X | X | X | ++---------------------+---------+---------+---------+---------+ +| odl_l2-moon | | X | | | ++---------------------+---------+---------+---------+---------+ +| multisite | | | | | ++---------------------+---------+---------+---------+---------+ +| lxd | | | | X | ++---------------------+---------+---------+---------+---------+ + +Functest defines a scenario scoring based on the sum of the unitary test +cases run in CI. +The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617 + +In Colorado, the functional tests have been sliced in different +categories: + ++----------------+-----------------------------------------------+ +| Category | Description | ++================+===============================================+ +| healthcheck | Basic OpenStack commands | ++----------------+-----------------------------------------------+ +| smoke | vPings, Tempest and rally smoke tests | ++----------------+-----------------------------------------------+ +| sdn_suites | Specific SDN feature tests | ++----------------+-----------------------------------------------+ +| features | OPNFV feature project functional test suites | ++----------------+-----------------------------------------------+ +| openstack | Advanced, long duration OpenStack tests | +| | (Tempest and Rally full suite). Each test may | +| | last several hours | ++----------------+-----------------------------------------------+ +| vnf | Complex scenarios dealing with orchestration, | +| | VNF deployment and tests (e.g. vIMS) | ++----------------+-----------------------------------------------+ + +For the scenario validation, we consider only the categories healthcheck, +smoke, sdn_suites and features. These tests are run systematically in +the CI daily loops. + +Success criteria have been defined for these test cases, they can be +PASS/FAIl or a success rate may be declared (100%, > 90%) +All the criteria, as well as the test dependencies are declared in the +ci/testcases.yaml file. + +* TODO scoring table * + +The reporting pages can be found at: + + * apex: * TODO / check branch * + * compass: * TODO / check branch * + * fuel: * TODO / check branch * + * joid: * TODO / check branch * + +Colorado limitations ----------------------- -- Fuel and Apex Tempest success rate was below 90% but above 80% on some -scenarios. Some of the error causes were identified (workers, lack of IP) +- vPing userdata and Tempest cases related to metada service excluded +from onos scenarios https://gerrit.opnfv.org/gerrit/#/c/18729/ -- vIMS failed in CI for joid/odl_l2 scenario - -- vPing userdata and vIMS excluded from onos scenario +- Tempest cases related to storage for joid scenarios +https://gerrit.opnfv.org/gerrit/#/c/17871/ - vPing_ssh and vIMS excluded from bgpvpn and kvm scenario -- None of the odl_l3 scenarios has been successful due to vPing ssh issue (ODL -bug reported https://bugs.opendaylight.org/show_bug.cgi?id=5582) - -- apex/nosdn never run (not a target scenario) but probably succesful - -- vPing SSH and vPing userdata no more run on CI since modification of bgpvpn -configuration regex. See known issues section for details + Test and installer/scenario dependencies ======================================== -It is not always possible to run all the test cases on all the scenarios. The -following table details the dependencies of the test cases per scenario. - -+----------------+-------------+-------------+-------------+-------------+ -| Test cases | Apex | Compass | Fuel | Joid | -+================+=============+=============+=============+=============+ -| vPing SSH | all | all | all | all | -+----------------+-------------+-------------+-------------+-------------+ -| vPing userdata | all | all | all | all | -+----------------+-------------+-------------+-------------+-------------+ -| Tempest | all | all | all | all | -+----------------+-------------+-------------+-------------+-------------+ -| Rally | all | all | all | all | -+----------------+-------------+-------------+-------------+-------------+ -| ODL | all ODL | all ODL | all ODL | all ODL | -+----------------+-------------+-------------+-------------+-------------+ -| ONOS | ONOS | ONOS | ONOS | ONOS | -+----------------+-------------+-------------+-------------+-------------+ -| Promise | no | no | all | all | -+----------------+-------------+-------------+-------------+-------------+ -| vIMS | all except | all except | all except | all except | -| | ONOS | ONOS | ONOS | ONOS | -+----------------+-------------+-------------+-------------+-------------+ -| Doctor | all | no | no | no | -+----------------+-------------+-------------+-------------+-------------+ -| Bgpvpn | all | no | all | no | -+----------------+-------------+-------------+-------------+-------------+ +It is not always possible to run all the test cases on all the scenarios. +The following table details the dependencies of the test cases per +scenario. The scenario dependencies (installer or scenario) are detailed +in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml Test results ============ Test results are available in: - - test results document: http://artifacts.opnfv.org/functest/docs/results/index.html + - test results document: http://artifacts.opnfv.org/functest - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/ - - Test dashboards: http://testresults.opnfv.org/dashboard Known issues ------------ - - IPv6 issues in tempest suite: - - - tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools - - - Tempest code which doesn't reserve big enough allocation - - - https://bugs.launchpad.net/tempest/+bug/1514457 - - - Lack of IP addresses available lead to several errors in different test cases - - - vIMS (http://testresults.opnfv.org/reporting/vims/): - - - the VM needs to have access to OpenStack API. - - - Technical architecture may not allow this access (for security reasons) - - - Orchestrator can be deployed but the vIMS VNF cannot - - - That is the reason why it fails on joid/odl_l2 scenario on Orange POD 2 - - - case needs to be consolidated on new scenaios (bgpvpn, sfc, ovs) Open JIRA tickets ================= -+------------------+-----------------------------------------+ -| JIRA | Description | -+==================+=========================================+ -| FUNCTEST-231 | vPing SSH no more run systematically | -| | in CI | -+------------------+-----------------------------------------+ -| FUNCTEST-230 | Heat issues in Rally scenarios | -+------------------+-----------------------------------------+ -| FUNCTEST-229 | Extend reporting to brahmaputra | -+------------------+-----------------------------------------+ -| FUNCTEST-139 | prepare_env failed due to | -| | https://pypi.python.org/samples is not | -| | accessible | -+------------------+-----------------------------------------+ -| FUNCTEST-135 | vPing scenario failed in odl_l3 scenario| -+------------------+-----------------------------------------+ ++------------------+-----------------------------------------------+ +| JIRA | Description | ++==================+===============================================+ ++------------------+-----------------------------------------------+ ++------------------+-----------------------------------------------+ ++------------------+-----------------------------------------------+ ++------------------+-----------------------------------------------+ ++------------------+-----------------------------------------------+ Useful links ============ - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing + - wiki Functest Colorado page: https://wiki.opnfv.org/display/functest/Functest+Colorado + - Functest repo: https://git.opnfv.org/cgit/functest - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/ - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611 - - Wiki page for B Release: https://wiki.opnfv.org/functest_release_2 + - Functest IRC chan: #opnfv-functest + + - Functest reporting: http://testresults.opnfv.org/reporting + + - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml + + - Functest Colorado user guide: * TODO * + + - Functest installation/configuration guide: * TODO * - - Functest IRC chan: #opnfv-testperf + - Functest developer guide: * TODO * - - Test dashboard: https://www.opnfv.org/opnfvtestgraphs/summary - - Functest dashboard: http://testresults.opnfv.org/dashboard diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst index c0c3aa1b2..0e13d3547 100644 --- a/docs/userguide/index.rst +++ b/docs/userguide/index.rst @@ -405,7 +405,7 @@ The Clearwater architecture is described as follows: parser ^^^^^^ -See parser user guide for details: ** TODO link ** +See parser user guide for details: `[12]`_ .. include:: ./runfunctest.rst @@ -490,6 +490,7 @@ References .. _`[7]`: https://github.com/OpenSCAP/openscap .. _`[9]`: https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/libraries/os_defaults.yaml .. _`[11]`: http://robotframework.org/ +.. _`[12]`: http://artifacts.opnfv.org/parser/docs/userguide/index.html OPNFV main site: opnfvmain_. diff --git a/docs/userguide/introduction.rst b/docs/userguide/introduction.rst index ce84a9c43..53b3df4ad 100644 --- a/docs/userguide/introduction.rst +++ b/docs/userguide/introduction.rst @@ -135,8 +135,14 @@ Features and VNF (Virtual Network Functions). | | | | It provides a fully functional | | | | | VoIP System | + +---------------+----------------+----------------------------------+ -| | | parser | Service template translation | -| | | | TODO See for details | +| | | parser | Parser is an integration project | +| | | | which aims to provide | +| | | | placement/deployment templates | +| | | | translation for OPNFV platform, | +| | | | including TOSCA -> HOT, POLICY ->| +| | | | TOSCA and YANG -> TOSCA. | +| | | | See `Parser User Guide`_ for | +| | | | details | +-------------+---------------+----------------+----------------------------------+ @@ -208,4 +214,5 @@ section `Executing the functest suites`_ of this document. .. _`ONOSFW User Guide`: http://artifacts.opnfv.org/onosfw/brahmaputra/docs/userguide/index.html .. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/brahmaputra/docs/userguide/featureusage.html .. _`Domino User Guide`: http://artifacts.opnfv.org/domino/docs/userguide/index.html +.. _`Parser User Guide`: http://artifacts.opnfv.org/parser/docs/userguide/index.html .. _`Functest Dashboard`: http://testresults.opnfv.org/kibana_dashboards/ diff --git a/docs/userguide/troubleshooting.rst b/docs/userguide/troubleshooting.rst index c23b00602..0288d295c 100644 --- a/docs/userguide/troubleshooting.rst +++ b/docs/userguide/troubleshooting.rst @@ -384,7 +384,7 @@ described in the following table: parser ^^^^^^ -See parser user guide for details: ** TODO link ** +For now log info is the only way to do trouble shooting .. _`OPNFV Functest Developer Guide`: http://artifacts.opnfv.org/functest/docs/devguide/# |