diff options
Diffstat (limited to 'docs/testing/user')
| -rw-r--r-- | docs/testing/user/configguide/ci.rst | 16 | ||||
| -rw-r--r-- | docs/testing/user/configguide/configguide.rst | 154 | ||||
| -rw-r--r-- | docs/testing/user/configguide/index.rst | 3 | ||||
| -rw-r--r-- | docs/testing/user/configguide/prerequisites.rst | 2 | ||||
| -rw-r--r-- | docs/testing/user/userguide/reporting.rst | 4 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_details.rst | 2 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_overview.rst | 6 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_results.rst | 152 | ||||
| -rw-r--r-- | docs/testing/user/userguide/troubleshooting.rst | 54 |
9 files changed, 231 insertions, 162 deletions
diff --git a/docs/testing/user/configguide/ci.rst b/docs/testing/user/configguide/ci.rst deleted file mode 100644 index ee815dbe1..000000000 --- a/docs/testing/user/configguide/ci.rst +++ /dev/null @@ -1,16 +0,0 @@ -.. SPDX-License-Identifier: CC-BY-4.0 - -Integration in CI -================= -In CI we use the Docker images and execute the appropriate commands within the -container from Jenkins. - -4 steps have been defined:: - * functest-cleanup: clean existing functest dockers on the jumphost - * functest-daily: run dockers opnfv/functest-* (healthcheck, smoke, - benchmarking, vnf) - * functest-store-results: push logs to artifacts - -See `[1]`_ for details. - -.. _`[1]`: https://github.com/opnfv/releng/blob/master/jjb/functest/functest-daily-jobs.yaml diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst index a0ef6bf4a..b764d9e66 100644 --- a/docs/testing/user/configguide/configguide.rst +++ b/docs/testing/user/configguide/configguide.rst @@ -13,7 +13,6 @@ Functest Dockers for OpenStack deployment ----------------------------------------- Docker images are available on the dockerhub: - * opnfv/functest-core * opnfv/functest-healthcheck * opnfv/functest-smoke * opnfv/functest-benchmarking @@ -29,6 +28,9 @@ cat env:: NAMESERVER=XXX # if not 8.8.8.8 EXTERNAL_NETWORK=XXX # if not first network with router:external=True NEW_USER_ROLE=XXX # if not member + SDN_CONTROLLER_IP=XXX # if odl scenario + VOLUME_DEVICE_NAME=XXX # if not vdb + FLAVOR_EXTRA_SPECS=hw:mem_page_size:large # if fdio scenarios See section on environment variables for details. @@ -41,6 +43,7 @@ cat env_file:: export OS_PROJECT_NAME=XXX export OS_PASSWORD=XXX export OS_IDENTITY_API_VERSION=3 + export OS_REGION_NAME=XXX See section on OpenStack credentials for details. @@ -51,8 +54,7 @@ Create a directory for the different images (attached as a Docker volume):: images/cirros-0.4.0-aarch64-disk.img images/cirros-0.4.0-x86_64-disk.img images/cloudify-docker-manager-community-19.01.24.tar - images/shaker-image-arm64.qcow2 - images/shaker-image.qcow2 + images/shaker-image-1.3.0+stretch.qcow2 images/ubuntu-14.04-server-cloudimg-amd64-disk1.img images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img images/ubuntu-16.04-server-cloudimg-amd64-disk1.img @@ -73,17 +75,17 @@ Results shall be displayed as follows:: +--------------------------+------------------+---------------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +--------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:01 | PASS | + | connection_check | functest | healthcheck | 00:03 | PASS | | tenantnetwork1 | functest | healthcheck | 00:06 | PASS | - | tenantnetwork2 | functest | healthcheck | 00:06 | PASS | - | vmready1 | functest | healthcheck | 00:08 | PASS | - | vmready2 | functest | healthcheck | 00:08 | PASS | - | singlevm1 | functest | healthcheck | 00:35 | PASS | - | singlevm2 | functest | healthcheck | 00:38 | PASS | - | vping_ssh | functest | healthcheck | 00:47 | PASS | - | vping_userdata | functest | healthcheck | 00:33 | PASS | - | cinder_test | functest | healthcheck | 01:16 | PASS | - | tempest_smoke | functest | healthcheck | 07:05 | PASS | + | tenantnetwork2 | functest | healthcheck | 00:07 | PASS | + | vmready1 | functest | healthcheck | 00:09 | PASS | + | vmready2 | functest | healthcheck | 00:09 | PASS | + | singlevm1 | functest | healthcheck | 00:30 | PASS | + | singlevm2 | functest | healthcheck | 00:33 | PASS | + | vping_ssh | functest | healthcheck | 00:42 | PASS | + | vping_userdata | functest | healthcheck | 00:38 | PASS | + | cinder_test | functest | healthcheck | 01:06 | PASS | + | tempest_smoke | functest | healthcheck | 04:03 | PASS | | odl | functest | healthcheck | 00:00 | SKIP | +--------------------------+------------------+---------------------+------------------+----------------+ @@ -104,18 +106,19 @@ Results shall be displayed as follows:: +------------------------------------+------------------+---------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +------------------------------------+------------------+---------------+------------------+----------------+ - | neutron-tempest-plugin-api | functest | smoke | 13:50 | PASS | - | rally_sanity | functest | smoke | 14:13 | PASS | - | refstack_compute | functest | smoke | 07:24 | PASS | - | refstack_object | functest | smoke | 02:08 | PASS | - | refstack_platform | functest | smoke | 08:32 | PASS | - | tempest_full | functest | smoke | 58:40 | PASS | - | tempest_scenario | functest | smoke | 09:47 | PASS | - | patrole | functest | smoke | 05:34 | PASS | + | neutron-tempest-plugin-api | functest | smoke | 09:26 | PASS | + | rally_sanity | functest | smoke | 16:43 | PASS | + | refstack_compute | functest | smoke | 06:48 | PASS | + | refstack_object | functest | smoke | 02:04 | PASS | + | refstack_platform | functest | smoke | 07:53 | PASS | + | tempest_full | functest | smoke | 31:51 | PASS | + | tempest_scenario | functest | smoke | 09:38 | PASS | + | tempest_slow | functest | smoke | 22:21 | PASS | + | patrole | functest | smoke | 02:32 | PASS | + | barbican | functest | smoke | 02:16 | PASS | | neutron_trunk | functest | smoke | 00:00 | SKIP | - | networking-bgpvpn | functest | smoke | 01:43 | PASS | - | networking-sfc | functest | smoke | 04:49 | PASS | - | barbican | functest | smoke | 05:09 | PASS | + | networking-bgpvpn | functest | smoke | 00:00 | SKIP | + | networking-sfc | functest | smoke | 00:00 | SKIP | +------------------------------------+------------------+---------------+------------------+----------------+ Note: if the scenario does not support some tests, they are indicated as SKIP. @@ -136,10 +139,10 @@ Results shall be displayed as follows:: +--------------------+------------------+----------------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +--------------------+------------------+----------------------+------------------+----------------+ - | rally_full | functest | benchmarking | 121:55 | PASS | - | rally_jobs | functest | benchmarking | 24:21 | PASS | - | vmtp | functest | benchmarking | 00:00 | SKIP | - | shaker | functest | benchmarking | 00:00 | SKIP | + | rally_full | functest | benchmarking | 92:16 | PASS | + | rally_jobs | functest | benchmarking | 17:31 | PASS | + | vmtp | functest | benchmarking | 15:20 | PASS | + | shaker | functest | benchmarking | 22:44 | PASS | +--------------------+------------------+----------------------+------------------+----------------+ Note: if the scenario does not support some tests, they are indicated as SKIP. @@ -160,20 +163,22 @@ Results shall be displayed as follows:: +----------------------+------------------+--------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +----------------------+------------------+--------------+------------------+----------------+ - | cloudify | functest | vnf | 06:42 | PASS | - | cloudify_ims | functest | vnf | 39:21 | PASS | - | heat_ims | functest | vnf | 34:33 | PASS | - | vyos_vrouter | functest | vnf | 20:20 | PASS | - | juju_epc | functest | vnf | 56:55 | PASS | + | cloudify | functest | vnf | 03:35 | PASS | + | cloudify_ims | functest | vnf | 23:26 | PASS | + | heat_ims | functest | vnf | 34:22 | PASS | + | vyos_vrouter | functest | vnf | 13:49 | PASS | + | juju_epc | functest | vnf | 41:49 | PASS | +----------------------+------------------+--------------+------------------+----------------+ Functest Dockers for Kubernetes deployment ------------------------------------------ Docker images are available on the dockerhub: - * opnfv/functest-kubernetes-core - * opnfv/functest-kubernetest-healthcheck - * opnfv/functest-kubernetest-smoke + * opnfv/functest-kubernetes-healthcheck + * opnfv/functest-kubernetes-smoke + * opnfv/functest-kubernetes-security + * opnfv/functest-kubernetes-benchmarking + * opnfv/functest-kubernetes-cnf Preparing your environment ^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -199,7 +204,8 @@ Results shall be displayed as follows:: +-------------------+------------------+---------------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +-------------------+------------------+---------------------+------------------+----------------+ - | k8s_smoke | functest | healthcheck | 02:27 | PASS | + | k8s_quick | functest | healthcheck | 00:14 | PASS | + | k8s_smoke | functest | healthcheck | 00:54 | PASS | +-------------------+------------------+---------------------+------------------+----------------+ Testing smoke suite @@ -213,16 +219,72 @@ Run smoke suite:: Results shall be displayed as follows:: - +-------------------------+------------------+---------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------------+------------------+---------------+------------------+----------------+ - | k8s_conformance | functest | smoke | 57:14 | PASS | - +-------------------------+------------------+---------------+------------------+----------------+ + +---------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+---------------+------------------+----------------+ + | k8s_conformance | functest | smoke | 94:11 | PASS | + | xrally_kubernetes | functest | smoke | 13:39 | PASS | + +---------------------------+------------------+---------------+------------------+----------------+ + +Testing security suite +^^^^^^^^^^^^^^^^^^^^^^ + +Run smoke suite:: + + sudo docker run -it --env-file env \ + -v $(pwd)/config:/root/.kube/config \ + opnfv/functest-kubernetes-security:iruya + +Results shall be displayed as follows:: + + +---------------------------+------------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+------------------+------------------+----------------+ + | kube_hunter | functest | security | 00:21 | PASS | + | kube_bench_master | functest | security | 00:01 | PASS | + | kube_bench_node | functest | security | 00:01 | PASS | + +---------------------------+------------------+------------------+------------------+----------------+ + +Testing benchmarking suite +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Run benchmarking suite:: + + sudo docker run -it --env-file env \ + -v $(pwd)/config:/root/.kube/config \ + opnfv/functest-kubernetes-benchmarking:iruya + +Results shall be displayed as follows:: + + +--------------------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------------------+------------------+----------------------+------------------+----------------+ + | xrally_kubernetes_full | functest | benchmarking | 38:21 | PASS | + +--------------------------------+------------------+----------------------+------------------+----------------+ + +Testing cnf suite +^^^^^^^^^^^^^^^^^ + +Run cnf suite:: + + sudo docker run -it --env-file env \ + -v $(pwd)/config:/root/.kube/config \ + opnfv/functest-kubernetes-cnf:iruya + +Results shall be displayed as follows:: + + +-------------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------+------------------+--------------+------------------+----------------+ + | k8s_vims | functest | cnf | 08:58 | PASS | + | helm_vims | functest | cnf | 08:45 | PASS | + | cnf_conformance | functest | cnf | 02:10 | PASS | + +-------------------------+------------------+--------------+------------------+----------------+ Environment variables ===================== -Several environement variables may be specified: +Several environment variables may be specified: * INSTALLER_IP=<Specific IP Address> * DEPLOY_SCENARIO=<vim>-<controller>-<nfv_feature>-<ha_mode> @@ -347,7 +409,7 @@ By default all the logs are put un /home/opnfv/functest/results/functest.log. If you want to have more logs in console, you may edit the logging.ini file manually. Connect on the docker then edit the file located in -/usr/lib/python2.7/site-packages/xtesting/ci/logging.ini +/usr/lib/python3.6/site-packages/xtesting/ci/logging.ini Change wconsole to console in the desired module to get more traces. @@ -357,8 +419,8 @@ Configuration You may also directly modify the python code or the configuration file (e.g. testcases.yaml used to declare test constraints) under -/usr/lib/python2.7/site-packages/xtesting and -/usr/lib/python2.7/site-packages/functest +/usr/lib/python3.6/site-packages/xtesting and +/usr/lib/python3.6/site-packages/functest Tips diff --git a/docs/testing/user/configguide/index.rst b/docs/testing/user/configguide/index.rst index e7967adb3..e5963435e 100644 --- a/docs/testing/user/configguide/index.rst +++ b/docs/testing/user/configguide/index.rst @@ -11,7 +11,6 @@ Functest Installation Guide intro.rst prerequisites.rst configguide.rst - ci.rst References ========== @@ -20,4 +19,4 @@ References IRC support channel: #opnfv-functest -.. _`[1]`: https://github.com/opnfv/releng/blob/master/jjb/functest/functest-daily-jobs.yaml +.. _`[1]`: https://github.com/opnfv/releng/blob/master/jjb/functest/functest.yaml diff --git a/docs/testing/user/configguide/prerequisites.rst b/docs/testing/user/configguide/prerequisites.rst index c7b52dc54..dab4a9e6b 100644 --- a/docs/testing/user/configguide/prerequisites.rst +++ b/docs/testing/user/configguide/prerequisites.rst @@ -92,4 +92,4 @@ should thus be known. Ensure you can reach each node in the SUT, from the Jumphost using the 'ping' command using the respective IP address on the public/external network for each node in the SUT. The details of how to determine the needed IP addresses for each node in the SUT may vary according -to the used installer and are therefore ommitted here. +to the used installer and are therefore omitted here. diff --git a/docs/testing/user/userguide/reporting.rst b/docs/testing/user/userguide/reporting.rst index 7c8c48ece..8fad55d33 100644 --- a/docs/testing/user/userguide/reporting.rst +++ b/docs/testing/user/userguide/reporting.rst @@ -48,7 +48,7 @@ and features) corresponding to this scenario. All the testcases (X) listed in the table are runnable on os-odl_l2-nofeature scenarios. Please note that other test cases (e.g. sfc_odl, bgpvpn) need ODL configuration -addons and, as a consequence, specific scenario. +add-ons and, as a consequence, specific scenario. There are not considered as runnable on the generic odl_l2 scenario. @@ -82,7 +82,7 @@ Therefore the scoring provides 3 types of indicators: * the maturity: if the percentage (scoring/target scoring * 100) is high, it means that all the tests are PASS * the stability: as the number of iteration is included in the calculation, - the pecentage can be high only if the scenario is run regularly (at least + the percentage can be high only if the scenario is run regularly (at least more than 4 iterations over the last 10 days in CI) In any case, the scoring is used to give feedback to the other projects and diff --git a/docs/testing/user/userguide/test_details.rst b/docs/testing/user/userguide/test_details.rst index 6061ce9ef..7c4aba0a7 100644 --- a/docs/testing/user/userguide/test_details.rst +++ b/docs/testing/user/userguide/test_details.rst @@ -407,6 +407,6 @@ The kubernetes testcases are distributed across various Tiers: .. _`[11]`: https://robotframework.org/ .. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter -.. _`[15]`: https://www.rebaca.com/what-we-do/abot-5g-network-simulator/ +.. _`[15]`: https://github.com/RebacaInc/abot_charm .. _`[16]`: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md .. _`[17]`: https://github.com/Metaswitch/clearwater-heat/blob/release-129/clearwater.yaml diff --git a/docs/testing/user/userguide/test_overview.rst b/docs/testing/user/userguide/test_overview.rst index 5aef27c1c..bc3e79dcb 100644 --- a/docs/testing/user/userguide/test_overview.rst +++ b/docs/testing/user/userguide/test_overview.rst @@ -17,7 +17,7 @@ The current list of test suites can be distributed over 5 main domains: Functest test suites are also distributed in the OPNFV testing categories: healthcheck, smoke, benchmarking, VNF, Stress tests. -All the Healthcheck and smoke tests of a given scenario must be succesful to +All the Healthcheck and smoke tests of a given scenario must be successful to validate the scenario for the release. +-------------+---------------+------------+----------------------------------+ @@ -90,7 +90,7 @@ validate the scenario for the release. | VNF | vnf | cloudify | Example of a real VNF deployment | | | | \_ims | to show the NFV capabilities of | | | | | the platform. The IP Multimedia | -| | | | Subsytem is a typical Telco test | +| | | | Subsystem is a typical Telco test| | | | | case, referenced by ETSI. | | | | | It provides a fully functional | | | | | VoIP System | @@ -177,7 +177,7 @@ deployed services. The system uses the environment variables to automatically determine the valid test cases, for each given environment. A convenience Functest CLI utility is also available to simplify setting up the -Functest evironment, management of the OpenStack environment (e.g. resource +Functest environment, management of the OpenStack environment (e.g. resource clean-up) and for executing tests. The Functest CLI organised the testcase into logical Tiers, which contain in turn one or more testcases. The CLI allows execution of a single specified diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst index c4d85f631..b2530d85c 100644 --- a/docs/testing/user/userguide/test_results.rst +++ b/docs/testing/user/userguide/test_results.rst @@ -10,7 +10,7 @@ In manual mode test results are displayed in the console and result files are put in /home/opnfv/functest/results. If you want additional logs, you may configure the logging.ini under -/usr/lib/python2.7/site-packages/xtesting/ci. +/usr/lib/python3.6/site-packages/xtesting/ci. Automated testing ----------------- @@ -21,75 +21,111 @@ end of each suite and can be described as follow. Healthcheck suite:: - +----------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:07 | PASS | - | api_check | functest | healthcheck | 07:46 | PASS | - | snaps_health_check | functest | healthcheck | 00:36 | PASS | - +----------------------------+------------------+---------------------+------------------+----------------+ + +--------------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------------+------------------+---------------------+------------------+----------------+ + | connection_check | functest | healthcheck | 00:03 | PASS | + | tenantnetwork1 | functest | healthcheck | 00:06 | PASS | + | tenantnetwork2 | functest | healthcheck | 00:07 | PASS | + | vmready1 | functest | healthcheck | 00:09 | PASS | + | vmready2 | functest | healthcheck | 00:09 | PASS | + | singlevm1 | functest | healthcheck | 00:30 | PASS | + | singlevm2 | functest | healthcheck | 00:33 | PASS | + | vping_ssh | functest | healthcheck | 00:42 | PASS | + | vping_userdata | functest | healthcheck | 00:38 | PASS | + | cinder_test | functest | healthcheck | 01:06 | PASS | + | tempest_smoke | functest | healthcheck | 04:03 | PASS | + | odl | functest | healthcheck | 00:00 | SKIP | + +--------------------------+------------------+---------------------+------------------+----------------+ Smoke suite:: - +------------------------------+------------------+---------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +------------------------------+------------------+---------------+------------------+----------------+ - | vping_ssh | functest | smoke | 00:57 | PASS | - | vping_userdata | functest | smoke | 00:33 | PASS | - | tempest_smoke_serial | functest | smoke | 13:22 | PASS | - | rally_sanity | functest | smoke | 24:07 | PASS | - | refstack_defcore | functest | smoke | 05:21 | PASS | - | patrole | functest | smoke | 04:29 | PASS | - | snaps_smoke | functest | smoke | 46:54 | PASS | - | odl | functest | smoke | 00:00 | SKIP | - | neutron_trunk | functest | smoke | 00:00 | SKIP | - +------------------------------+------------------+---------------+------------------+----------------+ - -Components suite:: - - +-------------------------------+------------------+--------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------------------+------------------+--------------------+------------------+----------------+ - | tempest_full_parallel | functest | components | 48:28 | PASS | - | rally_full | functest | components | 126:02 | PASS | - +-------------------------------+------------------+--------------------+------------------+----------------+ + +------------------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +------------------------------------+------------------+---------------+------------------+----------------+ + | neutron-tempest-plugin-api | functest | smoke | 09:26 | PASS | + | rally_sanity | functest | smoke | 16:43 | PASS | + | refstack_compute | functest | smoke | 06:48 | PASS | + | refstack_object | functest | smoke | 02:04 | PASS | + | refstack_platform | functest | smoke | 07:53 | PASS | + | tempest_full | functest | smoke | 31:51 | PASS | + | tempest_scenario | functest | smoke | 09:38 | PASS | + | tempest_slow | functest | smoke | 22:21 | PASS | + | patrole | functest | smoke | 02:32 | PASS | + | barbican | functest | smoke | 02:16 | PASS | + | neutron_trunk | functest | smoke | 00:00 | SKIP | + | networking-bgpvpn | functest | smoke | 00:00 | SKIP | + | networking-sfc | functest | smoke | 00:00 | SKIP | + +------------------------------------+------------------+---------------+------------------+----------------+ + +Benchmarking suite:: + + +--------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------+------------------+----------------------+------------------+----------------+ + | rally_full | functest | benchmarking | 92:16 | PASS | + | rally_jobs | functest | benchmarking | 17:31 | PASS | + | vmtp | functest | benchmarking | 15:20 | PASS | + | shaker | functest | benchmarking | 22:44 | PASS | + +--------------------+------------------+----------------------+------------------+----------------+ Vnf suite:: - +----------------------+------------------+--------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------+------------------+--------------+------------------+----------------+ - | cloudify_ims | functest | vnf | 28:15 | PASS | - | vyos_vrouter | functest | vnf | 17:59 | PASS | - | juju_epc | functest | vnf | 46:44 | PASS | - +----------------------+------------------+--------------+------------------+----------------+ - -Functest Kubernetes test result:: - - +--------------------------------------+------------------------------------------------------------+ - | ENV VAR | VALUE | - +--------------------------------------+------------------------------------------------------------+ - | INSTALLER_TYPE | compass | - | DEPLOY_SCENARIO | k8-nosdn-nofeature-ha | - | BUILD_TAG | jenkins-functest-compass-baremetal-daily-master-75 | - | CI_LOOP | daily | - +--------------------------------------+------------------------------------------------------------+ + +----------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +----------------------+------------------+--------------+------------------+----------------+ + | cloudify | functest | vnf | 03:35 | PASS | + | cloudify_ims | functest | vnf | 23:26 | PASS | + | heat_ims | functest | vnf | 34:22 | PASS | + | vyos_vrouter | functest | vnf | 13:49 | PASS | + | juju_epc | functest | vnf | 41:49 | PASS | + +----------------------+------------------+--------------+------------------+----------------+ Kubernetes healthcheck suite:: - +-------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------+------------------+---------------------+------------------+----------------+ - | k8s_smoke | functest | healthcheck | 01:54 | PASS | - +-------------------+------------------+---------------------+------------------+----------------+ + +-------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------+------------------+---------------------+------------------+----------------+ + | k8s_quick | functest | healthcheck | 00:14 | PASS | + | k8s_smoke | functest | healthcheck | 00:54 | PASS | + +-------------------+------------------+---------------------+------------------+----------------+ Kubernetes smoke suite:: - +-------------------------+------------------+---------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------------+------------------+---------------+------------------+----------------+ - | k8s_conformance | functest | smoke | 57:47 | PASS | - +-------------------------+------------------+---------------+------------------+----------------+ + +---------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+---------------+------------------+----------------+ + | k8s_conformance | functest | smoke | 94:11 | PASS | + | xrally_kubernetes | functest | smoke | 13:39 | PASS | + +---------------------------+------------------+---------------+------------------+----------------+ + +Kubernetes security suite:: + + +---------------------------+------------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+------------------+------------------+----------------+ + | kube_hunter | functest | security | 00:21 | PASS | + | kube_bench_master | functest | security | 00:01 | PASS | + | kube_bench_node | functest | security | 00:01 | PASS | + +---------------------------+------------------+------------------+------------------+----------------+ + +Kubernetes benchmarking suite:: + + +--------------------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------------------+------------------+----------------------+------------------+----------------+ + | xrally_kubernetes_full | functest | benchmarking | 38:21 | PASS | + +--------------------------------+------------------+----------------------+------------------+----------------+ + +Kubernetes cnf suite:: + + +-------------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------+------------------+--------------+------------------+----------------+ + | k8s_vims | functest | cnf | 08:58 | PASS | + | helm_vims | functest | cnf | 08:45 | PASS | + | cnf_conformance | functest | cnf | 02:10 | PASS | + +-------------------------+------------------+--------------+------------------+----------------+ Results are automatically pushed to the test results database, some additional result files are pushed to OPNFV artifact web sites. diff --git a/docs/testing/user/userguide/troubleshooting.rst b/docs/testing/user/userguide/troubleshooting.rst index 3e1d044b2..6370815b4 100644 --- a/docs/testing/user/userguide/troubleshooting.rst +++ b/docs/testing/user/userguide/troubleshooting.rst @@ -44,8 +44,8 @@ follows:: The Functest CLI is designed to route a call to the corresponding internal python scripts, located in paths:: - /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_ssh.py - /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_userdata.py + /usr/lib/python3.6/site-packages/functest/opnfv_tests/openstack/vping/vping_ssh.py + /usr/lib/python3.6/site-packages/functest/opnfv_tests/openstack/vping/vping_userdata.py Notes: @@ -125,7 +125,7 @@ This test case creates a floating IP on the external network and assigns it to the second instance **opnfv-vping-2**. The purpose of this is to establish a SSH connection to that instance and SCP a script that will ping the first instance. This script is located in the repository under -/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/ping.sh +/usr/lib/python3.6/site-packages/functest/opnfv_tests/openstack/vping/ping.sh and takes an IP as a parameter. When the SCP is completed, the test will do a SSH call to that script inside the second instance. Some problems can happen here:: @@ -133,8 +133,8 @@ here:: vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting If this is displayed, stop the test or wait for it to finish, if you have used -the special method of test invocation with specific supression of OpenStack -resource clean-up, as explained earler. It means that the Container can not +the special method of test invocation with specific suppression of OpenStack +resource clean-up, as explained earlier. It means that the Container can not reach the Public/External IP assigned to the instance **opnfv-vping-2**. There are many possible reasons, and they really depend on the chosen scenario. For most of the ODL-L3 and ONOS scenarios this has been noticed and it is a known @@ -158,7 +158,7 @@ container:: ping <public IP> If the ping does not return anything, try to ping from the Host where the -Docker container is running. If that solves the problem, check the iptable +Docker container is running. If that solves the problem, check the iptables rules because there might be some rules rejecting ICMP or TCP traffic coming/going from/to the container. @@ -261,40 +261,28 @@ Functest offers a possibility to test a customized list of Tempest test cases. To enable that, add a new entry in docker/smoke/testcases.yaml on the "smoke" container with the following content:: - - - case_name: tempest_custom - project_name: functest - criteria: 100 - blocking: false - description: >- - The test case allows running a customized list of tempest - test cases - dependencies: - installer: '' - scenario: '' - run: - module: 'functest.opnfv_tests.openstack.tempest.tempest' - class: 'TempestCustom' - -Also, a list of the Tempest test cases must be provided to the container or -modify the existing one in -/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt - -Example of custom list of tests 'my-custom-tempest-tests.txt':: - - tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops[compute,id-7fff3fb3-91d8-4fd0-bd7d-0204f1f180ba,network,smoke] - tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops[compute,id-f323b3ba-82f8-4db7-8ea6-6a895869ec49,network,smoke] + - + case_name: tempest_custom + project_name: functest + criteria: 100 + blocking: false + description: >- + The test case allows running a customized list of tempest + test cases + run: + name: tempest_common + args: + mode: "tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops|\ + tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops" This is an example of running a customized list of Tempest tests in Functest:: sudo docker run --env-file env \ -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ -v $(pwd)/images:/home/opnfv/functest/images \ - -v $(pwd)/my-custom-testcases.yaml:/usr/lib/python2.7/site-packages/functest/ci/testcases.yaml \ - -v $(pwd)/my-custom-tempest-tests.txt:/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt \ + -v $(pwd)/my-custom-testcases.yaml:/usr/lib/python3.6/site-packages/functest/ci/testcases.yaml \ opnfv/functest-smoke run_tests -t tempest_custom - Rally ^^^^^ @@ -314,7 +302,7 @@ Possible scenarios are: To know more about what those scenarios are doing, they are defined in directory: -/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/rally/scenario +/usr/lib/python3.6/site-packages/functest/opnfv_tests/openstack/rally/scenario For more info about Rally scenario definition please refer to the Rally official documentation. `[3]`_ |