diff options
Diffstat (limited to 'docs/testing/user')
| -rw-r--r-- | docs/testing/user/configguide/ci.rst | 16 | ||||
| -rw-r--r-- | docs/testing/user/configguide/configguide.rst | 235 | ||||
| -rw-r--r-- | docs/testing/user/configguide/index.rst | 3 | ||||
| -rw-r--r-- | docs/testing/user/configguide/prerequisites.rst | 2 | ||||
| -rw-r--r-- | docs/testing/user/userguide/index.rst | 6 | ||||
| -rw-r--r-- | docs/testing/user/userguide/reporting.rst | 4 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_details.rst | 51 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_overview.rst | 63 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_results.rst | 213 | ||||
| -rw-r--r-- | docs/testing/user/userguide/troubleshooting.rst | 56 |
10 files changed, 317 insertions, 332 deletions
diff --git a/docs/testing/user/configguide/ci.rst b/docs/testing/user/configguide/ci.rst deleted file mode 100644 index aee51a835..000000000 --- a/docs/testing/user/configguide/ci.rst +++ /dev/null @@ -1,16 +0,0 @@ -.. SPDX-License-Identifier: CC-BY-4.0 - -Integration in CI -================= -In CI we use the Docker images and execute the appropriate commands within the -container from Jenkins. - -4 steps have been defined:: - * functest-cleanup: clean existing functest dockers on the jumphost - * functest-daily: run dockers opnfv/functest-* (healthcheck, smoke, features, - vnf) - * functest-store-results: push logs to artifacts - -See `[1]`_ for details. - -.. _`[1]`: https://github.com/opnfv/releng/blob/master/jjb/functest/functest-daily-jobs.yaml diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst index 95aecb2df..47c25adb1 100644 --- a/docs/testing/user/configguide/configguide.rst +++ b/docs/testing/user/configguide/configguide.rst @@ -13,11 +13,11 @@ Functest Dockers for OpenStack deployment ----------------------------------------- Docker images are available on the dockerhub: - * opnfv/functest-core * opnfv/functest-healthcheck * opnfv/functest-smoke + * opnfv/functest-smoke-cntt * opnfv/functest-benchmarking - * opnfv/functest-features + * opnfv/functest-benchmarking-cntt * opnfv/functest-vnf @@ -29,7 +29,11 @@ cat env:: DEPLOY_SCENARIO=XXX # if not os-nosdn-nofeature-noha scenario NAMESERVER=XXX # if not 8.8.8.8 EXTERNAL_NETWORK=XXX # if not first network with router:external=True + DASHBOARD_URL=XXX # else tempest_horizon will be skipped NEW_USER_ROLE=XXX # if not member + SDN_CONTROLLER_IP=XXX # if odl scenario + VOLUME_DEVICE_NAME=XXX # if not vdb + FLAVOR_EXTRA_SPECS=hw:mem_page_size:large # if fdio scenarios See section on environment variables for details. @@ -42,18 +46,19 @@ cat env_file:: export OS_PROJECT_NAME=XXX export OS_PASSWORD=XXX export OS_IDENTITY_API_VERSION=3 + export OS_REGION_NAME=XXX See section on OpenStack credentials for details. Create a directory for the different images (attached as a Docker volume):: - mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh | bash -s -- images && ls -1 images/* + mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh | sh -s -- images && ls -1 images/* - images/cirros-0.4.0-aarch64-disk.img - images/cirros-0.4.0-x86_64-disk.img + images/cirros-0.5.1-aarch64-disk.img + images/cirros-0.5.1-x86_64-disk.img images/cloudify-docker-manager-community-19.01.24.tar - images/shaker-image-arm64.qcow2 - images/shaker-image.qcow2 + images/Fedora-Cloud-Base-30-1.2.x86_64.qcow2 + images/shaker-image-1.3.0+stretch.qcow2 images/ubuntu-14.04-server-cloudimg-amd64-disk1.img images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img images/ubuntu-16.04-server-cloudimg-amd64-disk1.img @@ -74,17 +79,18 @@ Results shall be displayed as follows:: +--------------------------+------------------+---------------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +--------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:01 | PASS | - | tenantnetwork1 | functest | healthcheck | 00:06 | PASS | + | connection_check | functest | healthcheck | 00:03 | PASS | + | tenantnetwork1 | functest | healthcheck | 00:05 | PASS | | tenantnetwork2 | functest | healthcheck | 00:06 | PASS | - | vmready1 | functest | healthcheck | 00:08 | PASS | + | vmready1 | functest | healthcheck | 00:06 | PASS | | vmready2 | functest | healthcheck | 00:08 | PASS | - | singlevm1 | functest | healthcheck | 00:35 | PASS | - | singlevm2 | functest | healthcheck | 00:38 | PASS | - | vping_ssh | functest | healthcheck | 00:47 | PASS | - | vping_userdata | functest | healthcheck | 00:33 | PASS | - | cinder_test | functest | healthcheck | 01:16 | PASS | - | tempest_smoke | functest | healthcheck | 07:05 | PASS | + | singlevm1 | functest | healthcheck | 00:32 | PASS | + | singlevm2 | functest | healthcheck | 00:37 | PASS | + | vping_ssh | functest | healthcheck | 00:46 | PASS | + | vping_userdata | functest | healthcheck | 00:39 | PASS | + | cinder_test | functest | healthcheck | 01:05 | PASS | + | tempest_smoke | functest | healthcheck | 05:39 | PASS | + | tempest_horizon | functest | healthcheck | 01:05 | PASS | | odl | functest | healthcheck | 00:00 | SKIP | +--------------------------+------------------+---------------------+------------------+----------------+ @@ -102,26 +108,57 @@ Run smoke suite:: Results shall be displayed as follows:: - +------------------------------------+------------------+---------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +------------------------------------+------------------+---------------+------------------+----------------+ - | neutron-tempest-plugin-api | functest | smoke | 13:50 | PASS | - | rally_sanity | functest | smoke | 14:13 | PASS | - | refstack_compute | functest | smoke | 07:24 | PASS | - | refstack_object | functest | smoke | 02:08 | PASS | - | refstack_platform | functest | smoke | 08:32 | PASS | - | tempest_full | functest | smoke | 58:40 | PASS | - | tempest_scenario | functest | smoke | 09:47 | PASS | - | patrole | functest | smoke | 05:34 | PASS | - | neutron_trunk | functest | smoke | 00:00 | SKIP | - | networking-bgpvpn | functest | smoke | 01:43 | PASS | - | networking-sfc | functest | smoke | 04:49 | PASS | - | barbican | functest | smoke | 05:09 | PASS | - +------------------------------------+------------------+---------------+------------------+----------------+ + +---------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+---------------+------------------+----------------+ + | tempest_neutron | functest | smoke | 15:30 | PASS | + | tempest_cinder | functest | smoke | 02:01 | PASS | + | tempest_keystone | functest | smoke | 01:17 | PASS | + | tempest_heat | functest | smoke | 22:14 | PASS | + | tempest_telemetry | functest | smoke | 00:00 | SKIP | + | rally_sanity | functest | smoke | 17:24 | PASS | + | refstack_compute | functest | smoke | 07:03 | PASS | + | refstack_object | functest | smoke | 02:09 | PASS | + | refstack_platform | functest | smoke | 07:31 | PASS | + | tempest_full | functest | smoke | 41:52 | PASS | + | tempest_scenario | functest | smoke | 08:42 | PASS | + | tempest_slow | functest | smoke | 43:42 | PASS | + | patrole_admin | functest | smoke | 21:06 | PASS | + | patrole_member | functest | smoke | 21:23 | PASS | + | patrole_reader | functest | smoke | 21:56 | PASS | + | tempest_barbican | functest | smoke | 02:30 | PASS | + | tempest_octavia | functest | smoke | 00:00 | SKIP | + | tempest_cyborg | functest | smoke | 00:00 | SKIP | + +---------------------------+------------------+---------------+------------------+----------------+ Note: if the scenario does not support some tests, they are indicated as SKIP. See User guide for details. +Testing smoke CNTT suite +^^^^^^^^^^^^^^^^^^^^^^^^ + +Run smoke-cntt suite:: + + sudo docker run --env-file env \ + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ + -v $(pwd)/images:/home/opnfv/functest/images \ + opnfv/functest-smoke-cntt + +Results shall be displayed as follows:: + + +-------------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------------+------------------+---------------+------------------+----------------+ + | tempest_neutron_cntt | functest | smoke | 11:35 | PASS | + | tempest_cinder_cntt | functest | smoke | 01:58 | PASS | + | tempest_keystone_cntt | functest | smoke | 01:13 | PASS | + | tempest_heat_cntt | functest | smoke | 22:32 | PASS | + | rally_sanity_cntt | functest | smoke | 17:16 | PASS | + | tempest_full_cntt | functest | smoke | 41:13 | PASS | + | tempest_scenario_cntt | functest | smoke | 08:57 | PASS | + | tempest_slow_cntt | functest | smoke | 35:58 | PASS | + +-------------------------------+------------------+---------------+------------------+----------------+ + Testing benchmarking suite ^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -137,39 +174,35 @@ Results shall be displayed as follows:: +--------------------+------------------+----------------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +--------------------+------------------+----------------------+------------------+----------------+ - | rally_full | functest | benchmarking | 121:55 | PASS | - | rally_jobs | functest | benchmarking | 24:21 | PASS | - | vmtp | functest | benchmarking | 00:00 | SKIP | - | shaker | functest | benchmarking | 00:00 | SKIP | + | rally_full | functest | benchmarking | 93:03 | PASS | + | rally_jobs | functest | benchmarking | 27:05 | PASS | + | vmtp | functest | benchmarking | 17:56 | PASS | + | shaker | functest | benchmarking | 24:02 | PASS | +--------------------+------------------+----------------------+------------------+----------------+ Note: if the scenario does not support some tests, they are indicated as SKIP. See User guide for details. -Testing features suite -^^^^^^^^^^^^^^^^^^^^^^ +Testing benchmarking CNTT suite +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Run features suite:: +Run benchmarking-cntt suite:: sudo docker run --env-file env \ -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-features + opnfv/functest-benchmarking-cntt Results shall be displayed as follows:: - +-----------------------------+-------------------+------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-----------------------------+-------------------+------------------+------------------+----------------+ - | doctor-notification | doctor | features | 00:00 | SKIP | - | functest-odl-sfc | sfc | features | 00:00 | SKIP | - | barometercollectd | barometer | features | 00:00 | SKIP | - | vgpu | functest | features | 00:00 | SKIP | - | stor4nfv_os | stor4nfv | features | 00:00 | SKIP | - +-----------------------------+-------------------+------------------+------------------+----------------+ - -Note: if the scenario does not support some tests, they are indicated as SKIP. -See User guide for details. + +-------------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------+------------------+----------------------+------------------+----------------+ + | rally_full_cntt | functest | benchmarking | 89:52 | PASS | + | rally_jobs_cntt | functest | benchmarking | 19:39 | PASS | + | vmtp | functest | benchmarking | 16:59 | PASS | + | shaker | functest | benchmarking | 23:43 | PASS | + +-------------------------+------------------+----------------------+------------------+----------------+ Testing vnf suite ^^^^^^^^^^^^^^^^^ @@ -186,21 +219,22 @@ Results shall be displayed as follows:: +----------------------+------------------+--------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +----------------------+------------------+--------------+------------------+----------------+ - | cloudify | functest | vnf | 06:42 | PASS | - | cloudify_ims | functest | vnf | 39:21 | PASS | - | heat_ims | functest | vnf | 34:33 | PASS | - | vyos_vrouter | functest | vnf | 20:20 | PASS | - | juju_epc | functest | vnf | 56:55 | PASS | + | cloudify | functest | vnf | 05:08 | PASS | + | cloudify_ims | functest | vnf | 24:46 | PASS | + | heat_ims | functest | vnf | 33:12 | PASS | + | vyos_vrouter | functest | vnf | 15:53 | PASS | + | juju_epc | functest | vnf | 27:52 | PASS | +----------------------+------------------+--------------+------------------+----------------+ Functest Dockers for Kubernetes deployment ------------------------------------------ Docker images are available on the dockerhub: - * opnfv/functest-kubernetes-core - * opnfv/functest-kubernetest-healthcheck - * opnfv/functest-kubernetest-smoke - * opnfv/functest-kubernetest-features + * opnfv/functest-kubernetes-healthcheck + * opnfv/functest-kubernetes-smoke + * opnfv/functest-kubernetes-security + * opnfv/functest-kubernetes-benchmarking + * opnfv/functest-kubernetes-cnf Preparing your environment ^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -226,7 +260,8 @@ Results shall be displayed as follows:: +-------------------+------------------+---------------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +-------------------+------------------+---------------------+------------------+----------------+ - | k8s_smoke | functest | healthcheck | 02:27 | PASS | + | k8s_quick | functest | healthcheck | 00:18 | PASS | + | k8s_smoke | functest | healthcheck | 01:14 | PASS | +-------------------+------------------+---------------------+------------------+----------------+ Testing smoke suite @@ -240,34 +275,72 @@ Run smoke suite:: Results shall be displayed as follows:: - +-------------------------+------------------+---------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------------+------------------+---------------+------------------+----------------+ - | k8s_conformance | functest | smoke | 57:14 | PASS | - +-------------------------+------------------+---------------+------------------+----------------+ + +---------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+---------------+------------------+----------------+ + | k8s_conformance | functest | smoke | 94:26 | PASS | + | xrally_kubernetes | functest | smoke | 13:05 | PASS | + +---------------------------+------------------+---------------+------------------+----------------+ -Testing features suite +Testing security suite ^^^^^^^^^^^^^^^^^^^^^^ -Run features suite:: +Run smoke suite:: + + sudo docker run -it --env-file env \ + -v $(pwd)/config:/root/.kube/config \ + opnfv/functest-kubernetes-security + +Results shall be displayed as follows:: + + +---------------------------+------------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+------------------+------------------+----------------+ + | kube_hunter | functest | security | 00:19 | PASS | + | kube_bench_master | functest | security | 00:02 | PASS | + | kube_bench_node | functest | security | 00:01 | PASS | + +---------------------------+------------------+------------------+------------------+----------------+ + +Testing benchmarking suite +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Run benchmarking suite:: + + sudo docker run -it --env-file env \ + -v $(pwd)/config:/root/.kube/config \ + opnfv/functest-kubernetes-benchmarking + +Results shall be displayed as follows:: + + +--------------------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------------------+------------------+----------------------+------------------+----------------+ + | xrally_kubernetes_full | functest | benchmarking | 34:16 | PASS | + +--------------------------------+------------------+----------------------+------------------+----------------+ + +Testing cnf suite +^^^^^^^^^^^^^^^^^ + +Run cnf suite:: sudo docker run -it --env-file env \ -v $(pwd)/config:/root/.kube/config \ - opnfv/functest-kubernetes-features + opnfv/functest-kubernetes-cnf Results shall be displayed as follows:: - +----------------------+------------------+------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------+------------------+------------------+------------------+----------------+ - | stor4nfv_k8s | stor4nfv | stor4nfv | 00:00 | SKIP | - | clover_k8s | clover | clover | 00:00 | SKIP | - +----------------------+------------------+------------------+------------------+----------------+ + +-------------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------+------------------+--------------+------------------+----------------+ + | k8s_vims | functest | cnf | 09:06 | PASS | + | helm_vims | functest | cnf | 08:54 | PASS | + | cnf_conformance | functest | cnf | 02:00 | PASS | + +-------------------------+------------------+--------------+------------------+----------------+ Environment variables ===================== -Several environement variables may be specified: +Several environment variables may be specified: * INSTALLER_IP=<Specific IP Address> * DEPLOY_SCENARIO=<vim>-<controller>-<nfv_feature>-<ha_mode> @@ -392,7 +465,7 @@ By default all the logs are put un /home/opnfv/functest/results/functest.log. If you want to have more logs in console, you may edit the logging.ini file manually. Connect on the docker then edit the file located in -/usr/lib/python3.6/site-packages/xtesting/ci/logging.ini +/usr/lib/python3.8/site-packages/xtesting/ci/logging.ini Change wconsole to console in the desired module to get more traces. @@ -402,8 +475,8 @@ Configuration You may also directly modify the python code or the configuration file (e.g. testcases.yaml used to declare test constraints) under -/usr/lib/python3.6/site-packages/xtesting and -/usr/lib/python3.6/site-packages/functest +/usr/lib/python3.8/site-packages/xtesting and +/usr/lib/python3.8/site-packages/functest Tips @@ -615,5 +688,5 @@ work with the above pre-requisite actions. .. _`Install Docker on CentOS`: https://docs.docker.com/engine/installation/linux/centos/ .. _`Functest User Guide`: http://docs.opnfv.org/en/stable-danube/submodules/functest/docs/testing/user/userguide/index.html .. _`images/CentOS-7-x86_64-GenericCloud.qcow2`: https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 -.. _`images/cirros-0.4.0-x86_64-disk.img`: http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img +.. _`images/cirros-0.5.1-x86_64-disk.img`: http://download.cirros-cloud.net/0.5.1/cirros-0.5.1-x86_64-disk.img .. _`images/ubuntu-14.04-server-cloudimg-amd64-disk1.img`: https://cloud-images.ubuntu.com/releases/14.04/release/ubuntu-14.04-server-cloudimg-amd64-disk1.img diff --git a/docs/testing/user/configguide/index.rst b/docs/testing/user/configguide/index.rst index e7967adb3..e5963435e 100644 --- a/docs/testing/user/configguide/index.rst +++ b/docs/testing/user/configguide/index.rst @@ -11,7 +11,6 @@ Functest Installation Guide intro.rst prerequisites.rst configguide.rst - ci.rst References ========== @@ -20,4 +19,4 @@ References IRC support channel: #opnfv-functest -.. _`[1]`: https://github.com/opnfv/releng/blob/master/jjb/functest/functest-daily-jobs.yaml +.. _`[1]`: https://github.com/opnfv/releng/blob/master/jjb/functest/functest.yaml diff --git a/docs/testing/user/configguide/prerequisites.rst b/docs/testing/user/configguide/prerequisites.rst index c7b52dc54..dab4a9e6b 100644 --- a/docs/testing/user/configguide/prerequisites.rst +++ b/docs/testing/user/configguide/prerequisites.rst @@ -92,4 +92,4 @@ should thus be known. Ensure you can reach each node in the SUT, from the Jumphost using the 'ping' command using the respective IP address on the public/external network for each node in the SUT. The details of how to determine the needed IP addresses for each node in the SUT may vary according -to the used installer and are therefore ommitted here. +to the used installer and are therefore omitted here. diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst index 07ab0e62d..1e73cd622 100644 --- a/docs/testing/user/userguide/index.rst +++ b/docs/testing/user/userguide/index.rst @@ -40,7 +40,7 @@ References `[11]`_: Robot Framework web site -`[13]`_: SNAPS wiki +`[13]`_: SNAPS `[14]`_: vRouter @@ -64,12 +64,12 @@ IRC support chan: #opnfv-functest .. _`[9]`: https://github.com/openstack/interop .. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst .. _`[11]`: https://robotframework.org/ -.. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO +.. _`[13]`: https://git.opnfv.org/snaps/ .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter .. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/ .. _`[16]`: http://testresults.opnfv.org/test/ .. _`OPNFV main site`: https://www.opnfv.org/ -.. _`Functest page`: https://wiki.opnfv.org/display/functest/Opnfv+Functional+Testing +.. _`Functest page`: https://github.com/opnfv/functest/ .. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html .. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html .. _`config_functest.yaml` : https://github.com/opnfv/functest/blob/master/functest/ci/config_functest.yaml diff --git a/docs/testing/user/userguide/reporting.rst b/docs/testing/user/userguide/reporting.rst index 7c8c48ece..8fad55d33 100644 --- a/docs/testing/user/userguide/reporting.rst +++ b/docs/testing/user/userguide/reporting.rst @@ -48,7 +48,7 @@ and features) corresponding to this scenario. All the testcases (X) listed in the table are runnable on os-odl_l2-nofeature scenarios. Please note that other test cases (e.g. sfc_odl, bgpvpn) need ODL configuration -addons and, as a consequence, specific scenario. +add-ons and, as a consequence, specific scenario. There are not considered as runnable on the generic odl_l2 scenario. @@ -82,7 +82,7 @@ Therefore the scoring provides 3 types of indicators: * the maturity: if the percentage (scoring/target scoring * 100) is high, it means that all the tests are PASS * the stability: as the number of iteration is included in the calculation, - the pecentage can be high only if the scenario is run regularly (at least + the percentage can be high only if the scenario is run regularly (at least more than 4 iterations over the last 10 days in CI) In any case, the scoring is used to give feedback to the other projects and diff --git a/docs/testing/user/userguide/test_details.rst b/docs/testing/user/userguide/test_details.rst index 0315498ce..98247d488 100644 --- a/docs/testing/user/userguide/test_details.rst +++ b/docs/testing/user/userguide/test_details.rst @@ -272,53 +272,6 @@ Note: the checks in OpenDaylight are based on the returned HTTP status code returned by OpenDaylight. -Features --------- - -Functest has been supporting several feature projects since Brahmaputra: - - -+-----------------+---------+----------+--------+-----------+-----------+ -| Test | Brahma | Colorado | Danube | Euphrates | Fraser | -+=================+=========+==========+========+===========+===========+ -| barometer | | | X | X | X | -+-----------------+---------+----------+--------+-----------+-----------+ -| bgpvpn | | X | X | X | X | -+-----------------+---------+----------+--------+-----------+-----------+ -| copper | | X | | | | -+-----------------+---------+----------+--------+-----------+-----------+ -| doctor | X | X | X | X | X | -+-----------------+---------+----------+--------+-----------+-----------+ -| domino | | X | X | X | | -+-----------------+---------+----------+--------+-----------+-----------+ -| fds | | | X | X | X | -+-----------------+---------+----------+--------+-----------+-----------+ -| moon | | X | | | | -+-----------------+---------+----------+--------+-----------+-----------+ -| multisite | | X | X | | | -+-----------------+---------+----------+--------+-----------+-----------+ -| netready | | | X | | | -+-----------------+---------+----------+--------+-----------+-----------+ -| odl_sfc | | X | X | X | X | -+-----------------+---------+----------+--------+-----------+-----------+ -| opera | | | X | | | -+-----------------+---------+----------+--------+-----------+-----------+ -| orchestra | | | X | X | X | -+-----------------+---------+----------+--------+-----------+-----------+ -| parser | | | X | X | X | -+-----------------+---------+----------+--------+-----------+-----------+ -| promise | X | X | X | X | X | -+-----------------+---------+----------+--------+-----------+-----------+ -| security_scan | | X | X | | | -+-----------------+---------+----------+--------+-----------+-----------+ -| clover | | | | | X | -+-----------------+---------+----------+--------+-----------+-----------+ -| stor4nfv | | | | | X | -+-----------------+---------+----------+--------+-----------+-----------+ - -Please refer to the dedicated feature user guides for details. - - VNF --- @@ -452,8 +405,8 @@ The kubernetes testcases are distributed across various Tiers: .. _`[9]`: https://github.com/openstack/interop .. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst .. _`[11]`: https://robotframework.org/ -.. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO +.. _`[13]`: https://git.opnfv.org/snaps/ .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter -.. _`[15]`: https://www.rebaca.com/what-we-do/abot-5g-network-simulator/ +.. _`[15]`: https://github.com/RebacaInc/abot_charm .. _`[16]`: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md .. _`[17]`: https://github.com/Metaswitch/clearwater-heat/blob/release-129/clearwater.yaml diff --git a/docs/testing/user/userguide/test_overview.rst b/docs/testing/user/userguide/test_overview.rst index e18e32748..bc3e79dcb 100644 --- a/docs/testing/user/userguide/test_overview.rst +++ b/docs/testing/user/userguide/test_overview.rst @@ -11,14 +11,13 @@ infrastructure. The current list of test suites can be distributed over 5 main domains: * VIM (Virtualised Infrastructure Manager) * Controllers (i.e. SDN Controllers) - * Features * VNF (Virtual Network Functions) * Kubernetes Functest test suites are also distributed in the OPNFV testing categories: -healthcheck, smoke, features, components, performance, VNF, Stress tests. +healthcheck, smoke, benchmarking, VNF, Stress tests. -All the Healthcheck and smoke tests of a given scenario must be succesful to +All the Healthcheck and smoke tests of a given scenario must be successful to validate the scenario for the release. +-------------+---------------+------------+----------------------------------+ @@ -88,48 +87,10 @@ validate the scenario for the release. | | | | upstream testcases. See below | | | | | for details | +-------------+---------------+------------+----------------------------------+ -| Features | features | bgpvpn | Implementation of the OpenStack | -| | | | bgpvpn API from the SDNVPN | -| | | | feature project. It allows for | -| | | | the creation of BGP VPNs. | -| | | | See `SDNVPN User Guide`_ for | -| | | | details | -| | +------------+----------------------------------+ -| | | doctor | Doctor platform, as of Colorado | -| | | | release, provides the three | -| | | | features: | -| | | | * Immediate Notification | -| | | | * Consistent resource state | -| | | | awareness for compute host down | -| | | | * Valid compute host status | -| | | | given to VM owner | -| | | | See `Doctor User Guide`_ for | -| | | | details | -| | +------------+----------------------------------+ -| | | odl-sfc | SFC testing for odl scenarios | -| | | | See `SFC User Guide`_ for details| -| | +------------+----------------------------------+ -| | | parser | Parser is an integration project | -| | | | which aims to provide | -| | | | placement/deployment templates | -| | | | translation for OPNFV platform, | -| | | | including TOSCA -> HOT, POLICY ->| -| | | | TOSCA and YANG -> TOSCA. it | -| | | | deals with a fake vRNC. | -| | | | See `Parser User Guide`_ for | -| | | | details | -| | +------------+----------------------------------+ -| | | fds | Test Suite for the OpenDaylight | -| | | | SDN Controller when the GBP | -| | | | features are installed. It | -| | | | integrates some test suites from | -| | | | upstream using Robot as the test | -| | | | framework | -+-------------+---------------+------------+----------------------------------+ | VNF | vnf | cloudify | Example of a real VNF deployment | | | | \_ims | to show the NFV capabilities of | | | | | the platform. The IP Multimedia | -| | | | Subsytem is a typical Telco test | +| | | | Subsystem is a typical Telco test| | | | | case, referenced by ETSI. | | | | | It provides a fully functional | | | | | VoIP System | @@ -154,13 +115,6 @@ validate the scenario for the release. | | smoke | k8s\_ | Run a subset of Kubernetes | | | | conformance| End-to-End tests, expected to | | | | | pass on any Kubernetes cluster | -| +---------------+------------+----------------------------------+ -| | stor4nfv | stor4nfv | Run tests necessary to | -| | | \_k8s | demonstrate conformance of the | -| | | | K8s+Stor4NFV deployment | -| +---------------+------------+----------------------------------+ -| | clover | clover_k8s | Test functionality of | -| | | | K8s+Istio+Clover deployment. | +-------------+---------------+------------+----------------------------------+ @@ -219,12 +173,11 @@ combinations (which may change from one version to another): Most of the tests are runnable by any combination, but some tests might have restrictions imposed by the utilized installers or due to the available -deployed features. The system uses the environment variables (INSTALLER_TYPE -and DEPLOY_SCENARIO) to automatically determine the valid test cases, for each -given environment. +deployed services. The system uses the environment variables to automatically +determine the valid test cases, for each given environment. A convenience Functest CLI utility is also available to simplify setting up the -Functest evironment, management of the OpenStack environment (e.g. resource +Functest environment, management of the OpenStack environment (e.g. resource clean-up) and for executing tests. The Functest CLI organised the testcase into logical Tiers, which contain in turn one or more testcases. The CLI allows execution of a single specified @@ -234,8 +187,4 @@ section. .. _`[2]`: https://docs.openstack.org/tempest/latest/ .. _`[3]`: https://rally.readthedocs.io/en/latest/index.html -.. _`Doctor User Guide`: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html -.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html -.. _`Parser User Guide`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html .. _`Functest Dashboard`: http://testresults.opnfv.org/ -.. _`SFC User Guide`: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst index be3765376..10f87d8ec 100644 --- a/docs/testing/user/userguide/test_results.rst +++ b/docs/testing/user/userguide/test_results.rst @@ -10,7 +10,7 @@ In manual mode test results are displayed in the console and result files are put in /home/opnfv/functest/results. If you want additional logs, you may configure the logging.ini under -/usr/lib/python3.6/site-packages/xtesting/ci. +/usr/lib/python3.8/site-packages/xtesting/ci. Automated testing ----------------- @@ -21,104 +21,143 @@ end of each suite and can be described as follow. Healthcheck suite:: - +----------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:07 | PASS | - | api_check | functest | healthcheck | 07:46 | PASS | - | snaps_health_check | functest | healthcheck | 00:36 | PASS | - +----------------------------+------------------+---------------------+------------------+----------------+ + +--------------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------------+------------------+---------------------+------------------+----------------+ + | connection_check | functest | healthcheck | 00:03 | PASS | + | tenantnetwork1 | functest | healthcheck | 00:05 | PASS | + | tenantnetwork2 | functest | healthcheck | 00:06 | PASS | + | vmready1 | functest | healthcheck | 00:06 | PASS | + | vmready2 | functest | healthcheck | 00:08 | PASS | + | singlevm1 | functest | healthcheck | 00:32 | PASS | + | singlevm2 | functest | healthcheck | 00:37 | PASS | + | vping_ssh | functest | healthcheck | 00:46 | PASS | + | vping_userdata | functest | healthcheck | 00:39 | PASS | + | cinder_test | functest | healthcheck | 01:05 | PASS | + | tempest_smoke | functest | healthcheck | 05:39 | PASS | + | tempest_horizon | functest | healthcheck | 01:05 | PASS | + | odl | functest | healthcheck | 00:00 | SKIP | + +--------------------------+------------------+---------------------+------------------+----------------+ Smoke suite:: - +------------------------------+------------------+---------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +------------------------------+------------------+---------------+------------------+----------------+ - | vping_ssh | functest | smoke | 00:57 | PASS | - | vping_userdata | functest | smoke | 00:33 | PASS | - | tempest_smoke_serial | functest | smoke | 13:22 | PASS | - | rally_sanity | functest | smoke | 24:07 | PASS | - | refstack_defcore | functest | smoke | 05:21 | PASS | - | patrole | functest | smoke | 04:29 | PASS | - | snaps_smoke | functest | smoke | 46:54 | PASS | - | odl | functest | smoke | 00:00 | SKIP | - | neutron_trunk | functest | smoke | 00:00 | SKIP | - +------------------------------+------------------+---------------+------------------+----------------+ - -Features suite:: - - +-----------------------------+------------------------+------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-----------------------------+------------------------+------------------+------------------+----------------+ - | doctor-notification | doctor | features | 00:00 | SKIP | - | bgpvpn | sdnvpn | features | 00:00 | SKIP | - | functest-odl-sfc | sfc | features | 00:00 | SKIP | - | barometercollectd | barometer | features | 00:00 | SKIP | - | fds | fastdatastacks | features | 00:00 | SKIP | - +-----------------------------+------------------------+------------------+------------------+----------------+ - -Components suite:: - - +-------------------------------+------------------+--------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------------------+------------------+--------------------+------------------+----------------+ - | tempest_full_parallel | functest | components | 48:28 | PASS | - | rally_full | functest | components | 126:02 | PASS | - +-------------------------------+------------------+--------------------+------------------+----------------+ + +---------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+---------------+------------------+----------------+ + | tempest_neutron | functest | smoke | 15:30 | PASS | + | tempest_cinder | functest | smoke | 02:01 | PASS | + | tempest_keystone | functest | smoke | 01:17 | PASS | + | tempest_heat | functest | smoke | 22:14 | PASS | + | tempest_telemetry | functest | smoke | 00:00 | SKIP | + | rally_sanity | functest | smoke | 17:24 | PASS | + | refstack_compute | functest | smoke | 07:03 | PASS | + | refstack_object | functest | smoke | 02:09 | PASS | + | refstack_platform | functest | smoke | 07:31 | PASS | + | tempest_full | functest | smoke | 41:52 | PASS | + | tempest_scenario | functest | smoke | 08:42 | PASS | + | tempest_slow | functest | smoke | 43:42 | PASS | + | patrole_admin | functest | smoke | 21:06 | PASS | + | patrole_member | functest | smoke | 21:23 | PASS | + | patrole_reader | functest | smoke | 21:56 | PASS | + | tempest_barbican | functest | smoke | 02:30 | PASS | + | tempest_octavia | functest | smoke | 00:00 | SKIP | + | tempest_cyborg | functest | smoke | 00:00 | SKIP | + +---------------------------+------------------+---------------+------------------+----------------+ + +Smoke CNTT suite:: + + +-------------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------------+------------------+---------------+------------------+----------------+ + | tempest_neutron_cntt | functest | smoke | 11:35 | PASS | + | tempest_cinder_cntt | functest | smoke | 01:58 | PASS | + | tempest_keystone_cntt | functest | smoke | 01:13 | PASS | + | tempest_heat_cntt | functest | smoke | 22:32 | PASS | + | rally_sanity_cntt | functest | smoke | 17:16 | PASS | + | tempest_full_cntt | functest | smoke | 41:13 | PASS | + | tempest_scenario_cntt | functest | smoke | 08:57 | PASS | + | tempest_slow_cntt | functest | smoke | 35:58 | PASS | + +-------------------------------+------------------+---------------+------------------+----------------+ + +Benchmarking suite:: + + +--------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------+------------------+----------------------+------------------+----------------+ + | rally_full | functest | benchmarking | 93:03 | PASS | + | rally_jobs | functest | benchmarking | 27:05 | PASS | + | vmtp | functest | benchmarking | 17:56 | PASS | + | shaker | functest | benchmarking | 24:02 | PASS | + +--------------------+------------------+----------------------+------------------+----------------+ + +Benchmarking CNTT suite:: + + +-------------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------+------------------+----------------------+------------------+----------------+ + | rally_full_cntt | functest | benchmarking | 89:52 | PASS | + | rally_jobs_cntt | functest | benchmarking | 19:39 | PASS | + | vmtp | functest | benchmarking | 16:59 | PASS | + | shaker | functest | benchmarking | 23:43 | PASS | + +-------------------------+------------------+----------------------+------------------+----------------+ Vnf suite:: - +----------------------+------------------+--------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------+------------------+--------------+------------------+----------------+ - | cloudify_ims | functest | vnf | 28:15 | PASS | - | vyos_vrouter | functest | vnf | 17:59 | PASS | - | juju_epc | functest | vnf | 46:44 | PASS | - +----------------------+------------------+--------------+------------------+----------------+ - -Parser testcase:: - - +-----------------------+-----------------+------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-----------------------+-----------------+------------------+------------------+----------------+ - | parser-basics | parser | features | 00:00 | SKIP | - +-----------------------+-----------------+------------------+------------------+----------------+ - -Functest Kubernetes test result:: - - +--------------------------------------+------------------------------------------------------------+ - | ENV VAR | VALUE | - +--------------------------------------+------------------------------------------------------------+ - | INSTALLER_TYPE | compass | - | DEPLOY_SCENARIO | k8-nosdn-nofeature-ha | - | BUILD_TAG | jenkins-functest-compass-baremetal-daily-master-75 | - | CI_LOOP | daily | - +--------------------------------------+------------------------------------------------------------+ + +----------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +----------------------+------------------+--------------+------------------+----------------+ + | cloudify | functest | vnf | 05:08 | PASS | + | cloudify_ims | functest | vnf | 24:46 | PASS | + | heat_ims | functest | vnf | 33:12 | PASS | + | vyos_vrouter | functest | vnf | 15:53 | PASS | + | juju_epc | functest | vnf | 27:52 | PASS | + +----------------------+------------------+--------------+------------------+----------------+ Kubernetes healthcheck suite:: - +-------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------+------------------+---------------------+------------------+----------------+ - | k8s_smoke | functest | healthcheck | 01:54 | PASS | - +-------------------+------------------+---------------------+------------------+----------------+ + +-------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------+------------------+---------------------+------------------+----------------+ + | k8s_quick | functest | healthcheck | 00:18 | PASS | + | k8s_smoke | functest | healthcheck | 01:14 | PASS | + +-------------------+------------------+---------------------+------------------+----------------+ Kubernetes smoke suite:: - +-------------------------+------------------+---------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------------+------------------+---------------+------------------+----------------+ - | k8s_conformance | functest | smoke | 57:47 | PASS | - +-------------------------+------------------+---------------+------------------+----------------+ - -Kubernetes features suite:: - - +----------------------+------------------+------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------+------------------+------------------+------------------+----------------+ - | stor4nfv_k8s | stor4nfv | stor4nfv | 00:00 | SKIP | - | clover_k8s | clover | clover | 00:00 | SKIP | - +----------------------+------------------+------------------+------------------+----------------+ + +---------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+---------------+------------------+----------------+ + | k8s_conformance | functest | smoke | 94:26 | PASS | + | xrally_kubernetes | functest | smoke | 13:05 | PASS | + +---------------------------+------------------+---------------+------------------+----------------+ + +Kubernetes security suite:: + + +---------------------------+------------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+------------------+------------------+----------------+ + | kube_hunter | functest | security | 00:19 | PASS | + | kube_bench_master | functest | security | 00:02 | PASS | + | kube_bench_node | functest | security | 00:01 | PASS | + +---------------------------+------------------+------------------+------------------+----------------+ + +Kubernetes benchmarking suite:: + + +--------------------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------------------+------------------+----------------------+------------------+----------------+ + | xrally_kubernetes_full | functest | benchmarking | 34:16 | PASS | + +--------------------------------+------------------+----------------------+------------------+----------------+ + +Kubernetes cnf suite:: + + +-------------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------+------------------+--------------+------------------+----------------+ + | k8s_vims | functest | cnf | 09:06 | PASS | + | helm_vims | functest | cnf | 08:54 | PASS | + | cnf_conformance | functest | cnf | 02:00 | PASS | + +-------------------------+------------------+--------------+------------------+----------------+ Results are automatically pushed to the test results database, some additional result files are pushed to OPNFV artifact web sites. diff --git a/docs/testing/user/userguide/troubleshooting.rst b/docs/testing/user/userguide/troubleshooting.rst index 279a47398..d857ed4c4 100644 --- a/docs/testing/user/userguide/troubleshooting.rst +++ b/docs/testing/user/userguide/troubleshooting.rst @@ -44,8 +44,8 @@ follows:: The Functest CLI is designed to route a call to the corresponding internal python scripts, located in paths:: - /usr/lib/python3.6/site-packages/functest/opnfv_tests/openstack/vping/vping_ssh.py - /usr/lib/python3.6/site-packages/functest/opnfv_tests/openstack/vping/vping_userdata.py + /usr/lib/python3.8/site-packages/functest/opnfv_tests/openstack/vping/vping_ssh.py + /usr/lib/python3.8/site-packages/functest/opnfv_tests/openstack/vping/vping_userdata.py Notes: @@ -102,7 +102,7 @@ In this case, proceed to create it manually. These are some hints:: --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0 The next step is to create the instances. The image used is located in -*/home/opnfv/functest/data/cirros-0.4.0-x86_64-disk.img* and a Glance image is +*/home/opnfv/functest/data/cirros-0.5.1-x86_64-disk.img* and a Glance image is created with the name **functest-vping**. If booting the instances fails (i.e. the status is not **ACTIVE**), you can check why it failed by doing:: @@ -125,7 +125,7 @@ This test case creates a floating IP on the external network and assigns it to the second instance **opnfv-vping-2**. The purpose of this is to establish a SSH connection to that instance and SCP a script that will ping the first instance. This script is located in the repository under -/usr/lib/python3.6/site-packages/functest/opnfv_tests/openstack/vping/ping.sh +/usr/lib/python3.8/site-packages/functest/opnfv_tests/openstack/vping/ping.sh and takes an IP as a parameter. When the SCP is completed, the test will do a SSH call to that script inside the second instance. Some problems can happen here:: @@ -133,8 +133,8 @@ here:: vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting If this is displayed, stop the test or wait for it to finish, if you have used -the special method of test invocation with specific supression of OpenStack -resource clean-up, as explained earler. It means that the Container can not +the special method of test invocation with specific suppression of OpenStack +resource clean-up, as explained earlier. It means that the Container can not reach the Public/External IP assigned to the instance **opnfv-vping-2**. There are many possible reasons, and they really depend on the chosen scenario. For most of the ODL-L3 and ONOS scenarios this has been noticed and it is a known @@ -158,7 +158,7 @@ container:: ping <public IP> If the ping does not return anything, try to ping from the Host where the -Docker container is running. If that solves the problem, check the iptable +Docker container is running. If that solves the problem, check the iptables rules because there might be some rules rejecting ICMP or TCP traffic coming/going from/to the container. @@ -261,40 +261,28 @@ Functest offers a possibility to test a customized list of Tempest test cases. To enable that, add a new entry in docker/smoke/testcases.yaml on the "smoke" container with the following content:: - - - case_name: tempest_custom - project_name: functest - criteria: 100 - blocking: false - description: >- - The test case allows running a customized list of tempest - test cases - dependencies: - installer: '' - scenario: '' - run: - module: 'functest.opnfv_tests.openstack.tempest.tempest' - class: 'TempestCustom' - -Also, a list of the Tempest test cases must be provided to the container or -modify the existing one in -/usr/lib/python3.6/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt - -Example of custom list of tests 'my-custom-tempest-tests.txt':: - - tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops[compute,id-7fff3fb3-91d8-4fd0-bd7d-0204f1f180ba,network,smoke] - tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops[compute,id-f323b3ba-82f8-4db7-8ea6-6a895869ec49,network,smoke] + - + case_name: tempest_custom + project_name: functest + criteria: 100 + blocking: false + description: >- + The test case allows running a customized list of tempest + test cases + run: + name: tempest_common + args: + mode: "tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops|\ + tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops" This is an example of running a customized list of Tempest tests in Functest:: sudo docker run --env-file env \ -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ -v $(pwd)/images:/home/opnfv/functest/images \ - -v $(pwd)/my-custom-testcases.yaml:/usr/lib/python3.6/site-packages/functest/ci/testcases.yaml \ - -v $(pwd)/my-custom-tempest-tests.txt:/usr/lib/python3.6/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt \ + -v $(pwd)/my-custom-testcases.yaml:/usr/lib/python3.8/site-packages/xtesting/ci/testcases.yaml \ opnfv/functest-smoke run_tests -t tempest_custom - Rally ^^^^^ @@ -314,7 +302,7 @@ Possible scenarios are: To know more about what those scenarios are doing, they are defined in directory: -/usr/lib/python3.6/site-packages/functest/opnfv_tests/openstack/rally/scenario +/usr/lib/python3.8/site-packages/functest/opnfv_tests/openstack/rally/scenario For more info about Rally scenario definition please refer to the Rally official documentation. `[3]`_ |