aboutsummaryrefslogtreecommitdiffstats
path: root/functest/opnfv_tests/openstack/tempest
AgeCommit message (Collapse)AuthorFilesLines
2020-04-19Run sequentially tempest_heatCédric Ollivier1-0/+2
It's forced by design else it remains a few resources. It also sets fixed_subnet_name and network_for_ssh to avoid falsy discovery as it seems [1] [1] http://artifacts.opnfv.org/functest/X5GDYF3B0R2B/functest-opnfv-functest-smoke-jerma-tempest_heat-run-18/tempest_heat/tempest-report.html Change-Id: Ieb584ddc1c0b4ebb543e9d69277a20aaee0fbac0 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2020-04-11Disable VolumeBackupRestoreIntegrationTest if OVNCédric Ollivier1-1/+2
It completes "Skip VolumeBackupRestoreIntegrationTest in temptest_heat" Change-Id: I798cefa3a3648277cc321cc9e85be9b634aa9e20 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2020-04-07Skip VolumeBackupRestoreIntegrationTest in temptest_heatCédric Ollivier1-1/+2
It's linked to vbd which is false in many cases [1]. e.g. IMAGE_PROPERTIES=hw_scsi_model:virtio-scsi,hw_disk_bus:scsi Enabling it requires upstream changes to configure the right device. [1] http://artifacts.opnfv.org/functest/FHHSBCPDUV6C/functest-opnfv-functest-smoke-latest-tempest_heat-run-7/tempest_heat/tempest-report.html Change-Id: I2632779e96b7b0850d1937a8ec3c7bf4607862a5 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2020-04-06Add tempest_heat (heat-tempest-plugin)Cédric Ollivier3-0/+76
It asks for the Fedora-Cloud-Base image and a second user account. It should be noted that heat-tempest-plugin duplicates all configs from tempest. Minimal image is still Cirros and the first account is the admin one. Then it conforms with the TempestCommon logic. AodhAlarmTest is skipped by default because pre conditions are missing. SoftwareConfigIntegrationTest is skipped as in gates because it requires a custom image [1] [1] https://github.com/openstack/heat/blob/master/devstack/lib/heat#L444 Change-Id: I8b39dc65ef3714411cd828e17b95e124f2e90f34 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2020-04-06Add swift sections in OVN's tempest_conf.yamlCédric Ollivier1-0/+5
http://artifacts.opnfv.org/functest/4LM7FMUVFMML/functest-ovn-opnfv-functest-smoke-jerma-tempest_slow-run-41/tempest_slow/tempest-report.html Change-Id: I1a3f684a9059eac21695bd655df976bceebf26f0 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2020-04-05Enable TrunkTestInheritJSONBase.test_add_subport if ovnCédric Ollivier1-1/+0
It has been fixed upstream by "Protect vs unordered results in TrunkTestInheritJSONBase" [1]. [1] https://review.opendev.org/#/c/708305/5 Change-Id: Ibbc23a53d9d20d9b3a192e15779ca0f3334e2a38 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2020-04-04Skip test_container_synchronizationCédric Ollivier1-0/+5
It's not supported by radosgw which is now used in Functest SUTs [1][2]. It's removed from CNTT which seems leveraging on ceph (at least RI) [3] [1] https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3.2/html/release_notes/bug-fixes [2] http://artifacts.opnfv.org/functest/E46RCZLBRYCC/functest-opnfv-functest-smoke-hunter-tempest_slow-run-127/tempest_slow/tempest-report.html [3] http://artifacts.opnfv.org/cntt/WKFRPQODPW34/cntt-opnfv-functest-smoke-latest-tempest_slow-run-37/tempest_slow/tempest-report.html Change-Id: I8127df786c41af0d0285ef0e5cd1b28901685f65 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2020-03-23Doesn't redirect stderr when getting verifier idCédric Ollivier1-1/+1
The logic fails when rally produces warnings [1]. It also updates Rally OpenStack and checks if the previous error has been solved[2]. TypeError: _discover_or_create_flavor() missing 1 required positional argument: 'flv_disk' [3] [1] https://build.opnfv.org/ci/job/functest-opnfv-functest-healthcheck-latest-tempest_smoke-run/757/console [2] https://github.com/openstack/rally-openstack/commit/e62315115af20e53f6c7205494ba0008634047ed [3] https://build.opnfv.org/ci/job/functest-opnfv-functest-healthcheck-latest-tempest_smoke-run/629/console Change-Id: Ia1daaa76d7cef7f30904c1f3bd42e2f6a6ba10af Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2020-02-22Blacklist tests asking for vxlanCédric Ollivier1-0/+5
http://artifacts.opnfv.org/functest/B4HOI3WWLNVK/functest-ovn-opnfv-functest-smoke-jerma-neutron-tempest-plugin-api-run-4/neutron-tempest-plugin-api/tempest-report.html Change-Id: I6ff82b92cb4c392ac1f453faf7e16ef587dc2060 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2020-02-22Publish tempest_conf.yaml for Functest OVN SUTCédric Ollivier1-0/+78
Change-Id: I96237e8bfe469f100dc72d022747ce0013b41284 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2020-02-21Update patrole regex (OVN)Cédric Ollivier1-2/+6
It takes into account last changes in Patrole. It was not covered since Apex stopped deploying OVN. Change-Id: I7ab8362f9998024ae65f49973b169b95bbfd4fb5 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2020-02-18Blacklick test_add_subport if OVNCédric Ollivier1-0/+1
https://bugs.launchpad.net/neutron/+bug/1863707 Change-Id: Ieebe10ee2a9040887d84aca3e8fb5ee66294634b Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2020-02-15Run test_agent_management vs OVNCédric Ollivier1-2/+0
They must not longer be blacklisted as opposed to test_dhcp_agent_scheduler [1]. https://bugs.launchpad.net/networking-ovn/+bug/1863432 Change-Id: Ib08bb87580a10e2124faa1b3853f3e27dcb3a0a0 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-12-27Set port_security: true in tempest_conf.yamlCédric Ollivier1-0/+4
It allows running test_boot_into_disabled_port_security_network_without_secgroup. It also sets a few default values. Change-Id: I851c5d13705e0419a7e664e05d32eec0efb417c6 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-12-27Set create_shared_resources to trueCédric Ollivier1-0/+1
It takes last Neutron Tempest plugin change into account [1]. It also increases test count. [1] https://review.opendev.org/#/c/692822/ Change-Id: I5e7218dd808069a0be6d481447576099aa9c63e9 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-12-03Disable tempest serial_console testingCédric Ollivier1-1/+1
DevStack seems incorrect when configuring serial console in case of multinode deployment (missing proxyclient_address in master, missing socket in Rocky, etc). Then the testing is disabled. Change-Id: I9d1d43017b0935b23ea2a939748b5f709212c70d Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-26Complete tempest_full for CNTT ComplianceCédric Ollivier1-1/+3
It forbids skipping any test and counts the testcases executed. It mosly avoids running tests accoring to the upstream skip tests (known bugs), related to nova-network, glance v1, keystone v2 and all features defined as optional by CNTT. It fixes an issue highlighted by yallint. Change-Id: Ib4678cb9e9fc37b16fcf21d8418931eae0f667d0 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-26Update scheduler_available_filtersCédric Ollivier2-7/+1
Else few tests are skipped if DifferentHostFilter is not available. Change-Id: Idfc524216dc8503e590d4fed569a43e99ea22123 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-22Verify additional Nova featuresCédric Ollivier1-3/+26
Our SUT allows verifying the next Nova features - shelve - vnc_console - cold_migration attach_encrypted_volume is now False because of ceph. It writes the tempest default values to ease highlighting the overall configuration. Change-Id: Ib9ebc8e5b390ea99edf2c3aacbd873c5eb6e7abc Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-22Set manage_volume_ref and manage_snapshot_refCédric Ollivier1-0/+2
Ceph doesn't conform with the LVM driver and asks for modifying the manage volume and snapshop refs [1]. It allows passing test_unmanage_manage_snapshot [2]. [1] https://opendev.org/openstack/tempest/commit/ebc752b59dd3e2164807930e31b313a92bce70d5 [2] http://artifacts.opnfv.org/functest/functest-opnfv-functest-smoke-latest-tempest_full-run-480/tempest_full/tempest-report.html Change-Id: I8d3b2484ba5cbd203c32fa446fa27e17a80888d5 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-21Conform with latest SUT updatesCédric Ollivier1-2/+2
Ceph is now used as backend which requires few adaptations for live migration. It differs from the main OpenStack gates but suits CNTT and most of the feedbacks received from enduser deployments. Change-Id: If6b76a7d4ec09f90467f3e63618c15a9b413bfbd Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-15Test identity project_tags and application_credentialsCédric Ollivier1-0/+5
Both are supported by OpenStack and Functest gates. It also sets the default values to highlight the coverage. Change-Id: Iae080a8bde0c411acc7811e7a0e70e3471db8c98 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-10Add features volume testingCédric Ollivier1-0/+8
The next features are supported by Functest SUT: - backup - snapshot - clone - manage_snapshot - manage_volume - volume_revert Change-Id: I0e83bb616a56b9e8651115bebf2373f85c642260 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-10Set provider_vlans in tempest Neutron_plugin_optionsCédric Ollivier1-0/+1
Few tests are skipped if this value is unset. Then it's needed to fully cover Neutron Trunk (Compliance). Change-Id: Ife4becb7ff8b47c2d90fc6c5eb4f585685205ed6 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-26Add healthcheck Horizon testsCédric Ollivier1-0/+15
It runs TestDashboardBasicOps as proposed by tempest-horizon [1]. It's skipped if DASHBOARD_URL is unset (default). [1] https://opendev.org/openstack/tempest-horizon Change-Id: I1b6b5609412770609f412da4927641588b173ed6 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-25Merge "Add neutron_plugin_options section in tempest.conf"Cedric Ollivier1-0/+3
2019-10-24Merge "Allow checking Tests count"Cedric Ollivier1-0/+4
2019-10-24Add neutron_plugin_options section in tempest.confCédric Ollivier1-0/+3
It avoids skipping the next 2 tests [1]: neutron_tempest_plugin.api.admin.test_agent_availability_zone.AgentAvailabilityZoneTestCase.test_agents_availability_zone neutron_tempest_plugin.api.admin.test_networks.NetworksTestAdmin.test_create_tenant_network_vxlan [1] http://artifacts.opnfv.org/functest/functest-opnfv-functest-smoke-latest-neutron-tempest-plugin-api-run-498/results/neutron-tempest-plugin-api/tempest-report.html Change-Id: I83f3f44302336045e47514e0aa0b67737161f0e1 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-23Allow checking Tests countCédric Ollivier1-0/+4
It meets CNTT requirements (verifying conformance with a reference implementation) and may protect vs synchronization issues (see gambia refstack_defcore). Change-Id: I842bc4ed91eec6bc545c489e3686d0d41dae7b44 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-23Allow blacklisting tests if public endpoint onlyCédric Ollivier2-3/+24
It allows reusing the blacklisting system if only public endpoints are reachable (ONAP Openlab testing). It contains a first list of tempest-based tests which force the use of admin endpoints. Change-Id: Ic1ab07dec3ae1f2a0ca50b80e08cd0ac396f2e56 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-08Count all hypervisors by defaultCédric Ollivier1-1/+1
If one hypervisor is down, few tempest test fails. User is free to set SKIP_DOWN_HYPERVISORS if it's down on purpose. Change-Id: I7b6a4d0d8f67755d8c1550fd1bc6fd707634f68b Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-09-30Count all active hypervisorsCédric Ollivier1-2/+1
Else tempest_slow fails if one hypervisor is down (detected in ONAP Openlab). It also improves Shaker, Rally and Vmtp which count them. Change-Id: Iee7c20e0357d9237501c5c451580d8f6409ac86b Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-09-19Remove any cores quota in tempest_scenario and shakerCédric Ollivier1-1/+3
Both can reach the default limit if lots of computes available. Co-Authored-By: Vincent Huet <vincent.huet@orange.com> Change-Id: Ifd8a1080a08f55bf4b0efe1858e79fa4aff75895 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-09-14Remove any instance quota in tempest_scenarioCédric Ollivier1-0/+12
The next testcase can reach the default limit if lots of computes available [1]. tempest.scenario.test_server_multinode.TestServerMultinode.test_schedule_to_all_nodes [1] http://artifacts.opnfv.org/functest/functest-opnfv-functest-smoke-hunter-tempest_scenario-run-325/results/tempest_scenario/test_raw_list.txt Change-Id: Ibf68e2a367199afee3f6198ebf683be6cd938d94 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-08-28Merge "Add Octavia testing"Cedric Ollivier1-0/+2
2019-08-27Add Octavia testingCédric Ollivier1-0/+2
It integrates Octavia Tempest Plugin. The new testcase octavia currently covers API only [1]. [1] https://logs.opendev.org/46/674846/2/check/octavia-v2-dsvm-noop-api/fe37c58/testr_results.html.gz Change-Id: I2a11a19aa0c43701499dc9e17bd69b9387b53b58 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-08-26Support regex in tempest blacklistJuha Kosonen1-5/+4
The expression both in the test cases as well as applicable scenarios support regex in blacklist. Change-Id: Id76297a1cfbf908bda67641bc7ba686db83d76e6 Signed-off-by: Juha Kosonen <juha.kosonen@nokia.com>
2019-07-27Switch to Python 3.7 and Alpine 3.10Cédric Ollivier1-10/+10
It also allows building docs as doc8 is broken due to latest OpenStack's upper-constraints. pylint is updated to 2.3.1 (lastest py3.7 version) It disables perm as umask is currently false on lf-virtual1. Change-Id: If09d4796d48b7d0591e7926621d3bbf5ca1b6c24 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-06-02Publish rally logsCédric Ollivier1-22/+2
Both Rally and Tempest publish Rally debug logs. It also moves the logics from tempest to rally. Change-Id: I5c057d830202baddd73577ade1b09ce304e3c5a5 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-05-25Set utf-8 in decode and encode callsCédric Ollivier1-11/+11
It eases backporting in iruya and hunter as a few logs can't be decoded as ascii [1]. [1] https://build.opnfv.org/ci/job/functest-opnfv-functest-vnf-iruya-juju_epc-run/56/console Change-Id: I52a6f9a25130cbf62e8ce3b9972064ab73177665 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-05-24Decode Bytes in logging callsCédric Ollivier1-5/+5
It fixes a few incorrect types in log calls [1]. [1] https://build.opnfv.org/ci/view/functest/job/functest-latest-daily/163/ Change-Id: I01790f6454694f434d31366f9c1cf9d98ff5a9cf Signed-off-by: Cédric Ollivier <ollivier.cedric@gmail.com>
2019-05-13Update to Python3Cédric Ollivier1-14/+14
Functest containers leverage on Python3 instead of python2. https://mail.python.org/pipermail/python-dev/2018-March/152348.html It also updates robotframework librairies to latest release and Vmtp to master ([1] is needed) It patches cloudify rest client to support python3. Vmtp is currently disabled because it currently supports python2 only. [1] https://github.com/openstack/vmtp/commit/a5d062881d91bf4f547d92c6e289bea30feb5d6e#diff-b4ef698db8ca845e5845c4618278f29a Change-Id: I39964a212ec2d4dbf53c9ea7c63f02e0b6a05b48 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-05-04Enable volume backup testingCédric Ollivier1-1/+1
It conforms with OpenStack gates by default. Change-Id: Icc1537b0f8342666d2a2a0de4655ba99f4f42470 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-04-30Use block devices for live migrationCédric Ollivier1-0/+2
The default values conform with OpenStack and Functest gates [1]. It could be set to false if shared storage-based live migration is supported [2]. It also currently disables cold migration. [1] http://logs.openstack.org/84/632984/24/check/tempest-multinode-full-py3/4216037/controller/logs/tempest_conf.txt.gz [2] https://docs.openstack.org/nova/pike/admin/configuring-migrations.html Change-Id: I0faa415eba610c20ba54353ac7ba7b0e83722c32 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-04-24Refactor modules to avoid duplicated codeCédric Ollivier1-12/+1
It also increases number of lines to warm about duplicated codes. Change-Id: I3914da8a90cbfe8fd024e9944c3adc8a617330d3 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-04-22Set max_microversion for placement and volumeCédric Ollivier1-0/+3
It sets max_microversion to latest because if both values are not specified, Tempest avoids tests which require a microversion [1]. It completes the previous change about nova [2]. [1] https://docs.openstack.org/tempest/latest/sampleconf.html [2] https://gerrit.opnfv.org/gerrit/#/c/67318/ Change-Id: Ieab56a239077476dd9a0e1ee97ac4293c4075f5c Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-04-15Stop forcing admin_scope_domain = TrueCédric Ollivier1-1/+0
tempest fails vs devstack stable/stein and it passes successfully vs devstack stable/rocky (Functest gates) without that option. It may be added to tempest_conf.yaml if SUT requires it. Change-Id: I4730c25b25403fd130d71166a8778e06fd479935 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-04-10Apply blacklist for OVN noha scenario as wellMichael Polenchuk1-0/+1
Change-Id: I1dd51a54f39967fcf6b3a522e2a548f81e5646a9 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2019-03-21Update RefStack list to 2018.11Cédric Ollivier1-0/+2
It sets max_microversion to latest because if both values are not specified, Tempest avoids tests which require a microversion [1]. Then test_keypairsv22_create_list_show_with_type is skipped and refstack_defcore fails [2]. It also partially reverts "Allow building containers when refstack.openstack.org is down" [3]. [1] https://docs.openstack.org/tempest/latest/sampleconf.html [2] http://artifacts.opnfv.org/functest/functest-opnfv-functest-smoke-latest-refstack_defcore-run-147/results/refstack_defcore/tempest-report.html [3] https://gerrit.opnfv.org/gerrit/#/c/67232/ Change-Id: Ia87543bb476d3047684e71ddfd05c2aa3f10bbfe Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-03-19Do not allow testcase skipping in refstackJuha Kosonen1-0/+8
Set the result of refstack_defcore as failed if there were skipped testcases in a run. Change-Id: Id782c2dee511cefbafd5b4440f386fd66f934a47 Signed-off-by: Juha Kosonen <juha.kosonen@nokia.com>