aboutsummaryrefslogtreecommitdiffstats
path: root/functest/opnfv_tests/openstack/tempest
AgeCommit message (Collapse)AuthorFilesLines
2019-10-23Allow blacklisting tests if public endpoint onlyCédric Ollivier2-3/+24
It allows reusing the blacklisting system if only public endpoints are reachable (ONAP Openlab testing). It contains a first list of tempest-based tests which force the use of admin endpoints. Change-Id: Ic1ab07dec3ae1f2a0ca50b80e08cd0ac396f2e56 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-08Count all hypervisors by defaultCédric Ollivier1-1/+1
If one hypervisor is down, few tempest test fails. User is free to set SKIP_DOWN_HYPERVISORS if it's down on purpose. Change-Id: I7b6a4d0d8f67755d8c1550fd1bc6fd707634f68b Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-09-30Count all active hypervisorsCédric Ollivier1-2/+1
Else tempest_slow fails if one hypervisor is down (detected in ONAP Openlab). It also improves Shaker, Rally and Vmtp which count them. Change-Id: Iee7c20e0357d9237501c5c451580d8f6409ac86b Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-09-19Remove any cores quota in tempest_scenario and shakerCédric Ollivier1-1/+3
Both can reach the default limit if lots of computes available. Co-Authored-By: Vincent Huet <vincent.huet@orange.com> Change-Id: Ifd8a1080a08f55bf4b0efe1858e79fa4aff75895 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-09-14Remove any instance quota in tempest_scenarioCédric Ollivier1-0/+12
The next testcase can reach the default limit if lots of computes available [1]. tempest.scenario.test_server_multinode.TestServerMultinode.test_schedule_to_all_nodes [1] http://artifacts.opnfv.org/functest/functest-opnfv-functest-smoke-hunter-tempest_scenario-run-325/results/tempest_scenario/test_raw_list.txt Change-Id: Ibf68e2a367199afee3f6198ebf683be6cd938d94 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-08-28Merge "Add Octavia testing"Cedric Ollivier1-0/+2
2019-08-27Add Octavia testingCédric Ollivier1-0/+2
It integrates Octavia Tempest Plugin. The new testcase octavia currently covers API only [1]. [1] https://logs.opendev.org/46/674846/2/check/octavia-v2-dsvm-noop-api/fe37c58/testr_results.html.gz Change-Id: I2a11a19aa0c43701499dc9e17bd69b9387b53b58 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-08-26Support regex in tempest blacklistJuha Kosonen1-5/+4
The expression both in the test cases as well as applicable scenarios support regex in blacklist. Change-Id: Id76297a1cfbf908bda67641bc7ba686db83d76e6 Signed-off-by: Juha Kosonen <juha.kosonen@nokia.com>
2019-07-27Switch to Python 3.7 and Alpine 3.10Cédric Ollivier1-10/+10
It also allows building docs as doc8 is broken due to latest OpenStack's upper-constraints. pylint is updated to 2.3.1 (lastest py3.7 version) It disables perm as umask is currently false on lf-virtual1. Change-Id: If09d4796d48b7d0591e7926621d3bbf5ca1b6c24 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-06-02Publish rally logsCédric Ollivier1-22/+2
Both Rally and Tempest publish Rally debug logs. It also moves the logics from tempest to rally. Change-Id: I5c057d830202baddd73577ade1b09ce304e3c5a5 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-05-25Set utf-8 in decode and encode callsCédric Ollivier1-11/+11
It eases backporting in iruya and hunter as a few logs can't be decoded as ascii [1]. [1] https://build.opnfv.org/ci/job/functest-opnfv-functest-vnf-iruya-juju_epc-run/56/console Change-Id: I52a6f9a25130cbf62e8ce3b9972064ab73177665 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-05-24Decode Bytes in logging callsCédric Ollivier1-5/+5
It fixes a few incorrect types in log calls [1]. [1] https://build.opnfv.org/ci/view/functest/job/functest-latest-daily/163/ Change-Id: I01790f6454694f434d31366f9c1cf9d98ff5a9cf Signed-off-by: Cédric Ollivier <ollivier.cedric@gmail.com>
2019-05-13Update to Python3Cédric Ollivier1-14/+14
Functest containers leverage on Python3 instead of python2. https://mail.python.org/pipermail/python-dev/2018-March/152348.html It also updates robotframework librairies to latest release and Vmtp to master ([1] is needed) It patches cloudify rest client to support python3. Vmtp is currently disabled because it currently supports python2 only. [1] https://github.com/openstack/vmtp/commit/a5d062881d91bf4f547d92c6e289bea30feb5d6e#diff-b4ef698db8ca845e5845c4618278f29a Change-Id: I39964a212ec2d4dbf53c9ea7c63f02e0b6a05b48 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-05-04Enable volume backup testingCédric Ollivier1-1/+1
It conforms with OpenStack gates by default. Change-Id: Icc1537b0f8342666d2a2a0de4655ba99f4f42470 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-04-30Use block devices for live migrationCédric Ollivier1-0/+2
The default values conform with OpenStack and Functest gates [1]. It could be set to false if shared storage-based live migration is supported [2]. It also currently disables cold migration. [1] http://logs.openstack.org/84/632984/24/check/tempest-multinode-full-py3/4216037/controller/logs/tempest_conf.txt.gz [2] https://docs.openstack.org/nova/pike/admin/configuring-migrations.html Change-Id: I0faa415eba610c20ba54353ac7ba7b0e83722c32 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-04-24Refactor modules to avoid duplicated codeCédric Ollivier1-12/+1
It also increases number of lines to warm about duplicated codes. Change-Id: I3914da8a90cbfe8fd024e9944c3adc8a617330d3 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-04-22Set max_microversion for placement and volumeCédric Ollivier1-0/+3
It sets max_microversion to latest because if both values are not specified, Tempest avoids tests which require a microversion [1]. It completes the previous change about nova [2]. [1] https://docs.openstack.org/tempest/latest/sampleconf.html [2] https://gerrit.opnfv.org/gerrit/#/c/67318/ Change-Id: Ieab56a239077476dd9a0e1ee97ac4293c4075f5c Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-04-15Stop forcing admin_scope_domain = TrueCédric Ollivier1-1/+0
tempest fails vs devstack stable/stein and it passes successfully vs devstack stable/rocky (Functest gates) without that option. It may be added to tempest_conf.yaml if SUT requires it. Change-Id: I4730c25b25403fd130d71166a8778e06fd479935 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-04-10Apply blacklist for OVN noha scenario as wellMichael Polenchuk1-0/+1
Change-Id: I1dd51a54f39967fcf6b3a522e2a548f81e5646a9 Signed-off-by: Michael Polenchuk <mpolenchuk@mirantis.com>
2019-03-21Update RefStack list to 2018.11Cédric Ollivier1-0/+2
It sets max_microversion to latest because if both values are not specified, Tempest avoids tests which require a microversion [1]. Then test_keypairsv22_create_list_show_with_type is skipped and refstack_defcore fails [2]. It also partially reverts "Allow building containers when refstack.openstack.org is down" [3]. [1] https://docs.openstack.org/tempest/latest/sampleconf.html [2] http://artifacts.opnfv.org/functest/functest-opnfv-functest-smoke-latest-refstack_defcore-run-147/results/refstack_defcore/tempest-report.html [3] https://gerrit.opnfv.org/gerrit/#/c/67232/ Change-Id: Ia87543bb476d3047684e71ddfd05c2aa3f10bbfe Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-03-19Do not allow testcase skipping in refstackJuha Kosonen1-0/+8
Set the result of refstack_defcore as failed if there were skipped testcases in a run. Change-Id: Id782c2dee511cefbafd5b4440f386fd66f934a47 Signed-off-by: Juha Kosonen <juha.kosonen@nokia.com>
2019-03-07Merge tempest.py and conf_utils.pyJuha Kosonen2-212/+185
Change-Id: I597abdbb5bd3de023aa7b66fa27764ca2ec27f0d Signed-off-by: Juha Kosonen <juha.kosonen@nokia.com>
2019-03-03Stop overriding ssh default timeoutCédric Ollivier1-3/+0
It should be modified by tempest custom yaml file instead. It's safe as the default timeout is bigger than the previous one. It also cleans the obsolete tempest/rally data in config_functest.yaml. [1] https://docs.openstack.org/tempest/latest/sampleconf.html Change-Id: Ibad02d8aaf90f154c82f3b60a6d3f04fc37bd7de Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-03-03Stop forcing non default operator roleCédric Ollivier1-2/+0
Member is the default value [1] and SwiftOperator is not created by default in OpenStack gates (devstack) This role should be modified by tempest custom yaml file instead. [1] https://docs.openstack.org/tempest/latest/sampleconf.html Change-Id: Ie77b91e36a7011e5730cbf5998e029c07ec7fe2f Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-02-27Gather all skipped tempest test casesPanagiotis Karalis1-1/+1
This patch applies a change to the regex responsible for identifying Tempest test cases. With the current regex some skipped test cases are erroneously ignored. The condition has been updated in order to match both format of skipped testcases ('skip:' and 'skip [xxx]:') Change-Id: Ie7144b01e07cae92918eede0324f3f1f2582c4aa Signed-off-by: Panagiotis Karalis <pkaralis@intracom-telecom.com>
2019-02-14Generate xunit reports (rally and tempest)Cédric Ollivier2-68/+8
It adds xunit reports for rally-based and tempest-based testcases. It completes the reports provided by snaps (thanks to Xtesting). All rally related operations are moved to rally. It allows removing the rally dependency to tempest which was false. Change-Id: Ia7d2476f58f4f68b7c88442e50cad844037a36e9 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-02-13Publish tempest details into databaseCédric Ollivier1-1/+1
It switches from tempest.log to rally.log to conform with rally/stestr. Change-Id: I00c2a268a2bc312a22a8d70278fad618c1c25235 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-01-28Allow running latest tempest_fullCédric Ollivier1-0/+5
It allows running test_user_update_own_password [1] [1] http://artifacts.opnfv.org/functest/functest-opnfv-functest-components-latest-tempest_full-run-8/results/tempest_full/tempest-report.html Change-Id: I9f273ba58871f1e16886af623dcbb91f5c20c2f6 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-01-20Fix one typo in docstring (tempest.py)Cédric Ollivier1-1/+1
Change-Id: I46053c92660139f4004d9cec8d1cfb6817cc0e07 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-01-19Take control over Rally logicsCédric Ollivier1-4/+25
Rally selects the first external network [1] which could raise side effects if: - another testcase is creating a dummy external network in // - all external subnets are not reachable from jumphost It's now overriden by EXTERNAL_NETWORK as defined in Functest Rally creates a new shared network if one already exists [2]. As juju now allows other shared networks, it can be safely created by Functest to allow only one logic and to handle provider networks, different name servers, etc... [1] https://github.com/openstack/rally-openstack/blob/master/rally_openstack/verification/tempest/config.py#L146 [2] https://github.com/openstack/rally-openstack/blob/master/rally_openstack/verification/tempest/context.py#L85 Change-Id: Icf8c08077d4b0a9eb1c2e1b7309c62957b0a3b63 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-01-19Move logics out of TempestCommon.__init__()Cédric Ollivier1-22/+33
It conforms with best programming practices. It allows muting console and increasing log level (debug). Change-Id: I0e0cfef6abe3b354a43817fcb242c71da701e237 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-11-10Check if at least 1 hypervisorsCédric Ollivier1-1/+1
Vmtp is skipped if only one hypervisor. As Vtmp returns 0 when all is skipped, it protects vs issues when generating charts. Change-Id: I5ea7c31c0688ae773425d2192a4f30a22360185f Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-11-06Disable agent testing in tempest-based testcasesCédric Ollivier3-3/+13
It blacklists agent testing and two tests asking for VXLAN as overlay if OVN (OVN supports GENEVE) [1] blacklist.txt is renamed blacklist.yaml and is verified by yamllint. [1] https://build.opnfv.org/ci/view/functest/job/functest-apex-baremetal-daily-gambia/33/console Change-Id: I73c1b8cbbd74973a7f15476feea6ee4532786660 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-10-26Move tempest logics in tempest_conf.yamlCédric Ollivier2-29/+14
It allows removing OPNFV logics out of Functest (releng) and then simplifies conf_utils. tempest.conf is filled by the current default values. functest-alpine.sh will be updated to allow modifying the default values without breaking the OPNFV installers. It deprecates the only tempest-related env var (STORAGE_PROTOCOL) Change-Id: I14bcb715826ef2d85af87e7af600261a3c08e72c Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-10-26Temporarily disable Cinder volumes backup testsCédric Ollivier1-0/+3
From the time being, all OPNFV installers [1][2][3] don't support Cinder volumes backup. We will find a better way to handle the lack of OPNFV requiremements (these tests pass succesfully in Orange Openlab). At least, it will allow verifying tempest_full in OPNFV gates. [1] Apex: https://artifacts.opnfv.org/logs/functest/lf-pod1/2018-10-25_02-17-25/tempest_full/tempest-report.html [2] Compass: https://artifacts.opnfv.org/logs/functest/huawei-virtual9/2018-10-24_04-54-25/tempest_full/tempest-report.html [3] Fuel: https://artifacts.opnfv.org/logs/functest/lf-pod2/gambia/2018-10-25_05-57-54/tempest_full/tempest-report.html Change-Id: I319536070aa77a5dbab6a692a1d76049f75e758b Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-10-26Temporarily disable VNC testsCédric Ollivier1-0/+3
At least Compass and Fuel configure Spice as console [1][2]. It disables vnc testing as it could fail for endusers as well. [1] https://artifacts.opnfv.org/logs/functest/huawei-virtual9/2018-10-24_04-54-25/tempest_full/tempest-report.html [2] https://artifacts.opnfv.org/logs/functest/lf-pod2/gambia/2018-10-25_05-57-54/tempest_full/tempest-report.html Change-Id: I2a1ecf345cb81170c1b1a3e49f8f3a58602fc3f0 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-10-23Disable Glance API v1Cédric Ollivier1-0/+4
It has been deprecated since Newton [1]. Even if the related tests are skipped in latest tempest [2], API v1 has to be disabled in Gambia's tempest.conf [3]. [1] https://docs.openstack.org/glance/latest/user/glanceapi.html [2] https://artifacts.opnfv.org/logs/functest/lf-pod1/2018-10-22_02-34-24/tempest_full/tempest-report.html [3] https://artifacts.opnfv.org/logs/functest/lf-pod1/gambia/2018-10-23_10-34-07/tempest_full/tempest-report.html Change-Id: I3b6bb632412c0fecaa659da262760af64d79fa71 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-10-21Harden Rally generate reportCédric Ollivier1-3/+2
It converts html_file to str and raises all possible exceptions. It sometimes fails when locally generating reports. Change-Id: Idf6a12aaac4561800cd5e364f8bcc495aabbee1d Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-10-14Fix image properties parsingCédric Ollivier2-20/+7
It removes useless spaces and now conforms with oslo.conf dict. New helpers and unit tests are available. Change-Id: I20335c5e5c3840cc872e158c6072d97a3f2cb98c Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-10-06Merge "Rename IMG_PROP to IMAGE_PROPERTIES"Cedric Ollivier1-4/+5
2018-10-06Rename IMG_PROP to IMAGE_PROPERTIESCédric Ollivier1-4/+5
Any other env vars are reduced. Change-Id: Ic2c37d6efd543603a2495f9d2621b66d61710a54 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-10-06Merge "Use environment variable for image properties"Cedric Ollivier1-0/+4
2018-10-05Use environment variable for image propertiesDelia Popescu1-0/+4
Use IMG_PROP environment variable to read extra image properties. Set image_extra_properties for scsi volume types without file patch, but using IMG_PROP env variable IMG_PROP is defined through releng. Image extra_properties will continue to get updates from functest config file With this change, remove unnecessary parameters from arm patch Related to:https://gerrit.opnfv.org/gerrit/#/c/63203/ JIRA:ARMBAND-402 Change-Id: I64139240751b1e109bbe24ba32f487ccd98b9a92 Signed-off-by: Delia Popescu <delia.popescu@enea.com>
2018-09-23Protect vs former OS env varsCédric Ollivier1-0/+5
rally_jobs is failing vs APEX which still sets OS_TENANT [1]. [1] https://build.opnfv.org/ci/view/functest/job/functest-apex-baremetal-daily-master/324/console Change-Id: I672bfa93b37b1b773ea7d20fdd91bea0ff14b79f Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-09-23Fix typos in tempest.confCédric Ollivier1-4/+5
It completes the previous change [1]. [1] https://gerrit.opnfv.org/gerrit/#/c/62793/2 Change-Id: Ie4c4c5cb374f2bfac6397841f8ddc2cd3b2f524b Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-09-23Set default_domain_id in tempest.confCédric Ollivier2-3/+6
It allows verifying via non-default domain. Change-Id: I95ae26fd3bfd1c10421c02d2b4f6e8a9b6aa49e6 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-09-22Set admin_domain_scope to True (tempest)Cédric Ollivier2-15/+15
It completes the previous commit [1]. It also sets cidr and fully forces Keystone v3. [1] https://gerrit.opnfv.org/gerrit/#/c/62777/ Change-Id: Ida948a4ff051dc1d1325d29a815c201366a1e33e Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-09-22List all sheduler filters instead of allCédric Ollivier1-1/+5
It doesn't work as expected as highlighted in [1]. [1] https://review.openstack.org/#/c/570207/ Change-Id: Iaf64345c40e8d62349cd50a6861c1191d25c40b9 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-09-22Grant new tempest user as admin in domain tooCédric Ollivier1-0/+3
It seems mandatory for test_list_projects_returns_only_authorized_projects Change-Id: Id994e66b088d751c4f6d7f260f8e482c2ee83307 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2018-09-22Stop creating network resources in tempestCédric Ollivier2-4/+5
fixed_network_name is not used in neutron or nova gates [1]. It may bypass juju bug (subnet discovery). [1] http://logs.openstack.org/46/603046/1/check/tempest-full/a8468a1/controller/logs/tempest_conf.txt Change-Id: I820e8c723918ed683ed1f8016363eb70bad0d14a Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>