aboutsummaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)AuthorFilesLines
2019-11-17Fix details in RallyCédric Ollivier1-5/+4
self.details is now a dict and summary is located under root [1]. [1] https://build.opnfv.org/ci/job/functest-opnfv-functest-smoke-hunter-rally_sanity-run/422/console Change-Id: Ic89e674b2f6660704b9dafe6070a224753e2af46 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-16Update to Xtesting 0.83Cédric Ollivier2-2/+2
It supports mimetypes and then eases reading the reports from browser. Change-Id: Ic38fca9fb05ca1ad76b589800b22bc78882c7806 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-16Merge "Correctly override dir_results and res_dir"Cedric Ollivier1-2/+2
2019-11-16Merge "Add tempest_keystone in CNTT API Compliance"Cedric Ollivier1-0/+18
2019-11-16Correctly override dir_results and res_dirCédric Ollivier1-2/+2
Both have to be modified else publishing artifacts fails [1]. [1] https://build.opnfv.org/ci/job/functest-opnfv-functest-healthcheck-latest-tempest_smoke-run/555/console Change-Id: Ibd110e4b57cb443b12adea73f73c8cdf7f828652 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-16Optimize lxml build if armCédric Ollivier1-0/+1
Else it raises timeouts on travis-ci.org: https://www.travis-ci.org/collivier/functest/builds/612068457 https://www.travis-ci.org/collivier/functest/builds/612116863 https://www.travis-ci.org/collivier/functest/jobs/612116916 Change-Id: Id34144ebcdf3f418d5a07068434eb7aa96b023d2 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-15Add rally_sanityCédric Ollivier1-1/+23
Gnocchi and Barbican are not currently selected by CNTT. Then they are removed from the optional list. Change-Id: I46d16b2cabd832b2655606b0708a6f43721cb3c5 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-15Add tempest_keystone in CNTT API ComplianceCédric Ollivier1-0/+18
It excludes test_federated_authentication as federation is currently optional [1]. Close cntt-n/CNTT#624 [1] https://github.com/collivier/CNTT/commit/2e4ae5d0f4ebf4ef4bbc2504ed6cd46388361c09 Change-Id: Ie322e7912c429543c5da92f8afb33de3e8e75c8a Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-15Test identity project_tags and application_credentialsCédric Ollivier1-0/+5
Both are supported by OpenStack and Functest gates. It also sets the default values to highlight the coverage. Change-Id: Iae080a8bde0c411acc7811e7a0e70e3471db8c98 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-10Add tempest_cinder in complianceCédric Ollivier1-219/+7
It conforms with Cinder Microversions and Feature [1]. It cleans the testcases which require CNTT updates (Keystone, Glance, Nova, etc.) [1] https://github.com/cntt-n/CNTT/pull/577 Change-Id: I8e719e9f48d70a860c0f879117144bb3ce6f4875 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-10Add features volume testingCédric Ollivier1-0/+8
The next features are supported by Functest SUT: - backup - snapshot - clone - manage_snapshot - manage_volume - volume_revert Change-Id: I0e83bb616a56b9e8651115bebf2373f85c642260 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-10Update neutron-tempest-plugin-api regex (CNTT)Cédric Ollivier1-9/+2
It conforms with the last updates in API [1] adding Neutron QoS and Trunk as mandatory. QosMinimumBandwidthRuleTestJSON is skipped as it's not covered by OpenStack (and Functest) gates. [1] https://github.com/cntt-n/CNTT/pull/463 Change-Id: I54ff5e1fdb9b1e7265c63e3f66dfcf2f10db07d5 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-10Set provider_vlans in tempest Neutron_plugin_optionsCédric Ollivier1-0/+1
Few tests are skipped if this value is unset. Then it's needed to fully cover Neutron Trunk (Compliance). Change-Id: Ife4becb7ff8b47c2d90fc6c5eb4f585685205ed6 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-10Buld cntt containers in a specific stageCédric Ollivier1-19/+21
They must be built after smoke. Change-Id: I939ef6c4663ad940c9f1b95440980628c4880385 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-09Update to Xtesting 0.82.0Cédric Ollivier3-1/+5
Change-Id: Ie94b35c9107c851bf0a412c1361629e0cf4154a2 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-07Update Xtesting to 0.81Cédric Ollivier2-2/+2
It contains a key fix about bucket creation. It also changes the Rebaca link which sometimes breaks sphinx spellcheck. Change-Id: Iaa7c791ad9a70c9fb65288557c5c8e8cddb5f2d0 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-02Update xtesting to 0.80.0Cédric Ollivier1-1/+1
It allows publishing the artifacts via Xtesting. It's required by the new Xtesting Ansible role [1]. [1] https://github.com/collivier/ansible-role-xtesting/commit/8c62525bb0d0b5f8dcc2c4436711d5d075de8811 Change-Id: I832e5e19467c7fd86a46574448faad4d6f2d4c0d Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-02Pin sphinxcontrib-spelling (4.3.0)Cédric Ollivier1-0/+1
Change-Id: I926d8e4cfd344f7177f2d2ad130b226a6726f214 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-11-01Temporarily switch to xrally_kubernetes pull requestCédric Ollivier1-1/+1
The pull request "Remove deprecated extensions/v1beta1 API" [1] is needed by Functest master and Jerma. It will be safely reverted once the pull request is merged. [1] https://github.com/xrally/xrally-kubernetes/pull/46 Change-Id: Ifaab89654a5b75a5948a0a5ccdb6e39349ce5cb4 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-29Merge "Publish functest-cntt-smoke"Juha Kosonen4-1/+310
2019-10-29Merge "Add keystone-tempest-plugin"Cedric Ollivier5-2/+40
2019-10-29Merge "Add cinder-tempest-plugin"Cedric Ollivier5-1/+44
2019-10-29Merge "Add healthcheck Horizon tests"Cedric Ollivier8-4/+59
2019-10-26Add keystone-tempest-pluginCédric Ollivier5-2/+40
It adds tempest_keystone which runs the tests proposed by keystone-tempest-plugin [1]. [1] https://opendev.org/openstack/keystone-tempest-plugin Change-Id: I83db584c37fe63a549ccf396b03e034778254b3a Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-26Add cinder-tempest-pluginCédric Ollivier5-1/+44
It adds tempest_cinder which runs the tests proposed by cinder-tempest-plugin [1]. test_incremental_backup is disabled because it currently fails vs Functest SUT (under analysis). [1] https://opendev.org/openstack/cinder-tempest-plugin Change-Id: I85b255eb05b9ab1c74c3829148da5230d6c7647e Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-26Add healthcheck Horizon testsCédric Ollivier8-4/+59
It runs TestDashboardBasicOps as proposed by tempest-horizon [1]. It's skipped if DASHBOARD_URL is unset (default). [1] https://opendev.org/openstack/tempest-horizon Change-Id: I1b6b5609412770609f412da4927641588b173ed6 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-26Publish functest-cntt-smokeCédric Ollivier4-1/+310
It tunes neutron-tempest-plugin-api to avoid skipping any test (conformance as expected by CNTT). Until CNTT clearly defines the optional and mandatory neutron extensions, it selects the OpenStack and Functest gate values as default. The next Neutron extensions are considered as optional (still verified by functest-smoke): - logging - network-segment-range - port-resource-request - qos - trunk - l3-conntrack-helper - metering - floating-ip-port-forwarding - uplink-status-propagation - dns-domain-ports - dns-integration - router-interface-fip The next tests (tempest_full, tempest_slow, tempest_scenario, etc.) will be updated in next changes. Change-Id: I003d1e509cd6dc7bd58261f27676fb53cb79e68c Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-26Add spelling exceptions (travis-ci)Cédric Ollivier1-0/+3
This words seem fine but it enforces the verification in travis-ci [1]. [1] https://travis-ci.org/collivier/functest/jobs/603020830 Change-Id: I44b8db5e63668899a27480a7f7da5d48f9f41f21 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-26Update RefStack lists to 2019.06Cédric Ollivier5-6/+15
Change-Id: I9d3abc3996857d8905007640250b94025735809c Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-25Merge "Add neutron_plugin_options section in tempest.conf"Cedric Ollivier1-0/+3
2019-10-24Merge "Allow checking Tests count"Cedric Ollivier3-0/+10
2019-10-24Add neutron_plugin_options section in tempest.confCédric Ollivier1-0/+3
It avoids skipping the next 2 tests [1]: neutron_tempest_plugin.api.admin.test_agent_availability_zone.AgentAvailabilityZoneTestCase.test_agents_availability_zone neutron_tempest_plugin.api.admin.test_networks.NetworksTestAdmin.test_create_tenant_network_vxlan [1] http://artifacts.opnfv.org/functest/functest-opnfv-functest-smoke-latest-neutron-tempest-plugin-api-run-498/results/neutron-tempest-plugin-api/tempest-report.html Change-Id: I83f3f44302336045e47514e0aa0b67737161f0e1 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-23Allow checking Tests countCédric Ollivier3-0/+10
It meets CNTT requirements (verifying conformance with a reference implementation) and may protect vs synchronization issues (see gambia refstack_defcore). Change-Id: I842bc4ed91eec6bc545c489e3686d0d41dae7b44 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-23Allow blacklisting tests if public endpoint onlyCédric Ollivier4-6/+30
It allows reusing the blacklisting system if only public endpoints are reachable (ONAP Openlab testing). It contains a first list of tempest-based tests which force the use of admin endpoints. Change-Id: Ic1ab07dec3ae1f2a0ca50b80e08cd0ac396f2e56 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-19Update Flake8Cédric Ollivier2-1/+2
It avoids the next incompatibility: ERROR: flake8 2.5.5 has requirement mccabe<0.5,>=0.2.1, but you'll have mccabe 0.6.1 which is incompatible. Change-Id: Ic3bb14b032adfee64109549abc0a09d8cc86968f Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-19Remove lfreleng docsCédric Ollivier6-14/+2
Its output may has never been used and it's obsolete: ERROR: lfdocs-conf 0.3.1 has requirement more-itertools~=5.0.0, but you'll have more-itertools 7.2.0 which is incompatible. ERROR: lfdocs-conf 0.3.1 has requirement Sphinx~=1.8.5, but you'll have sphinx 2.2.0 which is incompatible. Change-Id: Id5d24b94f6cdbbb93be8dbb18cde1a7fc9db6a42 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-19Check spelling in rst filesCédric Ollivier12-15/+164
It fixes a few typos too and installs enchant in TravisCI and CircleCI jobs. Change-Id: Id5dbf5f926edc8b0b4e4337d11bf63e21ea7e173 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-13Update Functest release notesCédric Ollivier1-3/+9
Change-Id: I7d3198ea92105870425164ce5ade6aca824f8dfd Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-13Update python dirs in docsCédric Ollivier3-11/+11
master is now leveraging on python3.7. Change-Id: I7076e143f162a6aacd2e8d8d8f38524aea71e365 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-13Updates all testcases and their resultsCédric Ollivier3-97/+117
Change-Id: I127146b4d6face1b6221e049636a29e601415fd8 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-11Leverage on Rally task tagsCédric Ollivier2-32/+33
It avoids parsing rally task outputs which may hang (subprocess communicate) as seen in rally_full and rally_jobs. It simply selects test names as tags. Change-Id: I88b54a8f155e557f8a606fdbd7d86c1f4d5dae3b Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-08Merge "Switch to Tempest password generator"Cedric Ollivier1-5/+2
2019-10-08Switch to Tempest password generatorCédric Ollivier1-5/+2
Functest depends on Tempest (master) then we can safely leverage on its password generator. '%' is forbidden in our logic then it's replaced by '!' (extra random logic seems overkilled). Change-Id: Ie1d1b78401999efbbb5aec4500c14595bde87987 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-08Count all hypervisors by defaultCédric Ollivier7-6/+16
If one hypervisor is down, few tempest test fails. User is free to set SKIP_DOWN_HYPERVISORS if it's down on purpose. Change-Id: I7b6a4d0d8f67755d8c1550fd1bc6fd707634f68b Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-04Update Shaker image (1.3.0 + Debian Stretch)Cédric Ollivier4-7/+5
It udates Shaker to 1.3.0 and switches to Debian Stretch as base image. It also eases debugging as ssh basic auth is enabled (shaker/shaker). The arm image has to be built before enabling the testcase for arm. Change-Id: I894bae86c7e8f7cafbba2b4e00e3334afed0b6e2 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-02Ease debugging ShakerCédric Ollivier1-1/+1
It turns Shaker debug on to ease troubleshooting it. Change-Id: I7f24a8d3776742d569d2d47a60d6985584514014 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-10-01Merge "Count all active hypervisors"Cedric Ollivier6-7/+14
2019-09-30Count all active hypervisorsCédric Ollivier6-7/+14
Else tempest_slow fails if one hypervisor is down (detected in ONAP Openlab). It also improves Shaker, Rally and Vmtp which count them. Change-Id: Iee7c20e0357d9237501c5c451580d8f6409ac86b Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-09-29List services via OS_INTERFACE in RallyCédric Ollivier1-1/+2
Rally now leverages on list_services as proposed in utils [1] [1] https://git.opnfv.org/functest/commit/?id=0837a745c81ed2a6e7df097f60d227d5997c805e Change-Id: Ib17f9b530c2933d4a49eb5b744c2ae80b24ad560 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
2019-09-29Detect OpenStack TrainCédric Ollivier2-1/+12
It takes into account latest max nova microversions [1] [1] https://docs.openstack.org/nova/latest/reference/api-microversion-history.html Change-Id: Icb9838b09ad22f6e87e9d8fb3c47525bdc1e3307 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>