diff options
Diffstat (limited to 'docs/testing/user/testspecification/tempest_osinterop')
6 files changed, 385 insertions, 211 deletions
diff --git a/docs/testing/user/testspecification/tempest_osinterop/index.rst b/docs/testing/user/testspecification/tempest_osinterop/index.rst index 6773275e..d2a54e86 100644 --- a/docs/testing/user/testspecification/tempest_osinterop/index.rst +++ b/docs/testing/user/testspecification/tempest_osinterop/index.rst @@ -3,11 +3,11 @@ .. (c) Huawei Technologies Co.,Ltd and others ============================================= -OpenStack Interoperability test specification +OpenStack Interoperability Test Specification ============================================= The test cases documented here are the API test cases in the OpenStack -Interop guideline 2017.09 as implemented by the RefStack client. +Interop guideline 2018.11 as implemented by the RefStack client. References ================ @@ -16,9 +16,9 @@ References - https://wiki.openstack.org/wiki/Governance/InteropWG -- OpenStack Interoperability guidelines (version 2017.09) +- OpenStack Interoperability guidelines (version 2018.11) - - https://github.com/openstack/interop/blob/master/2017.09.json + - https://github.com/openstack/interop/blob/master/2018.11.json - Refstack client diff --git a/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_compute.rst b/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_compute.rst index 63175306..3e663d98 100644 --- a/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_compute.rst +++ b/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_compute.rst @@ -11,7 +11,7 @@ Scope The VIM compute operations test area evaluates the ability of the system under test to support VIM compute operations. The test cases documented here are the -compute API test cases in the OpenStack Interop guideline 2017.09 as implemented +compute API test cases in the OpenStack Interop guideline 2018.11 as implemented by the RefStack client. These test cases will evaluate basic OpenStack (as a VIM) compute operations, including: @@ -22,7 +22,7 @@ compute operations, including: - Basic server operations - Volume management operations -Definitions and abbreviations +Definitions and Abbreviations ============================= The following terms and abbreviations are used in conjunction with this test area @@ -30,14 +30,15 @@ The following terms and abbreviations are used in conjunction with this test are - API - Application Programming Interface - NFVi - Network Functions Virtualization infrastructure - SUT - System Under Test -- UUID - Universally Unique Identifier +- UUID - Universally Unique IDentifier - VIM - Virtual Infrastructure Manager - VM - Virtual Machine System Under Test (SUT) ======================= -The system under test is assumed to be the NFVi and VIM deployed with a Pharos compliant infrastructure. +The system under test is assumed to be the NFVi and VIM deployed with a Pharos +compliant infrastructure. Test Area Structure ==================== @@ -50,7 +51,7 @@ the same state as before the test. For brevity, the test cases in this test area are summarized together based on the operations they are testing. -All these test cases are included in the test case dovetail.tempest.osinterop of +All these test cases are included in the test case functest.tempest.osinterop of OVP test suite. Test Descriptions @@ -759,3 +760,48 @@ Post conditions --------------- N/A + +-------------------------------------------------------- +Test Case 10 - Keypair operations within the Compute API +-------------------------------------------------------- + +Test case specification +----------------------- + +This test case evaluates the Compute API ability of creating keypair with type, +the reference is, + +tempest.api.compute.keypairs.test_keypairs_v22.KeyPairsV22TestJSON.test_keypairsv22_create_list_show_with_type + +Test preconditions +------------------ + +* Compute server extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair with type 'x509' and a random name +* **Test assertion 1:** The keypair type received in the response body is equal to 'x509' +* Test action 2: Show the details of this created keypair +* **Test assertion 2:** The keypair type received in the response body is equal to 'x509' +* Test action 3: List all keypairs and find the one with the same name as given in test action 1 +* **Test assertion 3:** The keypair type of this keypair is equal to 'x509' + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of keypair operations within the Compute API. +Specifically, the test verifies that: + +* Can create keypair by specifying keypair type. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A diff --git a/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_identity.rst b/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_identity.rst index 6c0d23b7..34a71168 100644 --- a/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_identity.rst +++ b/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_identity.rst @@ -14,7 +14,7 @@ support VIM identity operations. The tests in this area will evaluate API discovery operations within the Identity v3 API, auth operations within the Identity API. -Definitions and abbreviations +Definitions and Abbreviations ============================= The following terms and abbreviations are used in conjunction with this test area @@ -34,7 +34,7 @@ Test Area Structure The test area is structured based on VIM identity operations. Each test case is able to run independently, i.e. irrelevant of the state created by a previous test. -All these test cases are included in the test case dovetail.tempest.osinterop of +All these test cases are included in the test case functest.tempest.osinterop of OVP test suite. Dependency Description @@ -42,15 +42,15 @@ Dependency Description The VIM identity operations test cases are a part of the OpenStack interoperability tempest test cases. For Fraser based dovetail release, the -OpenStack interoperability guidelines (version 2017.09) is adopted, which is +OpenStack interoperability guidelines (version 2018.11) is adopted, which is valid for Mitaka, Newton, Ocata and Pike releases of Openstack. Test Descriptions ================= ----------------------------------------------------- -API discovery operations within the Identity v3 API ----------------------------------------------------- +----------------------------------------------------------------- +Test Case 1 - API discovery operations within the Identity v3 API +----------------------------------------------------------------- Use case specification ----------------------- @@ -79,7 +79,7 @@ Test execution 'stable', 'experimental', 'supported', 'deprecated' are all of the identity api 'status' values. -Pass / fail criteria +Pass / Fail criteria ''''''''''''''''''''' This test case passes if all test action steps execute successfully and all assertions @@ -91,14 +91,15 @@ Post conditions None ------------------------------------------- -Auth operations within the Identity API ------------------------------------------- +----------------------------------------------------- +Test Case 2 - Auth operations within the Identity API +----------------------------------------------------- Use case specification ----------------------- tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token +tempest.api.identity.v3.test_tokens.TokensV3Test.test_validate_token Test preconditions ------------------- @@ -119,8 +120,16 @@ Test execution passes if it is equal to the user_name which is used to get token. * Test action 4: Get the method in getting token response message, the test passes if it is equal to the password which is used to get token. +* Test action 5: Get the token by system credentials and show the token, + the test passes if the response bodies of the get and show operations are the same. +* Test action 6: Get the user_id in showing token response message, the test + passes if it is equal to the user_id which is used to get token. +* Test action 7: Get the username in showing token response message, the test + passes if it is equal to the username which is used to get token. +* Test action 8: Delete this token by non-admin compute client, the test passes + if it raises a NotFound exception. -Pass / fail criteria +Pass / Fail criteria ''''''''''''''''''''' This test case passes if all test action steps execute successfully and all assertions @@ -132,3 +141,39 @@ Post conditions None +-------------------------------------------------------- +Test Case 3 - Catalog operations within the Identity API +-------------------------------------------------------- + +Use case specification +----------------------- + +tempest.api.identity.v3.test_catalog.IdentityCatalogTest.test_catalog_standardization + +Test preconditions +------------------- + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Show all catalogs by non-admin catalog client, the test passes + if the catalog types getting in the show response message equal to the + standard service values. Standard catalog types of 'keystone', 'nova', 'glance' and + 'swift' should be 'identity', 'compute', 'image' and 'object-store' respectively. + +Pass / Fail criteria +''''''''''''''''''''' + +This test case passes if all test action steps execute successfully and all assertions +are affirmed. If any test steps fails to execute successfully or any of the assertions +is not met, the test case fails. + +Post conditions +--------------- + +None diff --git a/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_image.rst b/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_image.rst index c5926b02..b6cdb77f 100644 --- a/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_image.rst +++ b/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_image.rst @@ -11,11 +11,11 @@ Scope The VIM image test area evaluates the ability of the system under test to support VIM image operations. The test cases documented here are the Image API test cases -in the Openstack Interop guideline 2017.09 as implemented by the Refstack client. +in the Openstack Interop guideline 2018.11 as implemented by the Refstack client. These test cases will evaluate basic Openstack (as a VIM) image operations including image creation, image list, image update and image deletion capabilities using Glance v2 API. -Definitions and abbreviations +Definitions and Abbreviations ============================= The following terms and abbreviations are used in conjunction with this test area @@ -40,7 +40,7 @@ to run independently, i.e. irrelevant of the state created by a previous test. For brevity, the test cases in this test area are summarized together based on the operations they are testing. -All these test cases are included in the test case dovetail.tempest.osinterop of +All these test cases are included in the test case functest.tempest.osinterop of OVP test suite. Test Descriptions @@ -62,9 +62,9 @@ Images: https://docs.openstack.org/api-ref/image/v2/ - add image tag - delete image tag ---------------------------------------- -Image get tests using the Glance v2 API ---------------------------------------- +----------------------------------------------------- +Test Case 1 - Image get tests using the Glance v2 API +----------------------------------------------------- Test case specification ----------------------- @@ -104,7 +104,7 @@ Test execution whether the 6 images' ids are not in the show list. * **Test assertion 6:** The 6 images' ids are not found in the show list. -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' The first two test cases evaluate the ability to use Glance v2 API to show image @@ -124,9 +124,9 @@ Post conditions None --------------------------------------- -CRUD image operations in Images API v2 --------------------------------------- +---------------------------------------------------- +Test Case 2 - CRUD image operations in Images API v2 +---------------------------------------------------- Test case specification ----------------------- @@ -148,7 +148,7 @@ Test execution * Test action 2: List all images and check whether the ids listed are in the created images list. * **Test assertion 1:** The ids get from the list images API are in the created images list. -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' This test case evaluates the ability to use Glance v2 API to list images. @@ -163,9 +163,9 @@ Post conditions None ----------------------------------------- -Image list tests using the Glance v2 API ----------------------------------------- +------------------------------------------------------ +Test Case 3 - Image list tests using the Glance v2 API +------------------------------------------------------ Test case specification ----------------------- @@ -224,7 +224,7 @@ Test execution the 6 ids are not in the show list. * **Test assertion 8:** The stored 6 ids are not found in the show list. -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' This test case evaluates the ability to use Glance v2 API to list images with @@ -245,9 +245,9 @@ Post conditions None ------------------------------------------- -Image update tests using the Glance v2 API ------------------------------------------- +-------------------------------------------------------- +Test Case 4 - Image update tests using the Glance v2 API +-------------------------------------------------------- Test case specification ----------------------- @@ -291,7 +291,7 @@ Test execution and check whether the ids are not in the show list. * **Test assertion 6:** The two ids are not found in the show list. -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' This test case evaluates the ability to use Glance v2 API to update images with @@ -308,9 +308,9 @@ Post conditions None --------------------------------------------- -Image deletion tests using the Glance v2 API --------------------------------------------- +---------------------------------------------------------- +Test Case 5 - Image deletion tests using the Glance v2 API +---------------------------------------------------------- Test case specification ----------------------- @@ -349,7 +349,7 @@ Test execution and check whether the ids are in the list. * **Test assertion 5:** The two ids are not found in the list. -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' The first three test cases evaluate the ability to use Glance v2 API to delete images diff --git a/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_network.rst b/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_network.rst index 4e3e5a95..d8e7413b 100644 --- a/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_network.rst +++ b/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_network.rst @@ -11,11 +11,11 @@ Scope The VIM network test area evaluates the ability of the system under test to support VIM network operations. The test cases documented here are the network API test cases -in the Openstack Interop guideline 2017.09 as implemented by the Refstack client. +in the Openstack Interop guideline 2018.11 as implemented by the Refstack client. These test cases will evaluate basic Openstack (as a VIM) network operations including basic CRUD operations on L2 networks, L2 network ports and security groups. -Definitions and abbreviations +Definitions and Abbreviations ============================= The following terms and abbreviations are used in conjunction with this test area @@ -42,7 +42,7 @@ the same state as before the test. For brevity, the test cases in this test area are summarized together based on the operations they are testing. -All these test cases are included in the test case dovetail.tempest.osinterop of +All these test cases are included in the test case functest.tempest.osinterop of OVP test suite. Test Descriptions @@ -84,9 +84,9 @@ Network: https://docs.openstack.org/api-ref/network/v2/index.html - show security group rule - delete security group rule ---------------------------------------------------------- -Basic CRUD operations on L2 networks and L2 network ports ---------------------------------------------------------- +----------------------------------------------------------------------- +Test Case 1 - Basic CRUD operations on L2 networks and L2 network ports +----------------------------------------------------------------------- Test case specification ----------------------- @@ -100,6 +100,7 @@ tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_without tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_all_attributes tempest.api.network.test_networks.NetworksTest.test_create_update_delete_network_subnet tempest.api.network.test_networks.NetworksTest.test_delete_network_with_subnet +tempest.api.network.test_networks.NetworksTest.test_external_network_visibility tempest.api.network.test_networks.NetworksTest.test_list_networks tempest.api.network.test_networks.NetworksTest.test_list_networks_fields tempest.api.network.test_networks.NetworksTest.test_list_subnets @@ -114,6 +115,7 @@ tempest.api.network.test_ports.PortsTestJSON.test_create_port_in_allowed_allocat tempest.api.network.test_ports.PortsTestJSON.test_create_update_delete_port tempest.api.network.test_ports.PortsTestJSON.test_list_ports tempest.api.network.test_ports.PortsTestJSON.test_list_ports_fields +tempest.api.network.test_ports.PortsTestJSON.test_port_list_filter_by_router_id tempest.api.network.test_ports.PortsTestJSON.test_show_port tempest.api.network.test_ports.PortsTestJSON.test_show_port_fields @@ -163,55 +165,64 @@ Test execution found after deletion * Test action 9: Create a network and create a subnet of this network, then delete this network * **Test assertion 9:** The subnet has also been deleted after deleting the network -* Test action 10: Create a network and list all networks -* **Test assertion 10:** The network created is found in the list -* Test action 11: Create a network and list networks with the id and name of the created network -* **Test assertion 11:** The id and name of the list network equal to the created network's id and name -* Test action 12: Create a network and create a subnet of this network, then list all subnets -* **Test assertion 12:** The subnet created is found in the list -* Test action 13: Create a network and create a subnet of this network, then list subnets with +* Test action 10: List all external networks, find the one with the same public_network_id + as defined in tempest.conf and list its subnets +* **Test assertion 10:** The external network can be found, no internal network got by the + list operation, if this external network is shared, the subnets list is not empty, + otherwise, it should be empty +* Test action 11: Create a network and list all networks +* **Test assertion 11:** The network created is found in the list +* Test action 12: Create a network and list networks with the id and name of the created network +* **Test assertion 12:** The id and name of the list network equal to the created network's id and name +* Test action 13: Create a network and create a subnet of this network, then list all subnets +* **Test assertion 13:** The subnet created is found in the list +* Test action 14: Create a network and create a subnet of this network, then list subnets with the id and network_id of the created subnet -* **Test assertion 13:** The id and network_id of the list subnet equal to the created subnet -* Test action 14: Create a network and show network's details with the id of the created network -* **Test assertion 14:** The id and name returned in the response equal to the created network's id and name -* Test action 15: Create a network and just show network's id and name info with the id of the created network -* **Test assertion 15:** The keys returned in the response are only id and name, and the values +* **Test assertion 14:** The id and network_id of the list subnet equal to the created subnet +* Test action 15: Create a network and show network's details with the id of the created network +* **Test assertion 15:** The id and name returned in the response equal to the created network's id and name +* Test action 16: Create a network and just show network's id and name info with the id of the created network +* **Test assertion 16:** The keys returned in the response are only id and name, and the values of all the keys equal to network's id and name -* Test action 16: Create a network and create a subnet of this network, then show subnet's details +* Test action 17: Create a network and create a subnet of this network, then show subnet's details with the id of the created subnet -* **Test assertion 16:** The id and cidr info returned in the response equal to the created +* **Test assertion 17:** The id and cidr info returned in the response equal to the created subnet's id and cidr -* Test action 17: Create a network and create a subnet of this network, then show subnet's id and +* Test action 18: Create a network and create a subnet of this network, then show subnet's id and network_id info with the id of the created subnet -* **Test assertion 17:** The keys returned in the response are just id and network_id, and the values +* **Test assertion 18:** The keys returned in the response are just id and network_id, and the values of all the keys equal to subnet's id and network_id -* Test action 18: Create a network and create a subnet of this network, then update subnet's +* Test action 19: Create a network and create a subnet of this network, then update subnet's name, host_routes, dns_nameservers and gateway_ip -* **Test assertion 18:** The name, host_routes, dns_nameservers and gateway_ip returned in the +* **Test assertion 19:** The name, host_routes, dns_nameservers and gateway_ip returned in the response equal to the values used to update the subnet -* Test action 19: Create 2 networks and bulk create 2 ports with the ids of the created networks -* **Test assertion 19:** The network_id of each port equals to the one used to create the port and +* Test action 20: Create 2 networks and bulk create 2 ports with the ids of the created networks +* **Test assertion 20:** The network_id of each port equals to the one used to create the port and the admin_state_up of each port is True -* Test action 20: Create a network and create a subnet of this network by setting allocation_pools, +* Test action 21: Create a network and create a subnet of this network by setting allocation_pools, then create a port with the created network's id -* **Test assertion 20:** The ip_address of the created port is in the range of the allocation_pools -* Test action 21: Create a network and create a port with its id, then update the port's name and +* **Test assertion 21:** The ip_address of the created port is in the range of the allocation_pools +* Test action 22: Create a network and create a port with its id, then update the port's name and set its admin_state_up to be False -* **Test assertion 21:** The name returned in the response equals to the name used to update +* **Test assertion 22:** The name returned in the response equals to the name used to update the port and the port's admin_state_up is False -* Test action 22: Create a network and create a port with its id, then list all ports -* **Test assertion 22:** The created port is found in the list -* Test action 23: Create a network and create a port with its id, then list ports with the id - and mac_address of the created port +* Test action 23: Create a network and create a port with its id, then list all ports * **Test assertion 23:** The created port is found in the list -* Test action 24: Create a network and create a port with its id, then show the port's details -* **Test assertion 24:** The key 'id' is in the details -* Test action 25: Create a network and create a port with its id, then show the port's id +* Test action 24: Create a network and create a port with its id, then list ports with the id + and mac_address of the created port +* **Test assertion 24:** The created port is found in the list +* Test action 25: Create a network and create a subnet, port with its id, create a router + and add this port as this router's interface, then list ports with this router id +* **Test assertion 25:** The number of the ports list is 1, the port id and device_id + getting with the list operation are the same as the ones got when creating them +* Test action 26: Create a network and create a port with its id, then show the port's details +* **Test assertion 26:** The key 'id' is in the details +* Test action 27: Create a network and create a port with its id, then show the port's id and mac_address info with the port's id -* **Test assertion 25:** The keys returned in the response are just id and mac_address, +* **Test assertion 27:** The keys returned in the response are just id and mac_address, and the values of all the keys equal to port's id and mac_address -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' These test cases evaluate the ability of basic CRUD operations on L2 networks and L2 network ports. @@ -232,9 +243,9 @@ Post conditions N/A ----------------------------------------- -Basic CRUD operations on security groups ----------------------------------------- +------------------------------------------------------ +Test Case 2 - Basic CRUD operations on security groups +------------------------------------------------------ Test case specification ----------------------- @@ -321,7 +332,7 @@ Test execution * Test action 19: Generate a random uuid and use this id to show security group rule * **Test assertion 19:** Failed to show security group rule because of nonexistent id of security group rule -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' These test cases evaluate the ability of Basic CRUD operations on security groups and security group rules. @@ -342,9 +353,9 @@ Post conditions N/A -------------------------------- -CRUD operations on subnet pools -------------------------------- +--------------------------------------------- +Test Case 3 - CRUD operations on subnet pools +--------------------------------------------- Test case specification ----------------------- @@ -371,7 +382,7 @@ Test execution * Test action 4: Delete SNP1 -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' These test cases evaluate the ability of Basic CRUD operations on subnetpools. @@ -385,3 +396,131 @@ Post conditions --------------- N/A + +---------------------------------------- +Test Case 4 - CRUD operations on routers +---------------------------------------- + +Test case specification +----------------------- + +tempest.api.network.test_routers.RoutersTest.test_add_multiple_router_interfaces +tempest.api.network.test_routers.RoutersTest.test_add_remove_router_interface_with_port_id +tempest.api.network.test_routers.RoutersTest.test_add_remove_router_interface_with_subnet_id +tempest.api.network.test_routers.RoutersTest.test_create_show_list_update_delete_router +tempest.api.network.test_routers.RoutersTest.test_update_delete_extra_route +tempest.api.network.test_routers.RoutersTest.test_update_router_admin_state +tempest.api.network.test_routers_negative.RoutersNegativeTest.test_add_router_interfaces_on_overlapping_subnets_returns_400 +tempest.api.network.test_routers_negative.RoutersNegativeTest.test_delete_non_existent_router_returns_404 +tempest.api.network.test_routers_negative.RoutersNegativeTest.test_router_add_gateway_invalid_network_returns_404 +tempest.api.network.test_routers_negative.RoutersNegativeTest.test_router_add_gateway_net_not_external_returns_400 +tempest.api.network.test_routers_negative.RoutersNegativeTest.test_router_remove_interface_in_use_returns_409 +tempest.api.network.test_routers_negative.RoutersNegativeTest.test_show_non_existent_router_returns_404 +tempest.api.network.test_routers_negative.RoutersNegativeTest.test_update_non_existent_router_returns_404 + +Test preconditions +------------------ + +Neutron is available. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create 2 networks NET1 and NET2, create SubNet1 of NET1 and SubNet2 with cidr of NET2, + create a router R1, add SubNet1 and SubNet2 to be R1's interfaces and get port details +* **Test assertion 1:** The device_id and subnet_id of the port equals to the router id + and subnet id getting from the creating response body +* Test action 2: Create a network NET1, create SubNet1 of NET1, create P1 of NET1, create a router R1, + add P1 to be its interface, store the response body B1, show P1 and store the response body B2 and + remove the interface with port id +* **Test assertion 2:** B1 includes keys 'subnet_id' and 'port_id', 'device_id' of B2 equals to router id +* Test action 3: Create a network NET1, create SubNet1 of NET1, create a router R1, add SubNet1 to be its + interface, store the response body B1, show port details with 'port_id' in B1 +* **Test assertion 3:** B1 includes keys 'subnet_id' and 'port_id', 'device_id' equals to router id +* Test action 4: Create a router R1 with name, admin_state_up False and external_network_id Ext-Net, + store the request body B1, show R1 with 'id' in B1, list all routers, update R1's name +* **Test assertion 4:** 'name', 'admin_state_up' and 'network_id' in B1 equal to the name, False and Ext-Net, + 'name' in show details equals to 'name' in B1, 'network_id' in show details equals to Ext-Net, + R1 is in the router list, the 'name' has been updated +* Test action 5: Create a router R1 with admin_state_up=True, create 4 networks and 4 subnets with different + cidr, add these 4 subnets to R1 as its interfaces, update R1 by given routes and show R1, delete extra + route of R1 and then show R1 +* **Test assertion 5:** The number of routes queals to 4 and routes is empty after deletion +* Test action 6: Create a router R1, update 'admin_state_up' of R1 to be True +* **Test assertion 6:** 'admin_state_up' of R1 is False, 'admin_state_up' of R1 is True after updating +* Test action 7: Create 2 networks NET1 and NET2, create SubNet1 of NET1 and SubNet2 of NET2, + create a router R1, add SubNet1 to be R1's interface and then trying to add SubNet2 to be R1's interface +* **Test assertion 7:** It raises a BadRequest exception when trying to add SubNet2 +* Test action 8: Try to delete router with a random name +* **Test assertion 8:** It raises a NotFound exception when trying delete operation +* Test action 9: Create a router R1 and try to update it with wrong external_gateway_info +* **Test assertion 9:** It raises a NotFound exception when trying to update R1 +* Test action 10: Create an internal network NET1, create SubNet1 with cidr of NET1, + create a router R1 and try to update it with NET1 as external_gateway_info +* **Test assertion 10:** It raises a BadRequest exception when trying to update R1 +* Test action 11: Create a network NET1, create SubNet1 of NET1, create a router R1, add SubNet1 to + the interface of R1 and try to delete R1 +* **Test assertion 11:** It raises a Conflict exception when trying to delete R1 +* Test action 12: Try to show router with a random name +* **Test assertion 12:** It raises a NotFound exception when trying to show router +* Test action 13: Try to update router with a random name +* **Test assertion 13:** It raises a NotFound exception when trying to update router + + +Pass / Fail criteria +'''''''''''''''''''' + +These test cases evaluate the ability of Basic CRUD operations on routers. +Specifically it verifies that: + +* Routers can be created, updated, shown and deleted. +* Can not show, update and delete non existent router. +* Can not remove interface in use. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------------- +Test Case 5 - List versions within Network API +---------------------------------------------- + +Test case specification +----------------------- + +tempest.api.network.test_versions.NetworksApiDiscovery.test_api_version_resources + +Test preconditions +------------------ + +Neutron is available. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: List network API versions +* **Test assertion 1:** The network API version is 'v2.0' + +Pass / Fail criteria +'''''''''''''''''''' + +This test case evaluates the ability of listing network API versions. +Specifically it verifies that: + +* The network API version is 'v2.0'. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A diff --git a/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_volume.rst b/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_volume.rst index 191427de..263e87cc 100644 --- a/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_volume.rst +++ b/docs/testing/user/testspecification/tempest_osinterop/tempest_osinterop_volume.rst @@ -11,7 +11,7 @@ Scope The VIM volume operations test area evaluates the ability of the system under test to support VIM volume operations. The test cases documented here are the -volume API test cases in the OpenStack Interop guideline 2017.09 as implemented +volume API test cases in the OpenStack Interop guideline 2018.11 as implemented by the RefStack client. These test cases will evaluate basic OpenStack (as a VIM) volume operations, including: @@ -24,7 +24,7 @@ volume operations, including: - Volume metadata operations - Volume snapshot operations -Definitions and abbreviations +Definitions and Abbreviations ============================= The following terms and abbreviations are used in conjunction with this test area @@ -51,7 +51,7 @@ the same state as before the test. For brevity, the test cases in this test area are summarized together based on the operations they are testing. -All these test cases are included in the test case dovetail.tempest.osinterop of +All these test cases are included in the test case functest.tempest.osinterop of OVP test suite. Test Descriptions @@ -77,49 +77,8 @@ Block storage: https://docs.openstack.org/api-ref/block-storage/ - update snapshot - delete snapshot ------------------------------------------------------ -Test Case 1 - Upload volumes with Cinder v2 or v3 API ------------------------------------------------------ - -Test case specification ------------------------ - -tempest.api.volume.test_volumes_actions.VolumesActionsTest.test_volume_upload - -Test preconditions ------------------- - -* Volume extension API - -Basic test flow execution description and pass/fail criteria ------------------------------------------------------------- - -Test execution -'''''''''''''' -* Test action 1: Create a volume VOL1 -* Test action 2: Convert VOL1 and upload image IMG1 to the Glance -* Test action 3: Wait until the status of IMG1 is 'ACTIVE' and VOL1 is 'available' -* Test action 4: Show the details of IMG1 -* **Test assertion 1:** The name of IMG1 shown is the same as the name used to upload it -* **Test assertion 2:** The disk_format of IMG1 is the same as the disk_format of VOL1 - -Pass / fail criteria -'''''''''''''''''''' - -This test case evaluates the volume API ability of uploading images. -Specifically, the test verifies that: - -* The Volume can convert volumes and upload images. - -In order to pass this test, all test assertions listed in the test execution above need to pass. - -Post conditions ---------------- - -N/A - -------------------------------------------------------------------------------------- -Test Case 2 - Volume service availability zone operations with the Cinder v2 or v3 API +Test Case 1 - Volume service availability zone operations with the Cinder v2 or v3 API -------------------------------------------------------------------------------------- Test case specification @@ -140,7 +99,7 @@ Test execution * Test action 1: List all existent availability zones * **Test assertion 1:** Verify the availability zone list length is greater than 0 -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' This test case evaluates the volume API ability of listing availability zones. @@ -156,7 +115,7 @@ Post conditions N/A -------------------------------------------------------------------- -Test Case 3 - Volume cloning operations with the Cinder v2 or v3 API +Test Case 2 - Volume cloning operations with the Cinder v2 or v3 API -------------------------------------------------------------------- Test case specification @@ -190,7 +149,7 @@ Test execution * Test action 7: Update the name of VOL3 and description with the original value * **Test assertion 6:** Verify VOL3's bootable flag is 'False' -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' This test case evaluates the volume API ability of creating a cloned volume from a source volume, @@ -210,7 +169,7 @@ Post conditions N/A -------------------------------------------------------------------------- -Test Case 4 - Image copy-to-volume operations with the Cinder v2 or v3 API +Test Case 3 - Image copy-to-volume operations with the Cinder v2 or v3 API -------------------------------------------------------------------------- Test case specification @@ -249,7 +208,7 @@ Test execution * Test action 11: Update the name of VOL3 and description with the original value * **Test assertion 8:** Verify VOL3's bootable flag is 'True' -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' This test case evaluates the volume API ability of updating volume's bootable flag and creating @@ -270,7 +229,7 @@ Post conditions N/A ---------------------------------------------------------------------------------- -Test Case 5 - Volume creation and deletion operations with the Cinder v2 or v3 API +Test Case 4 - Volume creation and deletion operations with the Cinder v2 or v3 API ---------------------------------------------------------------------------------- Test case specification @@ -320,7 +279,7 @@ Test execution * Test action 13: Create a volume with volume size '0' * **Test assertion 12:** Verify create volume failed, a bad request error is returned in the response -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' This test case evaluates the volume API ability of creating a volume, getting volume @@ -343,7 +302,7 @@ Post conditions N/A -------------------------------------------------------------------------------------- -Test Case 6 - Volume service extension listing operations with the Cinder v2 or v3 API +Test Case 5 - Volume service extension listing operations with the Cinder v2 or v3 API -------------------------------------------------------------------------------------- Test case specification @@ -366,7 +325,7 @@ Test execution * Test action 1: List all cinder service extensions * **Test assertion 1:** Verify all extensions are list in the extension list -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' This test case evaluates the volume API ability of listing all existent volume service extensions. @@ -381,7 +340,7 @@ Post conditions N/A ---------------------------------------------------------------- -Test Case 7 - Volume GET operations with the Cinder v2 or v3 API +Test Case 6 - Volume GET operations with the Cinder v2 or v3 API ---------------------------------------------------------------- Test case specification @@ -409,7 +368,7 @@ Test execution * Test action 3: Retrieve a volume with a nonexistent volume ID * **Test assertion 3:** Verify retrieve volume failed, a 'Not Found' error is returned in the response -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' This test case evaluates the volume API ability of getting volumes. @@ -425,7 +384,7 @@ Post conditions N/A -------------------------------------------------------------------- -Test Case 8 - Volume listing operations with the Cinder v2 or v3 API +Test Case 7 - Volume listing operations with the Cinder v2 or v3 API -------------------------------------------------------------------- Test case specification @@ -510,7 +469,7 @@ Test execution * Test action 22: List all existent volumes and paginate the volume list by desired volume IDs * **Test assertion 24:** Verify only the desired volumes are listed in the filtered volume list -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' This test case evaluates the volume API ability of getting a list of volumes and filtering the volume list. @@ -529,7 +488,7 @@ Post conditions N/A --------------------------------------------------------------------- -Test Case 9 - Volume metadata operations with the Cinder v2 or v3 API +Test Case 8 - Volume metadata operations with the Cinder v2 or v3 API --------------------------------------------------------------------- Test case specification @@ -561,7 +520,7 @@ Test execution * Test action 6: Update one metadata item 'key3' of VOL2 * **Test assertion 5:** Verify the metadata of VOL2 is updated -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' This test case evaluates the volume API ability of creating metadata for a volume, getting the @@ -580,9 +539,9 @@ Post conditions N/A ---------------------------------------------------------------------------------------- -Test Case 10 - Verification of read-only status on volumes with the Cinder v2 or v3 API ---------------------------------------------------------------------------------------- +-------------------------------------------------------------------------------------- +Test Case 9 - Verification of read-only status on volumes with the Cinder v2 or v3 API +-------------------------------------------------------------------------------------- Test case specification ----------------------- @@ -605,7 +564,7 @@ Test execution * Test action 2: Update a provided volume VOL1's read-only access mode to 'False' * **Test assertion 2:** Verify VOL1 is not in read-only access mode -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' This test case evaluates the volume API ability of setting and updating volume read-only access mode. @@ -620,61 +579,8 @@ Post conditions N/A -------------------------------------------------------------------------- -Test Case 11 - Volume reservation operations with the Cinder v2 or v3 API -------------------------------------------------------------------------- - -Test case specification ------------------------ - -tempest.api.volume.test_volumes_actions.VolumesActionsTest.test_reserve_unreserve_volume -tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_reserve_volume_with_negative_volume_status -tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_reserve_volume_with_nonexistent_volume_id -tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_unreserve_volume_with_nonexistent_volume_id - -Test preconditions ------------------- - -* Volume extension API - -Basic test flow execution description and pass/fail criteria ------------------------------------------------------------- - -Test execution -'''''''''''''' - -* Test action 1: Update a provided volume VOL1 as reserved -* **Test assertion 1:** Verify VOL1 is in 'attaching' status -* Test action 2: Update VOL1 as un-reserved -* **Test assertion 2:** Verify VOL1 is in 'available' status -* Test action 3: Update a provided volume VOL2 as reserved -* Test action 4: Update VOL2 as reserved again -* **Test assertion 3:** Verify update VOL2 status failed, a bad request error is returned in the response -* Test action 5: Update VOL2 as un-reserved -* Test action 6: Update a non-existent volume as reserved by using an invalid volume ID -* **Test assertion 4:** Verify update non-existent volume as reserved failed, a 'Not Found' error is returned in the response -* Test action 7: Update a non-existent volume as un-reserved by using an invalid volume ID -* **Test assertion 5:** Verify update non-existent volume as un-reserved failed, a 'Not Found' error is returned in the response - -Pass / fail criteria -'''''''''''''''''''' - -This test case evaluates the volume API ability of reserving and un-reserving volumes. -Specifically, the test verifies that: - -* Volume can be reserved and un-reserved. -* Update a non-existent volume as reserved is not allowed. -* Update a non-existent volume as un-reserved is not allowed. - -In order to pass this test, all test assertions listed in the test execution above need to pass. - -Post conditions ---------------- - -N/A - ---------------------------------------------------------------------------------------- -Test Case 12 - Volume snapshot creation/deletion operations with the Cinder v2 or v3 API +Test Case 10 - Volume snapshot creation/deletion operations with the Cinder v2 or v3 API ---------------------------------------------------------------------------------------- Test case specification @@ -753,7 +659,7 @@ Test execution * Test action 27: Create a snapshot from a volume by using an empty volume ID * **Test assertion 21:** Verify create snapshot failed, a 'Not Found' error is returned in the response -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' This test case evaluates the volume API ability of managing snapshot and snapshot metadata. @@ -784,7 +690,7 @@ Post conditions N/A -------------------------------------------------------------------- -Test Case 13 - Volume update operations with the Cinder v2 or v3 API +Test Case 11 - Volume update operations with the Cinder v2 or v3 API -------------------------------------------------------------------- Test case specification @@ -812,7 +718,7 @@ Test execution * Test action 3: Update a non-existent volume by using a random generated volume ID * **Test assertion 3:** Verify update volume failed, a 'Not Found' error is returned in the response -Pass / fail criteria +Pass / Fail criteria '''''''''''''''''''' This test case evaluates the volume API ability of updating volume attributes. @@ -828,3 +734,41 @@ Post conditions --------------- N/A + +------------------------------------------------------------------- +Test Case 12 - Volume list version operation with the Cinder v3 API +------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_versions.VersionsTest.test_list_versions + +Test preconditions +------------------ + +* Volume API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: List volume API versions +* **Test assertion 1:** The list versions operation is successful executed + +Pass / Fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of listing volume API versions. +Specifically, the test verifies that: + +* Successfully listing volume API versions. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A |