diff options
Diffstat (limited to 'docs/testing/user/testspecification')
21 files changed, 7679 insertions, 753 deletions
diff --git a/docs/testing/user/testspecification/dynamicnetwork/index.rst b/docs/testing/user/testspecification/dynamicnetwork/index.rst new file mode 100644 index 00000000..8fa084c7 --- /dev/null +++ b/docs/testing/user/testspecification/dynamicnetwork/index.rst @@ -0,0 +1,413 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Huawei Technologies Co.,Ltd + +===================================================== +Dynamic Network Runtime Operations test specification +===================================================== + +.. toctree:: + :maxdepth: 2 + +Scope +===== + +The dynamic network runtime operations test area evaluates the ability of the +system under test to support dynamic network runtime operations through the +life of a VNF (e.g. attach/detach, enable/disable, read stats). +The tests in this test area will evaluate IPv4 network runtime operations +functionality. These runtime operations includes hotpluging network interface, +detaching floating-ip from VM, attaching floating-ip to VM, updating subnet's +DNS, updating VM instance port admin state and updating router admin state. + +References +========== + +- DNS + + - https://docs.openstack.org/newton/networking-guide/config-dns-res.html + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test +area + +- API - Application Programming Interface +- DNS - Domain Name System +- ICMP - Internet Control Message Protocol +- MAC - Media Access Control +- NIC - Network Interface Controller +- NFVi - Network Functions Virtualization infrastructure +- SSH - Secure Shell +- TCP - Transmission Control Protocol +- VIM - Virtual Infrastructure Manager +- VM - Virtual Machine + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + +Test Area Structure +=================== + +The test area is structured based on dynamic network runtime operations. Each +test case is able to run independently, i.e. irrelevant of the state created by +a previous test. Specifically, every test performs clean-up operations which +return the system to the same state as before the test. + +Test Descriptions +================= + +API Used and Reference +---------------------- + +Security Groups: https://developer.openstack.org/api-ref/network/v2/index.html#security-groups-security-groups + +- create security group +- delete security group + +Networks: https://developer.openstack.org/api-ref/networking/v2/index.html#networks + +- create network +- delete network + +Routers and interface: https://developer.openstack.org/api-ref/networking/v2/index.html#routers-routers + +- create router +- update router +- delete router +- add interface to router + +Subnets: https://developer.openstack.org/api-ref/networking/v2/index.html#subnets + +- create subnet +- update subnet +- delete subnet + +Servers: https://developer.openstack.org/api-ref/compute/ + +- create keypair +- create server +- delete server +- add/assign floating IP +- disassociate floating IP + +Ports: https://developer.openstack.org/api-ref/networking/v2/index.html#ports + +- create port +- update port +- delete port + +Floating IPs: https://developer.openstack.org/api-ref/networking/v2/index.html#floating-ips-floatingips + +- create floating IP +- delete floating IP + +-------------------------------------- +Test Case 1 - Basic network operations +-------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops + +Test preconditions +------------------ + +* Nova has been configured to boot VMs with Neutron-managed networking +* Openstack nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a security group SG1, which has rules for allowing + incoming SSH and ICMP traffic +* Test action 2: Create a neutron network NET1 +* Test action 3: Create a tenant router R1 which routes traffic to public network +* Test action 4: Create a subnet SUBNET1 and add it as router interface +* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating + IP FIP1 (via R1) to VM1 +* **Test assertion 1:** Ping FIP1 and SSH to VM1 via FIP1 successfully +* **Test assertion 2:** Ping the internal gateway from VM1 successfully +* **Test assertion 3:** Ping the default gateway from VM1 using its floating IP + FIP1 successfully +* Test action 6: Detach FIP1 from VM1 +* **Test assertion 4:** VM1 becomes unreachable after FIP1 disassociated +* Test action 7: Create a new server VM2 with NET1, and associate floating IP FIP1 to VM2 +* **Test assertion 5:** Ping FIP1 and SSH to VM2 via FIP1 successfully +* Test action 8: Delete SG1, NET1, SUBNET1, R1, VM1, VM2 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of basic network operations. +Specifically, the test verifies that: + +* The Tempest host can ping VM's IP address. This implies, but does not + guarantee (see the ssh check that follows), that the VM has been assigned the + correct IP address and has connectivity to the Tempest host. + +* The Tempest host can perform key-based authentication to an ssh server hosted + at VM's IP address. This check guarantees that the IP address is associated + with the target VM. + +* The Tempest host can ssh into the VM via the IP address and successfully ping + the internal gateway address, implying connectivity to another VM on the same network. + +* The Tempest host can ssh into the VM via the IP address and successfully ping + the default gateway, implying external connectivity. + +* Detach the floating-ip from the VM and VM becomes unreachable. + +* Associate attached floating ip to a new VM and the new VM is reachable. + +* Floating IP status is updated correctly after each change. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------- +Test Case 2 - Hotplug network interface +--------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic + +Test preconditions +------------------ + +* Nova has been configured to boot VMs with Neutron-managed networking +* Compute interface_attach feature is enabled +* VM vnic_type is not set to 'direct' or 'macvtap' +* Openstack nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a security group SG1, which has rules for allowing + incoming SSH and ICMP traffic +* Test action 2: Create a neutron network NET1 +* Test action 3: Create a tenant router R1 which routes traffic to public network +* Test action 4: Create a subnet SUBNET1 and add it as router interface +* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating + IP FIP1 (via R1) to VM1 +* **Test assertion 1:** Ping FIP1 and SSH to VM1 with FIP1 successfully +* Test action 6: Create a second neutron network NET2 and subnet SUBNET2, and + attach VM1 to NET2 +* Test action 7: Get VM1's ethernet interface NIC2 for NET2 +* **Test assertion 2:** Ping NET2's internal gateway successfully +* Test action 8: Delete SG1, NET1, NET2, SUBNET1, SUBNET2, R1, NIC2, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of adding network to an active VM. +Specifically, the test verifies that: + +* New network interface can be added to an existing VM successfully. + +* The Tempest host can ssh into the VM via the IP address and successfully ping + the new network's internal gateway address, implying connectivity to the new network. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------- +Test Case 3 - Update subnet's configuration +------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details + +Test preconditions +------------------ + +* Nova has been configured to boot VMs with Neutron-managed networking +* DHCP client is available +* Tenant networks should be non-shared and isolated +* Openstack nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a security group SG1, which has rules for allowing + incoming SSH and ICMP traffic +* Test action 2: Create a neutron network NET1 +* Test action 3: Create a tenant router R1 which routes traffic to public network +* Test action 4: Create a subnet SUBNET1 and add it as router interface, + configure SUBNET1 with dns nameserver '1.2.3.4' +* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating + IP FIP1 (via R1) to VM1 +* **Test assertion 1:** Ping FIP1 and SSH to VM1 with FIP1 successfully +* **Test assertion 2:** Retrieve the VM1's configured dns and verify it matches + the one configured for SUBNET1 +* Test action 6: Update SUBNET1's dns to '9.8.7.6' +* **Test assertion 3:** After triggering the DHCP renew from the VM manually, + retrieve the VM1's configured dns and verify it has been successfully updated +* Test action 7: Delete SG1, NET1, SUBNET1, R1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of updating subnet's configurations. +Specifically, the test verifies that: + +* Updating subnet's DNS server configurations are affecting the VMs. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------- +Test Case 4 - Update VM port admin state +---------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_instance_port_admin_state + +Test preconditions +------------------ + +* Nova has been configured to boot VMs with Neutron-managed networking +* Network port_admin_state_change feature is enabled +* Openstack nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a security group SG1, which has rules for allowing + incoming SSH and ICMP traffic +* Test action 2: Create a neutron network NET1 +* Test action 3: Create a tenant router R1 which routes traffic to public network +* Test action 4: Create a subnet SUBNET1 and add it as router interface +* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating + IP FIP1 (via R1) to VM1 +* Test action 6: Create a server VM2 with SG1 and NET1, and assign a floating + IP FIP2 to VM2 +* Test action 7: Get a SSH client SSHCLNT1 to VM2 +* **Test assertion 1:** Ping FIP1 and SSH to VM1 with FIP1 successfully +* **Test assertion 2:** Ping FIP1 via SSHCLNT1 successfully +* Test action 8: Update admin_state_up attribute of VM1 port to False +* **Test assertion 3:** Ping FIP1 and SSH to VM1 with FIP1 failed +* **Test assertion 4:** Ping FIP1 via SSHCLNT1 failed +* Test action 9: Update admin_state_up attribute of VM1 port to True +* **Test assertion 5:** Ping FIP1 and SSH to VM1 with FIP1 successfully +* **Test assertion 6:** Ping FIP1 via SSHCLNT1 successfully +* Test action 10: Delete SG1, NET1, SUBNET1, R1, SSHCLNT1, VM1, VM2 and FIP1, FIP2 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the VM public and project connectivity status by changing VM port +admin_state_up to True & False. Specifically, the test verifies that: + +* Public and project connectivity is reachable before updating admin_state_up + attribute of VM port to False. + +* Public and project connectivity is unreachable after updating admin_state_up + attribute of VM port to False. + +* Public and project connectivity is reachable after updating admin_state_up + attribute of VM port from False to True. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------- +Test Case 5 - Update router admin state +--------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state + +Test preconditions +------------------ + +* Nova has been configured to boot VMs with Neutron-managed networking +* Multi-tenant networks capabilities +* Openstack nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a security group SG1, which has rules for allowing + incoming SSH and ICMP traffic +* Test action 2: Create a neutron network NET1 +* Test action 3: Create a tenant router R1 which routes traffic to public network +* Test action 4: Create a subnet SUBNET1 and add it as router interface +* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating + IP FIP1 (via R1) to VM1 +* **Test assertion 1:** Ping FIP1 and SSH to VM1 with FIP1 successfully +* Test action 6: Update admin_state_up attribute of R1 to False +* **Test assertion 2:** Ping FIP1 and SSH to VM1 with FIP1 failed +* Test action 7: Update admin_state_up attribute of R1 to True +* **Test assertion 3:** Ping FIP1 and SSH to VM1 with FIP1 successfully +* Test action 8: Delete SG1, NET1, SUBNET1, R1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the router public connectivity status by changing +router admin_state_up to True & False. +Specifically, the test verifies that: + +* Public connectivity is reachable before updating admin_state_up attribute of router to False. + +* Public connectivity is unreachable after updating admin_state_up attribute of router to False. + +* Public connectivity is reachable after updating admin_state_up attribute of router. + from False to True + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A diff --git a/docs/testing/user/testspecification/forwardingpackets/index.rst b/docs/testing/user/testspecification/forwardingpackets/index.rst new file mode 100644 index 00000000..ed2d55f0 --- /dev/null +++ b/docs/testing/user/testspecification/forwardingpackets/index.rst @@ -0,0 +1,145 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Huawei Technologies Co.,Ltd + +================================================== +Forwarding Packets in Data Path test specification +================================================== + +.. toctree:: + :maxdepth: 2 + +Scope +===== + +This test area evaluates the ability of the system under test to support basic packet forwarding. +The test in this test area will evaluate basic packet forwarding through virtual IPv4 networks +in data path, including creating server and verifying network connectivity to the created server +with ping operation using MTU sized packets. + +References +========== + +N/A + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test +area + +- API - Application Programming Interface +- ICMP - Internet Control Message Protocol +- MTU - Maximum Transmission Unit +- NFVi - Network Functions Virtualization infrastructure +- SSH - Secure Shell +- VIM - Virtual Infrastructure Manager +- VM - Virtual Machine + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + +Test Area Structure +=================== + +The test area is structured based on the basic operations of forwarding packets +in data path through virtual networks. Specifically, the test performs +clean-up operations which return the system to the same state as before the test. + +Test Descriptions +================= + +API Used and Reference +---------------------- + +Security Groups: https://developer.openstack.org/api-ref/network/v2/index.html#security-groups-security-groups + +- create security group +- delete security group + +Networks: https://developer.openstack.org/api-ref/networking/v2/index.html#networks + +- create network +- delete network + +Routers and interface: https://developer.openstack.org/api-ref/networking/v2/index.html#routers-routers + +- create router +- delete router +- add interface to router + +Subnets: https://developer.openstack.org/api-ref/networking/v2/index.html#subnets + +- create subnet +- delete subnet + +Servers: https://developer.openstack.org/api-ref/compute/ + +- create keypair +- create server +- delete server +- add/assign floating ip + +Ports: https://developer.openstack.org/api-ref/networking/v2/index.html#ports + +- create port +- delete port + +Floating IPs: https://developer.openstack.org/api-ref/networking/v2/index.html#floating-ips-floatingips + +- create floating ip +- delete floating ip + +---------------------------- +MTU Sized Frames Fit Through +---------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_mtu_sized_frames + +Test preconditions +------------------ + +* Nova has been configured to boot VMs with Neutron-managed networking +* Neutron net-mtu extension API +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a security group SG1, which has rules for allowing + incoming and outgoing SSH and ICMP traffic +* Test action 2: Create a neutron network NET1 +* Test action 3: Create a tenant router R1 which routes traffic to public network +* Test action 4: Create a subnet SUBNET1 and add it as router interface +* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating + ip FIP1 (via R1) to VM1 +* Test action 6: Set MTU size to be the default MTU size of the SUT's network +* Test action 7: Host sends MTU sized ICMP packets to VM1 using ``ping`` +* **Test assertion 1:** Ping FIP1 using MTU sized packets successfully +* Test action 8: SSH to VM1 with FIP1 +* **Test assertion 2:** SSH to VM1 with FIP1 successfully +* Test action 9: Delete SG1, NET1, SUBNET1, R1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the network connectivity using MTU sized frames. +Specifically, the test verifies that: + +* With Neutron net-mtu extension configured, MTU sized packets can fit through network. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A diff --git a/docs/testing/user/testspecification/highavailability/index.rst b/docs/testing/user/testspecification/highavailability/index.rst index e69de29b..715f84d0 100644 --- a/docs/testing/user/testspecification/highavailability/index.rst +++ b/docs/testing/user/testspecification/highavailability/index.rst @@ -0,0 +1,743 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, China Mobile and others. + +========================================== +OpenStack Services HA test specification +========================================== + +.. toctree:: +:maxdepth: + +Scope +===== + +The HA test area evaluates the ability of the System Under Test to support service +continuity and recovery from component failures on part of OpenStack controller services("nova-api", +"neutron-server", "keystone", "glance-api", "cinder-api") and on "load balancer" service. + +The tests in this test area will emulate component failures by killing the +processes of above target services, stressing the CPU load or blocking +disk I/O on the selected controller node, and then check if the impacted +services are still available and the killed processes are recovered on the +selected controller node within a given time interval. + + +References +================ + +This test area references the following specifications: + +- ETSI GS NFV-REL 001 + + - http://www.etsi.org/deliver/etsi_gs/NFV-REL/001_099/001/01.01.01_60/gs_nfv-rel001v010101p.pdf + +- OpenStack High Availability Guide + + - https://docs.openstack.org/ha-guide/ + + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test area + +- SUT - system under test +- Monitor - tools used to measure the service outage time and the process + outage time +- Service outage time - the outage time (seconds) of the specific OpenStack + service +- Process outage time - the outage time (seconds) from the specific processes + being killed to recovered + + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + +SUT is assumed to be in high availability configuration, which typically means +more than one controller nodes are in the System Under Test. + +Test Area Structure +==================== + +The HA test area is structured with the following test cases in a sequential +manner. + +Each test case is able to run independently. Preceding test case's failure will +not affect the subsequent test cases. + +Preconditions of each test case will be described in the following test +descriptions. + + +Test Descriptions +================= + +--------------------------------------------------------------- +Test Case 1 - Controller node OpenStack service down - nova-api +--------------------------------------------------------------- + +Short name +---------- + +opnfv.ha.tc001.nova-api_service_down + +Use case specification +---------------------- + +This test case verifies the service continuity capability in the face of the +software process failure. It kills the processes of OpenStack "nova-api" +service on the selected controller node, then checks whether the "nova-api" +service is still available during the failure, by creating a VM then deleting +the VM, and checks whether the killed processes are recovered within a given +time interval. + + +Test preconditions +------------------ + +There is more than one controller node, which is providing the "nova-api" +service for API end-point. +Denoted a controller node as Node1 in the following configuration. + + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for verifying service continuity and recovery +''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +The service continuity and process recovery capabilities of "nova-api" service +is evaluated by monitoring service outage time, process outage time, and results +of nova operations. + +Service outage time is measured by continuously executing "openstack server list" +command in loop and checking if the response of the command request is returned +with no failure. +When the response fails, the "nova-api" service is considered in outage. +The time between the first response failure and the last response failure is +considered as service outage time. + +Process outage time is measured by checking the status of "nova-api" processes on +the selected controller node. The time of "nova-api" processes being killed to +the time of the "nova-api" processes being recovered is the process outage time. +Process recovery is verified by checking the existence of "nova-api" processes. + +All nova operations are carried out correctly within a given time interval which +suggests that the "nova-api" service is continuously available. + +Test execution +'''''''''''''' +* Test action 1: Connect to Node1 through SSH, and check that "nova-api" + processes are running on Node1 +* Test action 2: Create a image with "openstack image create test-cirros + --file cirros-0.3.5-x86_64-disk.img --disk-format qcow2 --container-format bare" +* Test action 3: Execute"openstack flavor create m1.test --id auto --ram 512 + --disk 1 --vcpus 1" to create flavor "m1.test". +* Test action 4: Start two monitors: one for "nova-api" processes and the other + for "openstack server list" command. + Each monitor will run as an independent process +* Test action 5: Connect to Node1 through SSH, and then kill the "nova-api" + processes +* Test action 6: When "openstack server list" returns with no error, calculate + the service outage time, and execute command "openstack server create + --flavor m1.test --image test-cirros test-instance" +* Test action 7: Continuously Execute "openstack server show test-instance" + to check if the status of VM "test-instance" is "Active" +* Test action 8: If VM "test-instance" is "Active", execute "openstack server + delete test-instance", then execute "openstack server list" to check if the + VM is not in the list +* Test action 9: Continuously measure process outage time from the monitor until + the process outage time is more than 30s + +Pass / fail criteria +'''''''''''''''''''' + +The process outage time is less than 30s. + +The service outage time is less than 5s. + +The nova operations are carried out in above order and no errors occur. + +A negative result will be generated if the above is not met in completion. + +Post conditions +--------------- + +Restart the process of "nova-api" if they are not running. +Delete image with "openstack image delete test-cirros" +Delete flavor with "openstack flavor delete m1.test" + + +--------------------------------------------------------------------- +Test Case 2 - Controller node OpenStack service down - neutron-server +--------------------------------------------------------------------- + +Short name +---------- + +opnfv.ha.tc002.neutron-server_service_down + +Use case specification +---------------------- + +This test verifies the high availability of the "neutron-server" service +provided by OpenStack controller nodes. It kills the processes of OpenStack +"neutron-server" service on the selected controller node, then checks whether +the "neutron-server" service is still available, by creating a network and +deleting the network, and checks whether the killed processes are recovered. + +Test preconditions +------------------ + +There is more than one controller node, which is providing the "neutron-server" +service for API end-point. +Denoted a controller node as Node1 in the following configuration. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for monitoring high availability +'''''''''''''''''''''''''''''''''''''''''''' + +The high availability of "neutron-server" service is evaluated by monitoring +service outage time, process outage time, and results of neutron operations. + +Service outage time is tested by continuously executing "openstack router list" +command in loop and checking if the response of the command request is returned +with no failure. +When the response fails, the "neutron-server" service is considered in outage. +The time between the first response failure and the last response failure is +considered as service outage time. + +Process outage time is tested by checking the status of "neutron-server" +processes on the selected controller node. The time of "neutron-server" +processes being killed to the time of the "neutron-server" processes being +recovered is the process outage time. Process recovery is verified by checking +the existence of "neutron-server" processes. + +Test execution +'''''''''''''' + +* Test action 1: Connect to Node1 through SSH, and check that "neutron-server" + processes are running on Node1 +* Test action 2: Start two monitors: one for "neutron-server" process and the + other for "openstack router list" command. + Each monitor will run as an independent process. +* Test action 3: Connect to Node1 through SSH, and then kill the + "neutron-server" processes +* Test action 4: When "openstack router list" returns with no error, calculate + the service outage time, and execute "openstack network create test-network" +* Test action 5: Continuously executing "openstack network show test-network", + check if the status of "test-network" is "Active" +* Test action 6: If "test-network" is "Active", execute "openstack network + delete test-network", then execute "openstack network list" to check if the + "test-network" is not in the list +* Test action 7: Continuously measure process outage time from the monitor until + the process outage time is more than 30s + +Pass / fail criteria +'''''''''''''''''''' + +The process outage time is less than 30s. + +The service outage time is less than 5s. + +The neutron operations are carried out in above order and no errors occur. + +A negative result will be generated if the above is not met in completion. + +Post conditions +--------------- + +Restart the processes of "neutron-server" if they are not running. + + +--------------------------------------------------------------- +Test Case 3 - Controller node OpenStack service down - keystone +--------------------------------------------------------------- + +Short name +---------- + +opnfv.ha.tc003.keystone_service_down + +Use case specification +---------------------- + +This test verifies the high availability of the "keystone" service provided by +OpenStack controller nodes. It kills the processes of OpenStack "keystone" +service on the selected controller node, then checks whether the "keystone" +service is still available by executing command "openstack user list" and +whether the killed processes are recovered. + +Test preconditions +------------------ + +There is more than one controller node, which is providing the "keystone" +service for API end-point. +Denoted a controller node as Node1 in the following configuration. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for monitoring high availability +'''''''''''''''''''''''''''''''''''''''''''' + +The high availability of "keystone" service is evaluated by monitoring service +outage time and process outage time + +Service outage time is tested by continuously executing "openstack user list" +command in loop and checking if the response of the command request is reutrned +with no failure. +When the response fails, the "keystone" service is considered in outage. +The time between the first response failure and the last response failure is +considered as service outage time. + +Process outage time is tested by checking the status of "keystone" processes on +the selected controller node. The time of "keystone" processes being killed to +the time of the "keystone" processes being recovered is the process outage +time. Process recovery is verified by checking the existence of "keystone" +processes. + +Test execution +'''''''''''''' + +* Test action 1: Connect to Node1 through SSH, and check that "keystone" + processes are running on Node1 +* Test action 2: Start two monitors: one for "keystone" process and the other + for "openstack user list" command. + Each monitor will run as an independent process. +* Test action 3: Connect to Node1 through SSH, and then kill the "keystone" + processes +* Test action 4: Calculate the service outage time and process outage time +* Test action 5: The test passes if process outage time is less than 20s and + service outage time is less than 5s +* Test action 6: Continuously measure process outage time from the monitor until + the process outage time is more than 30s + +Pass / fail criteria +'''''''''''''''''''' + +The process outage time is less than 30s. + +The service outage time is less than 5s. + +A negative result will be generated if the above is not met in completion. + +Post conditions +--------------- + +Restart the processes of "keystone" if they are not running. + + +----------------------------------------------------------------- +Test Case 4 - Controller node OpenStack service down - glance-api +----------------------------------------------------------------- + +Short name +---------- + +opnfv.ha.tc004.glance-api_service_down + +Use case specification +---------------------- + +This test verifies the high availability of the "glance-api" service provided +by OpenStack controller nodes. It kills the processes of OpenStack "glance-api" +service on the selected controller node, then checks whether the "glance-api" +service is still available, by creating image and deleting image, and checks +whether the killed processes are recovered. + +Test preconditions +------------------ + +There is more than one controller node, which is providing the "glance-api" +service for API end-point. +Denoted a controller node as Node1 in the following configuration. + + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for monitoring high availability +'''''''''''''''''''''''''''''''''''''''''''' + +The high availability of "glance-api" service is evaluated by monitoring +service outage time, process outage time, and results of glance operations. + +Service outage time is tested by continuously executing "openstack image list" +command in loop and checking if the response of the command request is returned +with no failure. +When the response fails, the "glance-api" service is considered in outage. +The time between the first response failure and the last response failure is +considered as service outage time. + +Process outage time is tested by checking the status of "glance-api" processes +on the selected controller node. The time of "glance-api" processes being +killed to the time of the "glance-api" processes being recovered is the process +outage time. Process recovery is verified by checking the existence of +"glance-api" processes. + +Test execution +'''''''''''''' + +* Test action 1: Connect to Node1 through SSH, and check that "glance-api" + processes are running on Node1 +* Test action 2: Start two monitors: one for "glance-api" process and the other + for "openstack image list" command. + Each monitor will run as an independent process. +* Test action 3: Connect to Node1 through SSH, and then kill the "glance-api" + processes +* Test action 4: When "openstack image list" returns with no error, calculate + the service outage time, and execute "openstack image create test-image + --file cirros-0.3.5-x86_64-disk.img --disk-format qcow2 --container-format bare" +* Test action 5: Continuously execute "openstack image show test-image", check + if status of "test-image" is "active" +* Test action 6: If "test-image" is "active", execute "openstack image delete + test-image". Then execute "openstack image list" to check if "test-image" is + not in the list +* Test action 7: Continuously measure process outage time from the monitor until + the process outage time is more than 30s + +Pass / fail criteria +'''''''''''''''''''' + +The process outage time is less than 30s. + +The service outage time is less than 5s. + +The glance operations are carried out in above order and no errors occur. + +A negative result will be generated if the above is not met in completion. + +Post conditions +--------------- + +Restart the processes of "glance-api" if they are not running. + +Delete image with "openstack image delete test-image". + + +----------------------------------------------------------------- +Test Case 5 - Controller node OpenStack service down - cinder-api +----------------------------------------------------------------- + +Short name +---------- + +opnfv.ha.tc005.cinder-api_service_down + +Use case specification +---------------------- + +This test verifies the high availability of the "cinder-api" service provided +by OpenStack controller nodes. It kills the processes of OpenStack "cinder-api" +service on the selected controller node, then checks whether the "cinder-api" +service is still available by executing command "openstack volume list" and +whether the killed processes are recovered. + +Test preconditions +------------------ + +There is more than one controller node, which is providing the "cinder-api" +service for API end-point. +Denoted a controller node as Node1 in the following configuration. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for monitoring high availability +'''''''''''''''''''''''''''''''''''''''''''' + +The high availability of "cinder-api" service is evaluated by monitoring +service outage time and process outage time + +Service outage time is tested by continuously executing "openstack volume list" +command in loop and checking if the response of the command request is returned +with no failure. +When the response fails, the "cinder-api" service is considered in outage. +The time between the first response failure and the last response failure is +considered as service outage time. + +Process outage time is tested by checking the status of "cinder-api" processes +on the selected controller node. The time of "cinder-api" processes being +killed to the time of the "cinder-api" processes being recovered is the process +outage time. Process recovery is verified by checking the existence of +"cinder-api" processes. + +Test execution +'''''''''''''' + +* Test action 1: Connect to Node1 through SSH, and check that "cinder-api" + processes are running on Node1 +* Test action 2: Start two monitors: one for "cinder-api" process and the other + for "openstack volume list" command. + Each monitor will run as an independent process. +* Test action 3: Connect to Node1 through SSH, and then execute kill the + "cinder-api" processes +* Test action 4: Continuously measure service outage time from the monitor until + the service outage time is more than 5s +* Test action 5: Continuously measure process outage time from the monitor until + the process outage time is more than 30s + +Pass / fail criteria +'''''''''''''''''''' + +The process outage time is less than 30s. + +The service outage time is less than 5s. + +The cinder operations are carried out in above order and no errors occur. + +A negative result will be generated if the above is not met in completion. + +Post conditions +--------------- + +Restart the processes of "cinder-api" if they are not running. + + +------------------------------------------------------------ +Test Case 6 - Controller Node CPU Overload High Availability +------------------------------------------------------------ + +Short name +---------- + +opnfv.ha.tc006.cpu_overload + +Use case specification +---------------------- + +This test verifies the availability of services when one of the controller node +suffers from heavy CPU overload. When the CPU usage of the specified controller +node is up to 100%, which breaks down the OpenStack services on this node, +the Openstack services should continue to be available. This test case stresses +the CPU usage of a specific controller node to 100%, then checks whether all +services provided by the SUT are still available with the monitor tools. + +Test preconditions +------------------ + +There is more than one controller node, which is providing the "cinder-api", +"neutron-server", "glance-api" and "keystone" services for API end-point. +Denoted a controller node as Node1 in the following configuration. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for monitoring high availability +'''''''''''''''''''''''''''''''''''''''''''' + +The high availability of related OpenStack service is evaluated by monitoring service +outage time + +Service outage time is tested by continuously executing "openstack router list", +"openstack stack list", "openstack volume list", "openstack image list" commands +in loop and checking if the response of the command request is returned with no +failure. +When the response fails, the related service is considered in outage. The time +between the first response failure and the last response failure is considered +as service outage time. + + +Methodology for stressing CPU usage +''''''''''''''''''''''''''''''''''' + +To evaluate the high availability of target OpenStack service under heavy CPU +load, the test case will first get the number of logical CPU cores on the +target controller node by shell command, then use the number to execute 'dd' +command to continuously copy from /dev/zero and output to /dev/null in loop. +The 'dd' operation only uses CPU, no I/O operation, which is ideal for +stressing the CPU usage. + +Since the 'dd' command is continuously executed and the CPU usage rate is +stressed to 100%, the scheduler will schedule each 'dd' command to be +processed on a different logical CPU core. Eventually to achieve all logical +CPU cores usage rate to 100%. + +Test execution +'''''''''''''' + +* Test action 1: Start four monitors: one for "openstack image list" command, + one for "openstack router list" command, one for "openstack stack list" + command and the last one for "openstack volume list" command. Each monitor + will run as an independent process. +* Test action 2: Connect to Node1 through SSH, and then stress all logical CPU + cores usage rate to 100% +* Test action 3: Continuously measure all the service outage times until they are + more than 5s +* Test action 4: Kill the process that stresses the CPU usage + +Pass / fail criteria +'''''''''''''''''''' + +All the service outage times are less than 5s. + +A negative result will be generated if the above is not met in completion. + +Post conditions +--------------- + +No impact on the SUT. + + +----------------------------------------------------------------- +Test Case 7 - Controller Node Disk I/O Overload High Availability +----------------------------------------------------------------- + +Short name +---------- + +opnfv.ha.tc007.disk_I/O_overload + +Use case specification +---------------------- + +This test verifies the high availability of control node. When the disk I/O of +the specific disk is overload, which breaks down the OpenStack services on this +node, the read and write services should continue to be available. This test +case blocks the disk I/O of the specific controller node, then checks whether +the services that need to read or write the disk of the controller node are +available with some monitor tools. + +Test preconditions +------------------ + +There is more than one controller node. +Denoted a controller node as Node1 in the following configuration. +The controller node has at least 20GB free disk space. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for monitoring high availability +'''''''''''''''''''''''''''''''''''''''''''' + +The high availability of nova service is evaluated by monitoring +service outage time + +Service availability is tested by continuously executing +"openstack flavor list" command in loop and checking if the response of the +command request is returned with no failure. +When the response fails, the related service is considered in outage. + + +Methodology for stressing disk I/O +'''''''''''''''''''''''''''''''''' + +To evaluate the high availability of target OpenStack service under heavy I/O +load, the test case will execute shell command on the selected controller node +to continuously writing 8kb blocks to /test.dbf + +Test execution +'''''''''''''' + +* Test action 1: Connect to Node1 through SSH, and then stress disk I/O by + continuously writing 8kb blocks to /test.dbf +* Test action 2: Start a monitor: for "openstack flavor list" command +* Test action 3: Create a flavor called "test-001" +* Test action 4: Check whether the flavor "test-001" is created +* Test action 5: Continuously measure service outage time from the monitor + until the service outage time is more than 5s +* Test action 6: Stop writing to /test.dbf and delete file /test.dbf + +Pass / fail criteria +'''''''''''''''''''' + +The service outage time is less than 5s. + +The nova operations are carried out in above order and no errors occur. + +A negative result will be generated if the above is not met in completion. + +Post conditions +--------------- + +Delete flavor with "openstack flavor delete test-001". + +-------------------------------------------------------------------- +Test Case 8 - Controller Load Balance as a Service High Availability +-------------------------------------------------------------------- + +Short name +---------- + +opnfv.ha.tc008.load_balance_service_down + +Use case specification +---------------------- + +This test verifies the high availability of "load balancer" service. When +the "load balancer" service of a specified controller node is killed, whether +"load balancer" service on other controller nodes will work, and whether the +controller node will restart the "load balancer" service are checked. This +test case kills the processes of "load balancer" service on the selected +controller node, then checks whether the request of the related OpenStack +command is processed with no failure and whether the killed processes are +recovered. + +Test preconditions +------------------ + +There is more than one controller node, which is providing the "load balancer" +service for rest-api. Denoted as Node1 in the following configuration. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for monitoring high availability +'''''''''''''''''''''''''''''''''''''''''''' + +The high availability of "load balancer" service is evaluated by monitoring +service outage time and process outage time + +Service outage time is tested by continuously executing "openstack image list" +command in loop and checking if the response of the command request is returned +with no failure. +When the response fails, the "load balancer" service is considered in outage. +The time between the first response failure and the last response failure is +considered as service outage time. + +Process outage time is tested by checking the status of processes of "load +balancer" service on the selected controller node. The time of those processes +being killed to the time of those processes being recovered is the process +outage time. +Process recovery is verified by checking the existence of processes of "load +balancer" service. + +Test execution +'''''''''''''' + +* Test action 1: Connect to Node1 through SSH, and check that processes of + "load balancer" service are running on Node1 +* Test action 2: Start two monitors: one for processes of "load balancer" + service and the other for "openstack image list" command. Each monitor will + run as an independent process +* Test action 3: Connect to Node1 through SSH, and then kill the processes of + "load balancer" service +* Test action 4: Continuously measure service outage time from the monitor until + the service outage time is more than 5s +* Test action 5: Continuously measure process outage time from the monitor until + the process outage time is more than 30s + +Pass / fail criteria +'''''''''''''''''''' + +The process outage time is less than 30s. + +The service outage time is less than 5s. + +A negative result will be generated if the above is not met in completion. + +Post conditions +--------------- +Restart the processes of "load balancer" if they are not running. + + + diff --git a/docs/testing/user/testspecification/ipv6/index.rst b/docs/testing/user/testspecification/ipv6/index.rst new file mode 100644 index 00000000..c3dc844b --- /dev/null +++ b/docs/testing/user/testspecification/ipv6/index.rst @@ -0,0 +1,1787 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV + +======================== +IPv6 test specification +======================== + +.. toctree:: + :maxdepth: 2 + +Scope +===== + +The IPv6 test area will evaluate the ability for a SUT to support IPv6 +Tenant Network features and functionality. The tests in this test area will +evaluate, + +- network, subnet, port, router API CRUD operations +- interface add and remove operations +- security group and security group rule API CRUD operations +- IPv6 address assignment with dual stack, dual net, multiprefix in mode DHCPv6 stateless or SLAAC + +References +================ + +- upstream openstack API reference + + - http://developer.openstack.org/api-ref + +- upstream openstack IPv6 reference + + - https://docs.openstack.org/newton/networking-guide/config-ipv6.html + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test area + +- API - Application Programming Interface +- CIDR - Classless Inter-Domain Routing +- CRUD - Create, Read, Update, and Delete +- DHCP - Dynamic Host Configuration Protocol +- DHCPv6 - Dynamic Host Configuration Protocol version 6 +- ICMP - Internet Control Message Protocol +- NFVI - Network Functions Virtualization Infrastructure +- NIC - Network Interface Controller +- RA - Router Advertisements +- radvd - The Router Advertisement Daemon +- SDN - Software Defined Network +- SLAAC - Stateless Address Auto Configuration +- TCP - Transmission Control Protocol +- UDP - User Datagram Protocol +- VM - Virtual Machine +- vNIC - virtual Network Interface Card + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVI and VIM deployed with a Pharos compliant infrastructure. + +Test Area Structure +==================== + +The test area is structured based on network, port and subnet operations. Each test case +is able to run independently, i.e. irrelevant of the state created by a previous test. + +Test Descriptions +================= + +API Used and Reference +---------------------- + +Networks: https://developer.openstack.org/api-ref/networking/v2/index.html#networks + +- show network details +- update network +- delete network +- list networks +- create netowrk +- bulk create networks + +Subnets: https://developer.openstack.org/api-ref/networking/v2/index.html#subnets + +- list subnets +- create subnet +- bulk create subnet +- show subnet details +- update subnet +- delete subnet + +Routers and interface: https://developer.openstack.org/api-ref/networking/v2/index.html#routers-routers + +- list routers +- create router +- show router details +- update router +- delete router +- add interface to router +- remove interface from router + +Ports: https://developer.openstack.org/api-ref/networking/v2/index.html#ports + +- show port details +- update port +- delete port +- list port +- create port +- bulk create ports + +Security groups: https://developer.openstack.org/api-ref/networking/v2/index.html#security-groups-security-groups + +- list security groups +- create security groups +- show security group +- update security group +- delete security group + +Security groups rules: https://developer.openstack.org/api-ref/networking/v2/index.html#security-group-rules-security-group-rules + +- list security group rules +- create security group rule +- show security group rule +- delete security group rule + +Servers: https://developer.openstack.org/api-ref/compute/ + +- list servers +- create server +- create multiple servers +- list servers detailed +- show server details +- update server +- delete server + +------------------------------------------------------------------ +Test Case 1 - Create and Delete Bulk Network, IPv6 Subnet and Port +------------------------------------------------------------------ + +Short name +---------- + +opnfv.ipv6.bulk_network_subnet_port_create_delete + +Use case specification +---------------------- + +This test case evaluates the SUT API ability of creating and deleting multiple networks, +IPv6 subnets, ports in one request, the reference is, + +tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network +tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet +tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create 2 networks using bulk create, storing the "id" parameters returned in the response +* Test action 2: List all networks, verifying the two network id's are found in the list +* **Test assertion 1:** The two "id" parameters are found in the network list +* Test action 3: Delete the 2 created networks using the stored network ids +* Test action 4: List all networks, verifying the network ids are no longer present +* **Test assertion 2:** The two "id" parameters are not present in the network list +* Test action 5: Create 2 networks using bulk create, storing the "id" parameters returned in the response +* Test action 6: Create an IPv6 subnets on each of the two networks using bulk create commands, + storing the associated "id" parameters +* Test action 7: List all subnets, verify the IPv6 subnets are found in the list +* **Test assertion 3:** The two IPv6 subnet "id" parameters are found in the network list +* Test action 8: Delete the 2 IPv6 subnets using the stored "id" parameters +* Test action 9: List all subnets, verify the IPv6 subnets are no longer present in the list +* **Test assertion 4:** The two IPv6 subnet "id" parameters, are not present in list +* Test action 10: Delete the 2 networks created in test action 5, using the stored network ids +* Test action 11: List all networks, verifying the network ids are no longer present +* **Test assertion 5:** The two "id" parameters are not present in the network list +* Test action 12: Create 2 networks using bulk create, storing the "id" parameters returned in the response +* Test action 13: Create a port on each of the two networks using bulk create commands, + storing the associated "port_id" parameters +* Test action 14: List all ports, verify the port_ids are found in the list +* **Test assertion 6:** The two "port_id" parameters are found in the ports list +* Test action 15: Delete the 2 ports using the stored "port_id" parameters +* Test action 16: List all ports, verify port_ids are no longer present in the list +* **Test assertion 7:** The two "port_id" parameters, are not present in list +* Test action 17: Delete the 2 networks created in test action 12, using the stored network ids +* Test action 18: List all networks, verifying the network ids are no longer present +* **Test assertion 8:** The two "id" parameters are not present in the network list + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to use bulk create commands to create networks, IPv6 subnets and ports on +the SUT API. Specifically it verifies that: + +* Bulk network create commands return valid "id" parameters which are reported in the list commands +* Bulk IPv6 subnet commands return valid "id" parameters which are reported in the list commands +* Bulk port commands return valid "port_id" parameters which are reported in the list commands +* All items created using bulk create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +N/A + +------------------------------------------------------------------- +Test Case 2 - Create, Update and Delete an IPv6 Network and Subnet +------------------------------------------------------------------- + +Short name +----------- + +opnfv.ipv6.network_subnet_create_update_delete + +Use case specification +---------------------- + +This test case evaluates the SUT API ability of creating, updating, deleting +network and IPv6 subnet with the network, the reference is + +tempest.api.network.test_networks.NetworksIpV6Test.test_create_update_delete_network_subnet + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create a network, storing the "id" and "status" parameters returned + in the response +* Test action 2: Verify the value of the created network's "status" is ACTIVE +* **Test assertion 1:** The created network's "status" is ACTIVE +* Test action 3: Update this network with a new_name +* Test action 4: Verify the network's name equals the new_name +* **Test assertion 2:** The network's name equals to the new_name after name updating +* Test action 5: Create an IPv6 subnet within the network, storing the "id" parameters + returned in the response +* Test action 6: Update this IPv6 subnet with a new_name +* Test action 7: Verify the IPv6 subnet's name equals the new_name +* **Test assertion 3:** The IPv6 subnet's name equals to the new_name after name updating +* Test action 8: Delete the IPv6 subnet created in test action 5, using the stored subnet id +* Test action 9: List all subnets, verifying the subnet id is no longer present +* **Test assertion 4:** The IPv6 subnet "id" is not present in the subnet list +* Test action 10: Delete the network created in test action 1, using the stored network id +* Test action 11: List all networks, verifying the network id is no longer present +* **Test assertion 5:** The network "id" is not present in the network list + + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to create, update, delete network, IPv6 subnet on the +SUT API. Specifically it verifies that: + +* Create network commands return ACTIVE "status" parameters which are reported in the list commands +* Update network commands return updated "name" parameters which equals to the "name" used +* Update subnet commands return updated "name" parameters which equals to the "name" used +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +------------------------------------------------- +Test Case 3 - Check External Network Visibility +------------------------------------------------- + +Short name +----------- + +opnfv.ipv6.external_network_visibility + +Use case specification +---------------------- + +This test case verifies user can see external networks but not subnets, the reference is, + +tempest.api.network.test_networks.NetworksIpV6Test.test_external_network_visibility + +Test preconditions +------------------ + +1. The SUT has at least one external network. +2. In the external network list, there is no network without external router, i.e., +all networks in this list are with external router. +3. There is one external network with configured public network id and there is +no subnet on this network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: List all networks with external router, storing the "id"s parameters returned in the response +* Test action 2: Verify list in test action 1 is not empty +* **Test assertion 1:** The network with external router list is not empty +* Test action 3: List all netowrks without external router in test action 1 list +* Test action 4: Verify list in test action 3 is empty +* **Test assertion 2:** networks without external router in the external network + list is empty +* Test action 5: Verify the configured public network id is found in test action 1 stored "id"s +* **Test assertion 3:** the public network id is found in the external network "id"s +* Test action 6: List the subnets of the external network with the configured + public network id +* Test action 7: Verify list in test action 6 is empty +* **Test assertion 4:** There is no subnet of the external network with the configured + public network id + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to use list commands to list external networks, pre-configured +public network. Specifically it verifies that: + +* Network list commands to find visible networks with external router +* Network list commands to find visible network with pre-configured public network id +* Subnet list commands to find no subnet on the pre-configured public network + +Post conditions +--------------- + +None + +--------------------------------------------- +Test Case 4 - List IPv6 Networks and Subnets +--------------------------------------------- + +Short name +----------- + +opnfv.ipv6.network_subnet_list + +Use case specification +---------------------- + +This test case evaluates the SUT API ability of listing netowrks, +subnets after creating a network and an IPv6 subnet, the reference is + +tempest.api.network.test_networks.NetworksIpV6Test.test_list_networks +tempest.api.network.test_networks.NetworksIpV6Test.test_list_subnets + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create a network, storing the "id" parameter returned in the response +* Test action 2: List all networks, verifying the network id is found in the list +* **Test assertion 1:** The "id" parameter is found in the network list +* Test action 3: Create an IPv6 subnet of the network created in test action 1. + storing the "id" parameter returned in the response +* Test action 4: List all subnets of this network, verifying the IPv6 subnet id + is found in the list +* **Test assertion 2:** The "id" parameter is found in the IPv6 subnet list +* Test action 5: Delete the IPv6 subnet using the stored "id" parameters +* Test action 6: List all subnets, verify subnet_id is no longer present in the list +* **Test assertion 3:** The IPv6 subnet "id" parameter is not present in list +* Test action 7: Delete the network created in test action 1, using the stored network ids +* Test action 8: List all networks, verifying the network id is no longer present +* **Test assertion 4:** The network "id" parameter is not present in the network list + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to use create commands to create network, IPv6 subnet, list +commands to list the created networks, IPv6 subnet on the SUT API. Specifically it verifies that: + +* Create commands to create network, IPv6 subnet +* List commands to find that netowrk, IPv6 subnet in the all networks, subnets list after creating +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +------------------------------------------------------------- +Test Case 5 - Show Details of an IPv6 Network and Subnet +------------------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.network_subnet_show + +Use case specification +---------------------- + +This test case evaluates the SUT API ability of showing the network, subnet +details, the reference is, + +tempest.api.network.test_networks.NetworksIpV6Test.test_show_network +tempest.api.network.test_networks.NetworksIpV6Test.test_show_subnet + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create a network, storing the "id" and "name" parameter returned in the response +* Test action 2: Show the network id and name, verifying the network id and name equal to the + "id" and "name" stored in test action 1 +* **Test assertion 1:** The id and name equal to the "id" and "name" stored in test action 1 +* Test action 3: Create an IPv6 subnet of the network, storing the "id" and CIDR parameter + returned in the response +* Test action 4: Show the details of the created IPv6 subnet, verifying the + id and CIDR in the details are equal to the stored id and CIDR in test action 3. +* **Test assertion 2:** The "id" and CIDR in show details equal to "id" and CIDR stored in test action 3 +* Test action 5: Delete the IPv6 subnet using the stored "id" parameter +* Test action 6: List all subnets on the network, verify the IPv6 subnet id is no longer present in the list +* **Test assertion 3:** The IPv6 subnet "id" parameter is not present in list +* Test action 7: Delete the network created in test action 1, using the stored network id +* Test action 8: List all networks, verifying the network id is no longer present +* **Test assertion 4:** The "id" parameter is not present in the network list + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to use create commands to create network, IPv6 subnet and show +commands to show network, IPv6 subnet details on the SUT API. Specifically it verifies that: + +* Network show commands return correct "id" and "name" parameter which equal to the returned response in the create commands +* IPv6 subnet show commands return correct "id" and CIDR parameter which equal to the returned response in the create commands +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +------------------------------------------------------------- +Test Case 6 - Create an IPv6 Port in Allowed Allocation Pools +------------------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.port_create_in_allocation_pool + +Use case specification +---------------------- + +This test case evaluates the SUT API ability of creating +an IPv6 subnet within allowed IPv6 address allocation pool and creating +a port whose address is in the range of the pool, the reference is, + +tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools + +Test preconditions +------------------ + +There should be an IPv6 CIDR configuration, which prefixlen is less than 126. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create a network, storing the "id" parameter returned in the response +* Test action 2: Check the allocation pools configuration, verifying the prefixlen + of the IPv6 CIDR configuration is less than 126. +* **Test assertion 1:** The prefixlen of the IPv6 CIDR configuration is less than 126 +* Test action 3: Get the allocation pool by setting the start_ip and end_ip + based on the IPv6 CIDR configuration. +* Test action 4: Create an IPv6 subnet of the network within the allocation pools, + storing the "id" parameter returned in the response +* Test action 5: Create a port of the network, storing the "id" parameter returned in the response +* Test action 6: Verify the port's id is in the range of the allocation pools which is got is test action 3 +* **Test assertion 2:** the port's id is in the range of the allocation pools +* Test action 7: Delete the port using the stored "id" parameter +* Test action 8: List all ports, verify the port id is no longer present in the list +* **Test assertion 3:** The port "id" parameter is not present in list +* Test action 9: Delete the IPv6 subnet using the stored "id" parameter +* Test action 10: List all subnets on the network, verify the IPv6 subnet id is no longer present in the list +* **Test assertion 4:** The IPv6 subnet "id" parameter is not present in list +* Test action 11: Delete the network created in test action 1, using the stored network id +* Test action 12: List all networks, verifying the network id is no longer present +* **Test assertion 5:** The "id" parameter is not present in the network list + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to use create commands to create an IPv6 subnet within allowed +IPv6 address allocation pool and create a port whose address is in the range of the pool. Specifically it verifies that: + +* IPv6 subnet create command to create an IPv6 subnet within allowed IPv6 address allocation pool +* Port create command to create a port whose id is in the range of the allocation pools +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +------------------------------------------------------------- +Test Case 7 - Create an IPv6 Port with Empty Security Groups +------------------------------------------------------------- + +Short name +----------- + +opnfv.ipv6.port_create_empty_security_group + +Use case specification +---------------------- + +This test case evaluates the SUT API ability of creating port with empty +security group, the reference is, + +tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_with_no_securitygroups + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create a network, storing the "id" parameter returned in the response +* Test action 2: Create an IPv6 subnet of the network, storing the "id" parameter returned in the response +* Test action 3: Create a port of the network with an empty security group, storing the "id" parameter returned in the response +* Test action 4: Verify the security group of the port is not none but is empty +* **Test assertion 1:** the security group of the port is not none but is empty +* Test action 5: Delete the port using the stored "id" parameter +* Test action 6: List all ports, verify the port id is no longer present in the list +* **Test assertion 2:** The port "id" parameter is not present in list +* Test action 7: Delete the IPv6 subnet using the stored "id" parameter +* Test action 8: List all subnets on the network, verify the IPv6 subnet id is no longer present in the list +* **Test assertion 3:** The IPv6 subnet "id" parameter is not present in list +* Test action 9: Delete the network created in test action 1, using the stored network id +* Test action 10: List all networks, verifying the network id is no longer present +* **Test assertion 4:** The "id" parameter is not present in the network list + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to use create commands to create port with +empty security group of the SUT API. Specifically it verifies that: + +* Port create commands to create a port with an empty security group +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +----------------------------------------------------- +Test Case 8 - Create, Update and Delete an IPv6 Port +----------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.port_create_update_delete + +Use case specification +---------------------- + +This test case evaluates the SUT API ability of creating, updating, +deleting IPv6 port, the reference is, + +tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_update_delete_port + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create a network, storing the "id" parameter returned in the response +* Test action 2: Create a port of the network, storing the "id" and "admin_state_up" parameters + returned in the response +* Test action 3: Verify the value of port's 'admin_state_up' is True +* **Test assertion 1:** the value of port's 'admin_state_up' is True after creating +* Test action 4: Update the port's name with a new_name and set port's admin_state_up to False, + storing the name and admin_state_up parameters returned in the response +* Test action 5: Verify the stored port's name equals to new_name and the port's admin_state_up is False. +* **Test assertion 2:** the stored port's name equals to new_name and the port's admin_state_up is False +* Test action 6: Delete the port using the stored "id" parameter +* Test action 7: List all ports, verify the port is no longer present in the list +* **Test assertion 3:** The port "id" parameter is not present in list +* Test action 8: Delete the network created in test action 1, using the stored network id +* Test action 9: List all networks, verifying the network id is no longer present +* **Test assertion 4:** The "id" parameter is not present in the network list + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to use create/update/delete commands to create/update/delete port +of the SUT API. Specifically it verifies that: + +* Port create commands return True of 'admin_state_up' in response +* Port update commands to update 'name' to new_name and 'admin_state_up' to false +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +------------------------------ +Test Case 9 - List IPv6 Ports +------------------------------ + +Short name +---------- + +opnfv.ipv6.tc009.port_list + +Use case specification +---------------------- + +This test case evaluates the SUT ability of creating a port on a network and +finding the port in the all ports list, the reference is, + +tempest.api.network.test_ports.PortsIpV6TestJSON.test_list_ports + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create a network, storing the "id" parameter returned in the response +* Test action 2: Create a port of the network, storing the "id" parameter returned in the response +* Test action 3: List all ports, verify the port id is found in the list +* **Test assertion 1:** The "id" parameter is found in the port list +* Test action 4: Delete the port using the stored "id" parameter +* Test action 5: List all ports, verify the port is no longer present in the list +* **Test assertion 2:** The port "id" parameter is not present in list +* Test action 6: Delete the network created in test action 1, using the stored network id +* Test action 7: List all networks, verifying the network id is no longer present +* **Test assertion 3:** The "id" parameter is not present in the network list + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to use list commands to list the networks and ports on +the SUT API. Specifically it verifies that: + +* Port list command to list all ports, the created port is found in the list. +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +------------------------------------------------------- +Test Case 10 - Show Key/Valus Details of an IPv6 Port +------------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.tc010.port_show_details + +Use case specification +---------------------- + +This test case evaluates the SUT ability of showing the port +details, the values in the details should be equal to the values to create the port, +the reference is, + +tempest.api.network.test_ports.PortsIpV6TestJSON.test_show_port + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create a network, storing the "id" parameter returned in the response +* Test action 2: Create a port of the network, storing the "id" parameter returned in the response +* Test action 3: Show the details of the port, verify the stored port's id + in test action 2 exists in the details +* **Test assertion 1:** The "id" parameter is found in the port shown details +* Test action 4: Verify the values in the details of the port are the same as the values + to create the port +* **Test assertion 2:** The values in the details of the port are the same as the values + to create the port +* Test action 5: Delete the port using the stored "id" parameter +* Test action 6: List all ports, verify the port is no longer present in the list +* **Test assertion 3:** The port "id" parameter is not present in list +* Test action 7: Delete the network created in test action 1, using the stored network id +* Test action 8: List all networks, verifying the network id is no longer present +* **Test assertion 4:** The "id" parameter is not present in the network list + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to use show commands to show port details on the SUT API. +Specifically it verifies that: + +* Port show commands to show the details of the port, whose id is in the details +* Port show commands to show the details of the port, whose values are the same as the values + to create the port +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +--------------------------------------------------------- +Test Case 11 - Add Multiple Interfaces for an IPv6 Router +--------------------------------------------------------- + +Short name +----------- + +opnfv.ipv6.router_add_multiple_interface + +Use case specification +---------------------- + +This test case evaluates the SUT ability of adding multiple interface +to a router, the reference is, + +tempest.api.network.test_routers.RoutersIpV6Test.test_add_multiple_router_interfaces + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create 2 networks named network01 and network02 sequentially, + storing the "id" parameters returned in the response +* Test action 2: Create an IPv6 subnet01 in network01, an IPv6 subnet02 in network02 sequentially, + storing the "id" parameters returned in the response +* Test action 3: Create a router, storing the "id" parameter returned in the response +* Test action 4: Create interface01 with subnet01 and the router +* Test action 5: Verify the router_id stored in test action 3 equals to the interface01's 'device_id' + and subnet01_id stored in test action 2 equals to the interface01's 'subnet_id' +* **Test assertion 1:** the router_id equals to the interface01's 'device_id' + and subnet01_id equals to the interface01's 'subnet_id' +* Test action 5: Create interface02 with subnet02 and the router +* Test action 6: Verify the router_id stored in test action 3 equals to the interface02's 'device_id' + and subnet02_id stored in test action 2 equals to the interface02's 'subnet_id' +* **Test assertion 2:** the router_id equals to the interface02's 'device_id' + and subnet02_id equals to the interface02's 'subnet_id' +* Test action 7: Delete the interfaces, router, IPv6 subnets and networks, networks, subnets, then list + all interfaces, ports, IPv6 subnets, networks, the test passes if the deleted ones + are not found in the list. +* **Test assertion 3:** The interfaces, router, IPv6 subnets and networks ids are not present in the lists + after deleting + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to use bulk create commands to create networks, IPv6 subnets and ports on +the SUT API. Specifically it verifies that: + +* Interface create commands to create interface with IPv6 subnet and router, interface 'device_id' and + 'subnet_id' should equal to the router id and IPv6 subnet id, respectively. +* Interface create commands to create multiple interface with the same router and multiple IPv6 subnets. +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +------------------------------------------------------------------- +Test Case 12 - Add and Remove an IPv6 Router Interface with port_id +------------------------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.router_interface_add_remove_with_port + +Use case specification +---------------------- + +This test case evaluates the SUT abiltiy of adding, removing router interface to +a port, the subnet_id and port_id of the interface will be checked, +the port's device_id will be checked if equals to the router_id or not. The +reference is, + +tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_port_id + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create a network, storing the "id" parameter returned in the response +* Test action 2: Create an IPv6 subnet of the network, storing the "id" parameter returned in the response +* Test action 3: Create a router, storing the "id" parameter returned in the response +* Test action 4: Create a port of the network, storing the "id" parameter returned in the response +* Test action 5: Add router interface to the port created, storing the "id" parameter returned in the response +* Test action 6: Verify the interface's keys include 'subnet_id' and 'port_id' +* **Test assertion 1:** the interface's keys include 'subnet_id' and 'port_id' +* Test action 7: Show the port details, verify the 'device_id' in port details equals to the router id stored + in test action 3 +* **Test assertion 2:** 'device_id' in port details equals to the router id +* Test action 8: Delete the interface, port, router, subnet and network, then list + all interfaces, ports, routers, subnets and networks, the test passes if the deleted + ones are not found in the list. +* **Test assertion 3:** interfaces, ports, routers, subnets and networks are not found in the lists after deleting + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to use add/remove commands to add/remove router interface to the port, +show commands to show port details on the SUT API. Specifically it verifies that: + +* Router_interface add commands to add router interface to a port, the interface's keys should include 'subnet_id' and 'port_id' +* Port show commands to show 'device_id' in port details, which should be equal to the router id +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +--------------------------------------------------------------------- +Test Case 13 - Add and Remove an IPv6 Router Interface with subnet_id +--------------------------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.router_interface_add_remove + +Use case specification +---------------------- + +This test case evaluates the SUT API ability of adding and removing a router interface with +the IPv6 subnet id, the reference is + +tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_subnet_id + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create a network, storing the "id" parameter returned in the response +* Test action 2: Create an IPv6 subnet with the network created, storing the "id" parameter + returned in the response +* Test action 3: Create a router, storing the "id" parameter returned in the response +* Test action 4: Add a router interface with the stored ids of the router and IPv6 subnet +* **Test assertion 1:** Key 'subnet_id' is included in the added interface's keys +* **Test assertion 2:** Key 'port_id' is included in the added interface's keys +* Test action 5: Show the port info with the stored interface's port id +* **Test assertion 3:**: The stored router id is equal to the device id shown in the port info +* Test action 6: Delete the router interface created in test action 4, using the stored subnet id +* Test action 7: List all router interfaces, verifying the router interface is no longer present +* **Test assertion 4:** The router interface with the stored subnet id is not present + in the router interface list +* Test action 8: Delete the router created in test action 3, using the stored router id +* Test action 9: List all routers, verifying the router id is no longer present +* **Test assertion 5:** The router "id" parameter is not present in the router list +* Test action 10: Delete the subnet created in test action 2, using the stored subnet id +* Test action 11: List all subnets, verifying the subnet id is no longer present +* **Test assertion 6:** The subnet "id" parameter is not present in the subnet list +* Test action 12: Delete the network created in test action 1, using the stored network id +* Test action 13: List all networks, verifying the network id is no longer present +* **Test assertion 7:** The network "id" parameter is not present in the network list + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to add and remove router interface with the subnet id on the +SUT API. Specifically it verifies that: + +* Router interface add command returns valid 'subnet_id' parameter which is reported + in the interface's keys +* Router interface add command returns valid 'port_id' parameter which is reported + in the interface's keys +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +------------------------------------------------------------------- +Test Case 14 - Create, Show, List, Update and Delete an IPv6 router +------------------------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.router_create_show_list_update_delete + +Use case specification +---------------------- + +This test case evaluates the SUT API ability of creating, showing, listing, updating +and deleting routers, the reference is + +tempest.api.network.test_routers.RoutersIpV6Test.test_create_show_list_update_delete_router + +Test preconditions +------------------ + +There should exist an OpenStack external network. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create a router, set the admin_state_up to be False and external_network_id + to be public network id, storing the "id" parameter returned in the response +* **Test assertion 1:** The created router's admin_state_up is False +* **Test assertion 2:** The created router's external network id equals to the public network id +* Test action 2: Show details of the router created in test action 1, using the stored router id +* **Test assertion 3:** The router's name shown is the same as the router created +* **Test assertion 4:** The router's external network id shown is the same as the public network id +* Test action 3: List all routers and verify if created router is in response message +* **Test assertion 5:** The stored router id is in the router list +* Test action 4: Update the name of router and verify if it is updated +* **Test assertion 6:** The name of router equals to the name used to update in test action 4 +* Test action 5: Show the details of router, using the stored router id +* **Test assertion 7:** The router's name shown equals to the name used to update in test action 4 +* Test action 6: Delete the router created in test action 1, using the stored router id +* Test action 7: List all routers, verifying the router id is no longer present +* **Test assertion 8:** The "id" parameter is not present in the router list + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to create, show, list, update and delete router on +the SUT API. Specifically it verifies that: + +* Router create command returns valid "admin_state_up" and "id" parameters which equal to the + "admin_state_up" and "id" returned in the response +* Router show command returns valid "name" parameter which equals to the "name" returned in the response +* Router show command returns valid "external network id" parameters which equals to the public network id +* Router list command returns valid "id" parameter which equals to the stored router "id" +* Router update command returns updated "name" parameters which equals to the "name" used to update +* Router created using create command is able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +--------------------------------------------------------------------------- +Test Case 15 - Create, List, Update, Show and Delete an IPv6 security group +--------------------------------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.security_group_create_list_update_show_delete + +Use case specification +---------------------- + +This test case evaluates the SUT API ability of creating, listing, updating, showing +and deleting security groups, the reference is + +tempest.api.network.test_security_groups.SecGroupIPv6Test.test_create_list_update_show_delete_security_group + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create a security group, storing the "id" parameter returned in the response +* Test action 2: List all security groups and verify if created security group is there in response +* **Test assertion 1:** The created security group's "id" is found in the list +* Test action 3: Update the name and description of this security group, using the stored id +* Test action 4: Verify if the security group's name and description are updated +* **Test assertion 2:** The security group's name equals to the name used in test action 3 +* **Test assertion 3:** The security group's description equals to the description used in test action 3 +* Test action 5: Show details of the updated security group, using the stored id +* **Test assertion 4:** The security group's name shown equals to the name used in test action 3 +* **Test assertion 5:** The security group's description shown equals to the description used in test action 3 +* Test action 6: Delete the security group created in test action 1, using the stored id +* Test action 7: List all security groups, verifying the security group's id is no longer present +* **Test assertion 6:** The "id" parameter is not present in the security group list + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to create list, update, show and delete security group on +the SUT API. Specifically it verifies that: + +* Security group create commands return valid "id" parameter which is reported in the list commands +* Security group update commands return valid "name" and "description" parameters which are + reported in the show commands +* Security group created using create command is able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +--------------------------------------------------------------- +Test Case 16 - Create, Show and Delete IPv6 security group rule +--------------------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.security_group_rule_create_show_delete + +Use case specification +---------------------- + +This test case evaluates the SUT API ability of creating, showing, listing and deleting +security group rules, the reference is + +tempest.api.network.test_security_groups.SecGroupIPv6Test.test_create_show_delete_security_group_rule + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create a security group, storing the "id" parameter returned in the response +* Test action 2: Create a rule of the security group with protocol tcp, udp and icmp, respectively, + using the stored security group's id, storing the "id" parameter returned in the response +* Test action 3: Show details of the created security group rule, using the stored id of the + security group rule +* **Test assertion 1:** All the created security group rule's values equal to the rule values + shown in test action 3 +* Test action 4: List all security group rules +* **Test assertion 2:** The stored security group rule's id is found in the list +* Test action 5: Delete the security group rule, using the stored security group rule's id +* Test action 6: List all security group rules, verifying the security group rule's id is no longer present +* **Test assertion 3:** The security group rule "id" parameter is not present in the list +* Test action 7: Delete the security group, using the stored security group's id +* Test action 8: List all security groups, verifying the security group's id is no longer present +* **Test assertion 4:** The security group "id" parameter is not present in the list + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to create, show, list and delete security group rules on +the SUT API. Specifically it verifies that: + +* Security group rule create command returns valid values which are reported in the show command +* Security group rule created using create command is able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +---------------------------------------- +Test Case 17 - List IPv6 Security Groups +---------------------------------------- + +Short name +---------- + +opnfv.ipv6.security_group_list + +Use case specification +---------------------- + +This test case evaluates the SUT API ability of listing security groups, the reference is + +tempest.api.network.test_security_groups.SecGroupIPv6Test.test_list_security_groups + +Test preconditions +------------------ + +There should exist a default security group. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: List all security groups +* Test action 2: Verify the default security group exists in the list, the test passes + if the default security group exists +* **Test assertion 1:** The default security group is in the list + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to list security groups on the SUT API. +Specifically it verifies that: + +* Security group list command return valid security groups which include the default security group + +Post conditions +--------------- + +None + +---------------------------------------------------------------------------- +Test Case 18 - IPv6 Address Assignment - Dual Stack, SLAAC, DHCPv6 Stateless +---------------------------------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.dhcpv6_stateless + +Use case specification +---------------------- + +This test case evaluates IPv6 address assignment in ipv6_ra_mode 'dhcpv6_stateless' +and ipv6_address_mode 'dhcpv6_stateless'. +In this case, guest instance obtains IPv6 address from OpenStack managed radvd +using SLAAC and optional info from dnsmasq using DHCPv6 stateless. This test case then +verifies the ping6 available VM can ping the other VM's v4 and v6 addresses +as well as the v6 subnet's gateway ip in the same network, the reference is + +tempest.scenario.test_network_v6.TestGettingAddress.test_dhcp6_stateless_from_os + +Test preconditions +------------------ + +There should exist a public router or a public network. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create one network, storing the "id" parameter returned in the response +* Test action 2: Create one IPv4 subnet of the created network, storing the "id" + parameter returned in the response +* Test action 3: If there exists a public router, use it as the router. Otherwise, + use the public network to create a router +* Test action 4: Connect the IPv4 subnet to the router, using the stored IPv4 subnet id +* Test action 5: Create one IPv6 subnet of the network created in test action 1 in + ipv6_ra_mode 'dhcpv6_stateless' and ipv6_address_mode 'dhcpv6_stateless', + storing the "id" parameter returned in the response +* Test action 6: Connect the IPv6 subnet to the router, using the stored IPv6 subnet id +* Test action 7: Boot two VMs on this network, storing the "id" parameters returned in the response +* **Test assertion 1:** The vNIC of each VM gets one v4 address and one v6 address actually assigned +* **Test assertion 2:** Each VM can ping the other's v4 private address +* **Test assertion 3:** The ping6 available VM can ping the other's v6 address + as well as the v6 subnet's gateway ip +* Test action 8: Delete the 2 VMs created in test action 7, using the stored ids +* Test action 9: List all VMs, verifying the ids are no longer present +* **Test assertion 4:** The two "id" parameters are not present in the VM list +* Test action 10: Delete the IPv4 subnet created in test action 2, using the stored id +* Test action 11: Delete the IPv6 subnet created in test action 5, using the stored id +* Test action 12: List all subnets, verifying the ids are no longer present +* **Test assertion 5:** The "id" parameters of IPv4 and IPv6 are not present in the list +* Test action 13: Delete the network created in test action 1, using the stored id +* Test action 14: List all networks, verifying the id is no longer present +* **Test assertion 6:** The "id" parameter is not present in the network list + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to assign IPv6 addresses in ipv6_ra_mode +'dhcpv6_stateless' and ipv6_address_mode 'dhcpv6_stateless', +and verify the ping6 available VM can ping the other VM's v4 and v6 addresses as well as +the v6 subnet's gateway ip in the same network. Specifically it verifies that: + +* The IPv6 addresses in mode 'dhcpv6_stateless' assigned successfully +* The VM can ping the other VM's IPv4 and IPv6 private addresses as well as the v6 subnet's gateway ip +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +-------------------------------------------------------------------------------------- +Test Case 19 - IPv6 Address Assignment - Dual Net, Dual Stack, SLAAC, DHCPv6 Stateless +-------------------------------------------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.dualnet_dhcpv6_stateless + +Use case specification +---------------------- + +This test case evaluates IPv6 address assignment in ipv6_ra_mode 'dhcpv6_stateless' +and ipv6_address_mode 'dhcpv6_stateless'. +In this case, guest instance obtains IPv6 address from OpenStack managed radvd +using SLAAC and optional info from dnsmasq using DHCPv6 stateless. This test case then +verifies the ping6 available VM can ping the other VM's v4 address in one network +and v6 address in another network as well as the v6 subnet's gateway ip, the reference is + +tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_dhcp6_stateless_from_os + +Test preconditions +------------------ + +There should exists a public router or a public network. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create one network, storing the "id" parameter returned in the response +* Test action 2: Create one IPv4 subnet of the created network, storing the "id" + parameter returned in the response +* Test action 3: If there exists a public router, use it as the router. Otherwise, + use the public network to create a router +* Test action 4: Connect the IPv4 subnet to the router, using the stored IPv4 subnet id +* Test action 5: Create another network, storing the "id" parameter returned in the response +* Test action 6: Create one IPv6 subnet of network created in test action 5 in + ipv6_ra_mode 'dhcpv6_stateless' and ipv6_address_mode 'dhcpv6_stateless', + storing the "id" parameter returned in the response +* Test action 7: Connect the IPv6 subnet to the router, using the stored IPv6 subnet id +* Test action 8: Boot two VMs on these two networks, storing the "id" parameters returned in the response +* Test action 9: Turn on 2nd NIC of each VM for the network created in test action 5 +* **Test assertion 1:** The 1st vNIC of each VM gets one v4 address assigned and + the 2nd vNIC of each VM gets one v6 address actually assigned +* **Test assertion 2:** Each VM can ping the other's v4 private address +* **Test assertion 3:** The ping6 available VM can ping the other's v6 address + as well as the v6 subnet's gateway ip +* Test action 10: Delete the 2 VMs created in test action 8, using the stored ids +* Test action 11: List all VMs, verifying the ids are no longer present +* **Test assertion 4:** The two "id" parameters are not present in the VM list +* Test action 12: Delete the IPv4 subnet created in test action 2, using the stored id +* Test action 13: Delete the IPv6 subnet created in test action 6, using the stored id +* Test action 14: List all subnets, verifying the ids are no longer present +* **Test assertion 5:** The "id" parameters of IPv4 and IPv6 are not present in the list +* Test action 15: Delete the 2 networks created in test action 1 and 5, using the stored ids +* Test action 16: List all networks, verifying the ids are no longer present +* **Test assertion 6:** The two "id" parameters are not present in the network list + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to assign IPv6 addresses in ipv6_ra_mode 'dhcpv6_stateless' +and ipv6_address_mode 'dhcpv6_stateless', and verify the ping6 available VM can ping +the other VM's v4 address in one network and v6 address in another network as well as +the v6 subnet's gateway ip. Specifically it verifies that: + +* The IPv6 addresses in mode 'dhcpv6_stateless' assigned successfully +* The VM can ping the other VM's IPv4 address in one network and IPv6 address in another + network as well as the v6 subnet's gateway ip +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +----------------------------------------------------------------------------------------------- +Test Case 20 - IPv6 Address Assignment - Multiple Prefixes, Dual Stack, SLAAC, DHCPv6 Stateless +----------------------------------------------------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.multiple_prefixes_dhcpv6_stateless + +Use case specification +---------------------- + +This test case evaluates IPv6 address assignment in ipv6_ra_mode 'dhcpv6_stateless' +and ipv6_address_mode 'dhcpv6_stateless'. +In this case, guest instance obtains IPv6 addresses from OpenStack managed radvd +using SLAAC and optional info from dnsmasq using DHCPv6 stateless. This test case then +verifies the ping6 available VM can ping the other VM's one v4 address and two v6 +addresses with different prefixes as well as the v6 subnets' gateway ips in the +same network, the reference is + +tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_dhcpv6_stateless + +Test preconditions +------------------ + +There should exist a public router or a public network. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create one network, storing the "id" parameter returned in the response +* Test action 2: Create one IPv4 subnet of the created network, storing the "id" + parameter returned in the response +* Test action 3: If there exists a public router, use it as the router. Otherwise, + use the public network to create a router +* Test action 4: Connect the IPv4 subnet to the router, using the stored IPv4 subnet id +* Test action 5: Create two IPv6 subnets of the network created in test action 1 in + ipv6_ra_mode 'dhcpv6_stateless' and ipv6_address_mode 'dhcpv6_stateless', + storing the "id" parameters returned in the response +* Test action 6: Connect the two IPv6 subnets to the router, using the stored IPv6 subnet ids +* Test action 7: Boot two VMs on this network, storing the "id" parameters returned in the response +* **Test assertion 1:** The vNIC of each VM gets one v4 address and two v6 addresses with + different prefixes actually assigned +* **Test assertion 2:** Each VM can ping the other's v4 private address +* **Test assertion 3:** The ping6 available VM can ping the other's v6 addresses + as well as the v6 subnets' gateway ips +* Test action 8: Delete the 2 VMs created in test action 7, using the stored ids +* Test action 9: List all VMs, verifying the ids are no longer present +* **Test assertion 4:** The two "id" parameters are not present in the VM list +* Test action 10: Delete the IPv4 subnet created in test action 2, using the stored id +* Test action 11: Delete two IPv6 subnets created in test action 5, using the stored ids +* Test action 12: List all subnets, verifying the ids are no longer present +* **Test assertion 5:** The "id" parameters of IPv4 and IPv6 are not present in the list +* Test action 13: Delete the network created in test action 1, using the stored id +* Test action 14: List all networks, verifying the id is no longer present +* **Test assertion 6:** The "id" parameter is not present in the network list + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to assign IPv6 addresses in ipv6_ra_mode 'dhcpv6_stateless' +and ipv6_address_mode 'dhcpv6_stateless', +and verify the ping6 available VM can ping the other VM's v4 address and two +v6 addresses with different prefixes as well as the v6 subnets' gateway ips in the same network. +Specifically it verifies that: + +* The different prefixes IPv6 addresses in mode 'dhcpv6_stateless' assigned successfully +* The VM can ping the other VM's IPv4 and IPv6 private addresses as well as the v6 subnets' gateway ips +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +--------------------------------------------------------------------------------------------------------- +Test Case 21 - IPv6 Address Assignment - Dual Net, Multiple Prefixes, Dual Stack, SLAAC, DHCPv6 Stateless +--------------------------------------------------------------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.dualnet_multiple_prefixes_dhcpv6_stateless + +Use case specification +---------------------- + +This test case evaluates IPv6 address assignment in ipv6_ra_mode 'dhcpv6_stateless' +and ipv6_address_mode 'dhcpv6_stateless'. +In this case, guest instance obtains IPv6 addresses from OpenStack managed radvd +using SLAAC and optional info from dnsmasq using DHCPv6 stateless. This test case then +verifies the ping6 available VM can ping the other VM's v4 address in one network +and two v6 addresses with different prefixes in another network as well as the +v6 subnets' gateway ips, the reference is + +tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_dhcpv6_stateless + +Test preconditions +------------------ + +There should exist a public router or a public network. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create one network, storing the "id" parameter returned in the response +* Test action 2: Create one IPv4 subnet of the created network, storing the "id" + parameter returned in the response +* Test action 3: If there exists a public router, use it as the router. Otherwise, + use the public network to create a router +* Test action 4: Connect the IPv4 subnet to the router, using the stored IPv4 subnet id +* Test action 5: Create another network, storing the "id" parameter returned in the response +* Test action 6: Create two IPv6 subnets of network created in test action 5 in + ipv6_ra_mode 'dhcpv6_stateless' and ipv6_address_mode 'dhcpv6_stateless', + storing the "id" parameters returned in the response +* Test action 7: Connect the two IPv6 subnets to the router, using the stored IPv6 subnet ids +* Test action 8: Boot two VMs on these two networks, storing the "id" parameters returned in the response +* Test action 9: Turn on 2nd NIC of each VM for the network created in test action 5 +* **Test assertion 1:** The vNIC of each VM gets one v4 address and two v6 addresses + with different prefixes actually assigned +* **Test assertion 2:** Each VM can ping the other's v4 private address +* **Test assertion 3:** The ping6 available VM can ping the other's v6 addresses + as well as the v6 subnets' gateway ips +* Test action 10: Delete the 2 VMs created in test action 8, using the stored ids +* Test action 11: List all VMs, verifying the ids are no longer present +* **Test assertion 4:** The two "id" parameters are not present in the VM list +* Test action 12: Delete the IPv4 subnet created in test action 2, using the stored id +* Test action 13: Delete two IPv6 subnets created in test action 6, using the stored ids +* Test action 14: List all subnets, verifying the ids are no longer present +* **Test assertion 5:** The "id" parameters of IPv4 and IPv6 are not present in the list +* Test action 15: Delete the 2 networks created in test action 1 and 5, using the stored ids +* Test action 16: List all networks, verifying the ids are no longer present +* **Test assertion 6:** The two "id" parameters are not present in the network list + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to assign IPv6 addresses in ipv6_ra_mode 'dhcpv6_stateless' +and ipv6_address_mode 'dhcpv6_stateless', +and verify the ping6 available VM can ping the other VM's v4 address in one network and two +v6 addresses with different prefixes in another network as well as the v6 subnets' +gateway ips. Specifically it verifies that: + +* The IPv6 addresses in mode 'dhcpv6_stateless' assigned successfully +* The VM can ping the other VM's IPv4 and IPv6 private addresses as well as the v6 subnets' gateway ips +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +---------------------------------------------------------- +Test Case 22 - IPv6 Address Assignment - Dual Stack, SLAAC +---------------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.slaac + +Use case specification +---------------------- + +This test case evaluates IPv6 address assignment in ipv6_ra_mode 'slaac' and +ipv6_address_mode 'slaac'. +In this case, guest instance obtains IPv6 address from OpenStack managed radvd +using SLAAC. This test case then verifies the ping6 available VM can ping the other +VM's v4 and v6 addresses as well as the v6 subnet's gateway ip in the +same network, the reference is + +tempest.scenario.test_network_v6.TestGettingAddress.test_slaac_from_os + +Test preconditions +------------------ + +There should exist a public router or a public network. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create one network, storing the "id" parameter returned in the response +* Test action 2: Create one IPv4 subnet of the created network, storing the "id" + parameter returned in the response +* Test action 3: If there exists a public router, use it as the router. Otherwise, + use the public network to create a router +* Test action 4: Connect the IPv4 subnet to the router, using the stored IPv4 subnet id +* Test action 5: Create one IPv6 subnet of the network created in test action 1 in + ipv6_ra_mode 'slaac' and ipv6_address_mode 'slaac', storing the "id" parameter returned in the response +* Test action 6: Connect the IPv6 subnet to the router, using the stored IPv6 subnet id +* Test action 7: Boot two VMs on this network, storing the "id" parameters returned in the response +* **Test assertion 1:** The vNIC of each VM gets one v4 address and one v6 address actually assigned +* **Test assertion 2:** Each VM can ping the other's v4 private address +* **Test assertion 3:** The ping6 available VM can ping the other's v6 address + as well as the v6 subnet's gateway ip +* Test action 8: Delete the 2 VMs created in test action 7, using the stored ids +* Test action 9: List all VMs, verifying the ids are no longer present +* **Test assertion 4:** The two "id" parameters are not present in the VM list +* Test action 10: Delete the IPv4 subnet created in test action 2, using the stored id +* Test action 11: Delete the IPv6 subnet created in test action 5, using the stored id +* Test action 12: List all subnets, verifying the ids are no longer present +* **Test assertion 5:** The "id" parameters of IPv4 and IPv6 are not present in the list +* Test action 13: Delete the network created in test action 1, using the stored id +* Test action 14: List all networks, verifying the id is no longer present +* **Test assertion 6:** The "id" parameter is not present in the network list + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to assign IPv6 addresses in ipv6_ra_mode 'slaac' +and ipv6_address_mode 'slaac', +and verify the ping6 available VM can ping the other VM's v4 and v6 addresses as well as +the v6 subnet's gateway ip in the same network. Specifically it verifies that: + +* The IPv6 addresses in mode 'slaac' assigned successfully +* The VM can ping the other VM's IPv4 and IPv6 private addresses as well as the v6 subnet's gateway ip +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +-------------------------------------------------------------------- +Test Case 23 - IPv6 Address Assignment - Dual Net, Dual Stack, SLAAC +-------------------------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.dualnet_slaac + +Use case specification +---------------------- + +This test case evaluates IPv6 address assignment in ipv6_ra_mode 'slaac' and +ipv6_address_mode 'slaac'. +In this case, guest instance obtains IPv6 address from OpenStack managed radvd +using SLAAC. This test case then verifies the ping6 available VM can ping the other +VM's v4 address in one network and v6 address in another network as well as the +v6 subnet's gateway ip, the reference is + +tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_slaac_from_os + +Test preconditions +------------------ + +There should exist a public router or a public network. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create one network, storing the "id" parameter returned in the response +* Test action 2: Create one IPv4 subnet of the created network, storing the "id" + parameter returned in the response +* Test action 3: If there exists a public router, use it as the router. Otherwise, + use the public network to create a router +* Test action 4: Connect the IPv4 subnet to the router, using the stored IPv4 subnet id +* Test action 5: Create another network, storing the "id" parameter returned in the response +* Test action 6: Create one IPv6 subnet of network created in test action 5 in + ipv6_ra_mode 'slaac' and ipv6_address_mode 'slaac', storing the "id" parameter returned in the response +* Test action 7: Connect the IPv6 subnet to the router, using the stored IPv6 subnet id +* Test action 8: Boot two VMs on these two networks, storing the "id" parameters returned in the response +* Test action 9: Turn on 2nd NIC of each VM for the network created in test action 5 +* **Test assertion 1:** The 1st vNIC of each VM gets one v4 address assigned and + the 2nd vNIC of each VM gets one v6 address actually assigned +* **Test assertion 2:** Each VM can ping the other's v4 private address +* **Test assertion 3:** The ping6 available VM can ping the other's v6 address + as well as the v6 subnet's gateway ip +* Test action 10: Delete the 2 VMs created in test action 8, using the stored ids +* Test action 11: List all VMs, verifying the ids are no longer present +* **Test assertion 4:** The two "id" parameters are not present in the VM list +* Test action 12: Delete the IPv4 subnet created in test action 2, using the stored id +* Test action 13: Delete the IPv6 subnet created in test action 6, using the stored id +* Test action 14: List all subnets, verifying the ids are no longer present +* **Test assertion 5:** The "id" parameters of IPv4 and IPv6 are not present in the list +* Test action 15: Delete the 2 networks created in test action 1 and 5, using the stored ids +* Test action 16: List all networks, verifying the ids are no longer present +* **Test assertion 6:** The two "id" parameters are not present in the network list + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to assign IPv6 addresses in ipv6_ra_mode 'slaac' +and ipv6_address_mode 'slaac', +and verify the ping6 available VM can ping the other VM's v4 address in one network and +v6 address in another network as well as the v6 subnet's gateway ip. Specifically it verifies that: + +* The IPv6 addresses in mode 'slaac' assigned successfully +* The VM can ping the other VM's IPv4 address in one network and IPv6 address + in another network as well as the v6 subnet's gateway ip +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +----------------------------------------------------------------------------- +Test Case 24 - IPv6 Address Assignment - Multiple Prefixes, Dual Stack, SLAAC +----------------------------------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.multiple_prefixes_slaac + +Use case specification +---------------------- + +This test case evaluates IPv6 address assignment in ipv6_ra_mode 'slaac' and +ipv6_address_mode 'slaac'. +In this case, guest instance obtains IPv6 addresses from OpenStack managed radvd +using SLAAC. This test case then verifies the ping6 available VM can ping the other +VM's one v4 address and two v6 addresses with different prefixes as well as the v6 +subnets' gateway ips in the same network, the reference is + +tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_slaac + +Test preconditions +------------------ + +There should exists a public router or a public network. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create one network, storing the "id" parameter returned in the response +* Test action 2: Create one IPv4 subnet of the created network, storing the "id + parameter returned in the response +* Test action 3: If there exists a public router, use it as the router. Otherwise, + use the public network to create a router +* Test action 4: Connect the IPv4 subnet to the router, using the stored IPv4 subnet id +* Test action 5: Create two IPv6 subnets of the network created in test action 1 in + ipv6_ra_mode 'slaac' and ipv6_address_mode 'slaac', storing the "id" parameters returned in the response +* Test action 6: Connect the two IPv6 subnets to the router, using the stored IPv6 subnet ids +* Test action 7: Boot two VMs on this network, storing the "id" parameters returned in the response +* **Test assertion 1:** The vNIC of each VM gets one v4 address and two v6 addresses with + different prefixes actually assigned +* **Test assertion 2:** Each VM can ping the other's v4 private address +* **Test assertion 3:** The ping6 available VM can ping the other's v6 addresses + as well as the v6 subnets' gateway ips +* Test action 8: Delete the 2 VMs created in test action 7, using the stored ids +* Test action 9: List all VMs, verifying the ids are no longer present +* **Test assertion 4:** The two "id" parameters are not present in the VM list +* Test action 10: Delete the IPv4 subnet created in test action 2, using the stored id +* Test action 11: Delete two IPv6 subnets created in test action 5, using the stored ids +* Test action 12: List all subnets, verifying the ids are no longer present +* **Test assertion 5:** The "id" parameters of IPv4 and IPv6 are not present in the list +* Test action 13: Delete the network created in test action 1, using the stored id +* Test action 14: List all networks, verifying the id is no longer present +* **Test assertion 6:** The "id" parameter is not present in the network list + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to assign IPv6 addresses in ipv6_ra_mode 'slaac' +and ipv6_address_mode 'slaac', +and verify the ping6 available VM can ping the other VM's v4 address and two +v6 addresses with different prefixes as well as the v6 subnets' gateway ips in the same network. +Specifically it verifies that: + +* The different prefixes IPv6 addresses in mode 'slaac' assigned successfully +* The VM can ping the other VM's IPv4 and IPv6 private addresses as well as the v6 subnets' gateway ips +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + +--------------------------------------------------------------------------------------- +Test Case 25 - IPv6 Address Assignment - Dual Net, Dual Stack, Multiple Prefixes, SLAAC +--------------------------------------------------------------------------------------- + +Short name +---------- + +opnfv.ipv6.dualnet_multiple_prefixes_slaac + +Use case specification +---------------------- + +This test case evaluates IPv6 address assignment in ipv6_ra_mode 'slaac' and +ipv6_address_mode 'slaac'. +In this case, guest instance obtains IPv6 addresses from OpenStack managed radvd +using SLAAC. This test case then verifies the ping6 available VM can ping the other +VM's v4 address in one network and two v6 addresses with different prefixes in another +network as well as the v6 subnets' gateway ips, the reference is + +tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_slaac + +Test preconditions +------------------ + +There should exist a public router or a public network. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +''''''''''''''' + +* Test action 1: Create one network, storing the "id" parameter returned in the response +* Test action 2: Create one IPv4 subnet of the created network, storing the "id" + parameter returned in the response +* Test action 3: If there exists a public router, use it as the router. Otherwise, + use the public network to create a router +* Test action 4: Connect the IPv4 subnet to the router, using the stored IPv4 subnet id +* Test action 5: Create another network, storing the "id" parameter returned in the response +* Test action 6: Create two IPv6 subnets of network created in test action 5 in + ipv6_ra_mode 'slaac' and ipv6_address_mode 'slaac', storing the "id" parameters returned in the response +* Test action 7: Connect the two IPv6 subnets to the router, using the stored IPv6 subnet ids +* Test action 8: Boot two VMs on these two networks, storing the "id" parameters returned in the response +* Test action 9: Turn on 2nd NIC of each VM for the network created in test action 5 +* **Test assertion 1:** The vNIC of each VM gets one v4 address and two v6 addresses + with different prefixes actually assigned +* **Test assertion 2:** Each VM can ping the other's v4 private address +* **Test assertion 3:** The ping6 available VM can ping the other's v6 addresses + as well as the v6 subnets' gateway ips +* Test action 10: Delete the 2 VMs created in test action 8, using the stored ids +* Test action 11: List all VMs, verifying the ids are no longer present +* **Test assertion 4:** The two "id" parameters are not present in the VM list +* Test action 12: Delete the IPv4 subnet created in test action 2, using the stored id +* Test action 13: Delete two IPv6 subnets created in test action 6, using the stored ids +* Test action 14: List all subnets, verifying the ids are no longer present +* **Test assertion 5:** The "id" parameters of IPv4 and IPv6 are not present in the list +* Test action 15: Delete the 2 networks created in test action 1 and 5, using the stored ids +* Test action 16: List all networks, verifying the ids are no longer present +* **Test assertion 6:** The two "id" parameters are not present in the network list + +Pass / fail criteria +''''''''''''''''''''' + +This test evaluates the ability to assign IPv6 addresses in ipv6_ra_mode 'slaac' +and ipv6_address_mode 'slaac', +and verify the ping6 available VM can ping the other VM's v4 address in one network and two +v6 addresses with different prefixes in another network as well as the v6 subnets' gateway ips. +Specifically it verifies that: + +* The IPv6 addresses in mode 'slaac' assigned successfully +* The VM can ping the other VM's IPv4 and IPv6 private addresses as well as the v6 subnets' gateway ips +* All items created using create commands are able to be removed using the returned identifiers + +Post conditions +--------------- + +None + + + diff --git a/docs/testing/user/testspecification/machinelifecycle/index.rst b/docs/testing/user/testspecification/machinelifecycle/index.rst new file mode 100644 index 00000000..598812f5 --- /dev/null +++ b/docs/testing/user/testspecification/machinelifecycle/index.rst @@ -0,0 +1,757 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Huawei Technologies Co.,Ltd + +=========================================================== +Common virtual machine life cycle events test specification +=========================================================== + +.. toctree:: + :maxdepth: 2 + +Scope +===== + +The common virtual machine life cycle events test area evaluates the ability of +the system under test to behave correctly after common virtual machine life +cycle events. The tests in this test area will evaluate: + +- Stop/Start a server +- Reboot a server +- Rebuild a server +- Pause/Unpause a server +- Suspend/Resume a server +- Resize a server +- Resizing a volume-backed server +- Sequence suspend resume +- Shelve/Unshelve a server +- Cold migrate a server +- Live migrate a server + +References +========== + +- iSCSI + + - https://docs.openstack.org/liberty/config-reference/content/config-iscsi-storage.html + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test area + +- API - Application Programming Interface +- NFVi - Network Functions Virtualization infrastructure +- VIM - Virtual Infrastructure Manager +- VM - Virtual Machine + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + +Test Area Structure +=================== + +The test area is structured based on common virtual machine life cycle events. +Each test case is able to run independently, i.e. irrelevant of the state +created by a previous test. Specifically, every test performs clean-up +operations which return the system to the same state as before the test. + +Test Descriptions +================= + +API Used and Reference +---------------------- + +Block storage: https://developer.openstack.org/api-ref/block-storage + +- create volume +- delete volume +- attach volume to server +- detach volume from server + +Security Groups: https://developer.openstack.org/api-ref/network/v2/index.html#security-groups-security-groups + +- create security group +- delete security group + +Networks: https://developer.openstack.org/api-ref/networking/v2/index.html#networks + +- create network +- delete network + +Routers and interface: https://developer.openstack.org/api-ref/networking/v2/index.html#routers-routers + +- create router +- delete router +- add interface to router + +Subnets: https://developer.openstack.org/api-ref/networking/v2/index.html#subnets + +- create subnet +- delete subnet + +Servers: https://developer.openstack.org/api-ref/compute/ + +- create keypair +- create server +- show server +- delete server +- add/assign floating IP +- resize server +- revert resized server +- confirm resized server +- pause server +- unpause server +- start server +- stop server +- reboot server +- rebuild server +- suspend server +- resume suspended server +- shelve server +- unshelve server +- migrate server +- live-migrate server + +Ports: https://developer.openstack.org/api-ref/networking/v2/index.html#ports + +- create port +- delete port + +Floating IPs: https://developer.openstack.org/api-ref/networking/v2/index.html#floating-ips-floatingips + +- create floating IP +- delete floating IP + +Availability zone: https://developer.openstack.org/api-ref/compute/ + +- get availability zone + +------------------------------------ +Test Case 1 - Minimum basic scenario +------------------------------------ + +Test case specification +----------------------- + +tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario + +Test preconditions +------------------ + +* Nova, cinder, glance, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create an image IMG1 +* Test action 2: Create a keypair KEYP1 +* Test action 3: Create a server VM1 with IMG1 and KEYP1 +* **Test assertion 1:** Verify VM1 is created successfully +* Test action 4: Create a volume VOL1 +* **Test assertion 2:** Verify VOL1 is created successfully +* Test action 5: Attach VOL1 to VM1 +* **Test assertion 3:** Verify VOL1's status has been updated after attached to VM1 +* Test action 6: Create a floating IP FIP1 and assign FIP1 to VM1 +* **Test assertion 4:** Verify VM1's addresses have been refreshed after associating FIP1 +* Test action 7: Create and add security group SG1 to VM1 +* **Test assertion 5:** Verify can SSH to VM1 via FIP1 +* Test action 8: Reboot VM1 +* **Test assertion 6:** Verify can SSH to VM1 via FIP1 +* **Test assertion 7:** Verify VM1's disk count equals to 1 +* Test action 9: Delete the floating IP FIP1 from VM1 +* **Test assertion 8:** Verify VM1's addresses have been refreshed after disassociating FIP1 +* Test action 10: Delete SG1, IMG1, KEYP1, VOL1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates a minimum basic scenario. Specifically, the test verifies that: + +* The server can be connected before reboot. + +* The server can be connected after reboot. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------- +Test Case 2 - Cold migration +---------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration + +Test preconditions +------------------ + +* At least 2 compute nodes +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Get VM1's host info SRC_HOST +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 6: Cold migrate VM1 +* Test action 7: Wait for VM1 to reach 'VERIFY_RESIZE' status +* Test action 8: Confirm resize VM1 +* Test action 9: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 2:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 10: Get VM1's host info DST_HOST +* **Test assertion 3:** Verify SRC_HOST does not equal to DST_HOST +* Test action 11: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to cold migrate VMs. Specifically, the test verifies that: + +* Servers can be cold migrated from one compute node to another computer node. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +----------------------------------- +Test Case 3 - Cold migration revert +----------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration_revert + +Test preconditions +------------------ + +* At least 2 compute nodes +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Get VM1's host info SRC_HOST +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 6: Cold migrate VM1 +* Test action 7: Wait for VM1 to reach 'VERIFY_RESIZE' status +* Test action 8: Revert resize VM1 +* Test action 9: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 2:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 10: Get VM1's host info DST_HOST +* **Test assertion 3:** Verify SRC_HOST equals to DST_HOST +* Test action 11: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to revert cold-migrated VMs. Specifically, the test verifies that: + +* Cold migrate operation can be reverted. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------- +Test Case 4 - Pause and unpause server +-------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_pause_unpause + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Pause VM1 +* Test action 5: Wait for VM1 to reach 'PAUSED' status +* **Test assertion 1:** Verify FIP1 status is 'ACTIVE' +* **Test assertion 2:** Verify ping FIP1 failed and SSH to VM1 via FIP1 failed +* Test action 6: Unpause VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 3:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 8: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to pause and unpause VMs. Specifically, the test verifies that: + +* When paused, servers cannot be reached. + +* When unpaused, servers can recover its reachability. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------- +Test Case 5 - Reboot server +--------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_reboot + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Soft reboot VM1 +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 6: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to reboot servers. Specifically, the test verifies that: + +* After reboot, servers can still be connected. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------- +Test Case 6 - Rebuild server +---------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_rebuild + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Rebuild VM1 with another image +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 6: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to rebuild servers. Specifically, the test verifies that: + +* Servers can be rebuilt with specific image correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------- +Test Case 7 - Resize server +--------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_resize + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Resize VM1 with another flavor +* Test action 5: Wait for VM1 to reach 'VERIFY_RESIZE' status +* Test action 6: Confirm resize VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 8: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to resize servers. Specifically, the test verifies that: + +* Servers can be resized with specific flavor correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +----------------------------------- +Test Case 8 - Stop and start server +----------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_stop_start + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Stop VM1 +* Test action 5: Wait for VM1 to reach 'SHUTOFF' status +* **Test assertion 1:** Verify ping FIP1 failed and SSH to VM1 via FIP1 failed +* Test action 6: Start VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 2:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 8: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to stop and start servers. Specifically, the test verifies that: + +* When stopped, servers cannot be reached. + +* When started, servers can recover its reachability. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------- +Test Case 9 - Suspend and resume server +--------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_suspend_resume + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Suspend VM1 +* Test action 5: Wait for VM1 to reach 'SUSPENDED' status +* **Test assertion 1:** Verify ping FIP1 failed and SSH to VM1 via FIP1 failed +* Test action 6: Resume VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 2:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 8: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to suspend and resume servers. Specifically, the test verifies that: + +* When suspended, servers cannot be reached. + +* When resumed, servers can recover its reachability. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------------------- +Test Case 10 - Suspend and resume server in sequence +---------------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_server_advanced_ops.TestServerAdvancedOps.test_server_sequence_suspend_resume + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a server VM1 +* Test action 2: Suspend VM1 +* Test action 3: Wait for VM1 to reach 'SUSPENDED' status +* **Test assertion 1:** Verify VM1's status is 'SUSPENDED' +* Test action 4: Resume VM1 +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 2:** Verify VM1's status is 'ACTIVE' +* Test action 6: Suspend VM1 +* Test action 7: Wait for VM1 to reach 'SUSPENDED' status +* **Test assertion 3:** Verify VM1 status is 'SUSPENDED' +* Test action 8: Resume VM1 +* Test action 9: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 4:** Verify VM1 status is 'ACTIVE' +* Test action 10: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to suspend and resume servers in sequence. +Specifically, the test verifies that: + +* Servers can be suspend and resume in sequence correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------ +Test Case 11 - Resize volume backed server +------------------------------------------ + +Test case specification +----------------------- + +tempest.scenario.test_server_advanced_ops.TestServerAdvancedOps.test_resize_volume_backed_server_confirm + +Test preconditions +------------------ + +* Nova, neutron, cinder services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a volume backed server VM1 +* Test action 2: Resize VM1 with another flavor +* Test action 3: Wait for VM1 to reach 'VERIFY_RESIZE' status +* Test action 4: Confirm resize VM1 +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** VM1's status is 'ACTIVE' +* Test action 6: Delete VM1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to resize volume backed servers. +Specifically, the test verifies that: + +* Volume backed servers can be resized with specific flavor correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +----------------------------------------- +Test Case 12 - Shelve and unshelve server +----------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_instance + +Test preconditions +------------------ + +* Nova, neutron, image services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a security group SG1, which has rules for allowing + incoming SSH and ICMP traffic +* Test action 3: Create a server with SG1 and KEYP1 +* Test action 4: Create a timestamp and store it in a file F1 inside VM1 +* Test action 5: Shelve VM1 +* Test action 6: Unshelve VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* Test action 8: Read F1 and compare if the read value and the previously written value + are the same or not +* **Test assertion 1:** Verify the values written and read are the same +* Test action 9: Delete SG1, KEYP1 and VM1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to shelve and unshelve servers. +Specifically, the test verifies that: + +* Servers can be shelved and unshelved correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------------------- +Test Case 13 - Shelve and unshelve volume backed server +------------------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_volume_backed_instance + +Test preconditions +------------------ + +* Nova, neutron, image, cinder services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a security group SG1, which has rules for allowing + incoming and outgoing SSH and ICMP traffic +* Test action 3: Create a volume backed server VM1 with SG1 and KEYP1 +* Test action 4: SSH to VM1 to create a timestamp T_STAMP1 and store it in a file F1 inside VM1 +* Test action 5: Shelve VM1 +* Test action 6: Unshelve VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* Test action 8: SSH to VM1 to read the timestamp T_STAMP2 stored in F1 +* **Test assertion 1:** Verify T_STAMP1 equals to T_STAMP2 +* Test action 9: Delete SG1, KEYP1 and VM1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to shelve and unshelve volume backed servers. +Specifically, the test verifies that: + +* Volume backed servers can be shelved and unshelved correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A diff --git a/docs/testing/user/testspecification/multiplenodes/index.rst b/docs/testing/user/testspecification/multiplenodes/index.rst new file mode 100644 index 00000000..72a131a3 --- /dev/null +++ b/docs/testing/user/testspecification/multiplenodes/index.rst @@ -0,0 +1,370 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Huawei Technologies Co.,Ltd + +=========================================================== +VM Resource Scheduling on Multiple Nodes test specification +=========================================================== + +.. toctree:: + :maxdepth: 2 + +Scope +===== + +The VM resource scheduling test area evaluates the ability of the system under +test to support VM resource scheduling on multiple nodes. +The tests in this test area will evaluate capabilities to schedule VM to multiple +compute nodes directly with scheduler hints, and create server groups with policy +affinity and anti-affinity. + +References +========== + +- Availability zone + + - https://docs.openstack.org/newton/networking-guide/config-az.html + +- Scheduling + + - https://docs.openstack.org/kilo/config-reference/content/section_compute-scheduler.html + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test area + +- API - Application Programming Interface +- NFVi - Network Functions Virtualization infrastructure +- VIM - Virtual Infrastructure Manager +- VM - Virtual Machine + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + +Test Area Structure +=================== + +The test area is structured based on server group operations and server operations +on multiple nodes. Each test case is able to run independently, i.e. irrelevant of +the state created by a previous test. Specifically, every test performs clean-up +operations which return the system to the same state as before the test. + +Test Descriptions +================= + +API Used and Reference +---------------------- + +Security Groups: https://developer.openstack.org/api-ref/network/v2/index.html#security-groups-security-groups + +- create security group +- delete security group + +Networks: https://developer.openstack.org/api-ref/networking/v2/index.html#networks + +- create network +- delete network + +Routers and interface: https://developer.openstack.org/api-ref/networking/v2/index.html#routers-routers + +- create router +- delete router +- add interface to router + +Subnets: https://developer.openstack.org/api-ref/networking/v2/index.html#subnets + +- create subnet +- delete subnet + +Servers: https://developer.openstack.org/api-ref/compute/ + +- create keypair +- create server +- show server +- delete server +- add/assign floating IP +- create server group +- delete server group +- list server groups +- show server group details + +Ports: https://developer.openstack.org/api-ref/networking/v2/index.html#ports + +- create port +- delete port + +Floating IPs: https://developer.openstack.org/api-ref/networking/v2/index.html#floating-ips-floatingips + +- create floating IP +- delete floating IP + +Availability zone: https://developer.openstack.org/api-ref/compute/ + +- get availability zone + +------------------------------------------ +Test Case 1 - Schedule VM to compute nodes +------------------------------------------ + +Test case specification +----------------------- + +tempest.scenario.test_server_multinode.TestServerMultinode.test_schedule_to_all_nodes + +Test preconditions +------------------ + +* At least 2 compute nodes +* Openstack nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Get all availability zones AZONES1 in the SUT +* Test action 2: Get all compute nodes in AZONES1 +* Test action 3: Get the value of 'min_compute_nodes' which is set by user in tempest + config file and means the minimum number of compute nodes expected +* **Test assertion 1:** Verify that SUT has at least as many compute nodes as + specified by the 'min_compute_nodes' threshold +* Test action 4: Create one server for each compute node, up to the 'min_compute_nodes' threshold +* **Test assertion 2:** Verify the number of servers matches the 'min_compute_nodes' threshold +* Test action 5: Get every server's 'hostId' and store them in a set which has no duplicate values +* **Test assertion 3:** Verify the length of the set equals to the number of servers to ensure + that every server ended up on a different host +* Test action 6: Delete the created servers + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of VM resource scheduling. +Specifically, the test verifies that: + +* VMs are scheduled to the requested compute nodes correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------------------------------------------------- +Test Case 2 - Test create and delete multiple server groups with same name and policy +------------------------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.servers.test_server_group.ServerGroupTestJSON.test_create_delete_multiple_server_groups_with_same_name_policy + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Generate a random name N1 +* Test action 2: Create a server group SERG1 with N1 and policy affinity +* Test action 3: Create another server group SERG2 with N1 and policy affinity +* **Test assertion 1:** The names of SERG1 and SERG2 are the same +* **Test assertion 2:** The 'policies' of SERG1 and SERG2 are the same +* **Test assertion 3:** The ids of SERG1 and SERG2 are different +* Test action 4: Delete SERG1 and SERG2 +* Test action 5: List all server groups +* **Test assertion 4:** SERG1 and SERG2 are not in the list + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of creating and deleting server groups with the same name and policy. +Specifically, the test verifies that: + +* Server groups can be created with the same name and policy. +* Server groups with the same name and policy can be deleted successfully. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------------------------------------- +Test Case 3 - Test create and delete server group with affinity policy +---------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.servers.test_server_group.ServerGroupTestJSON.test_create_delete_server_group_with_affinity_policy + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Generate a random name N1 +* Test action 2: Create a server group SERG1 with N1 and policy affinity +* **Test assertion 1:** The name of SERG1 returned in the response is the same as N1 +* **Test assertion 2:** The 'policies' of SERG1 returned in the response is affinity +* Test action 3: Delete SERG1 and list all server groups +* **Test assertion 3:** SERG1 is not in the list + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of creating and deleting server group with affinity policy. +Specifically, the test verifies that: + +* Server group can be created with affinity policy and deleted successfully. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------------------------------------------- +Test Case 4 - Test create and delete server group with anti-affinity policy +--------------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.servers.test_server_group.ServerGroupTestJSON.test_create_delete_server_group_with_anti_affinity_policy + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Generate a random name N1 +* Test action 2: Create a server group SERG1 with N1 and policy anti-affinity +* **Test assertion 1:** The name of SERG1 returned in the response is the same as N1 +* **Test assertion 2:** The 'policies' of SERG1 returned in the response is anti-affinity +* Test action 3: Delete SERG1 and list all server groups +* **Test assertion 3:** SERG1 is not in the list + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of creating and deleting server group with anti-affinity policy. +Specifically, the test verifies that: + +* Server group can be created with anti-affinity policy and deleted successfully. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------- +Test Case 5 - Test list server groups +------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.servers.test_server_group.ServerGroupTestJSON.test_list_server_groups + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Generate a random name N1 +* Test action 2: Create a server group SERG1 with N1 and policy affinity +* Test action 3: List all server groups +* **Test assertion 1:** SERG1 is in the list +* Test action 4: Delete SERG1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of listing server groups. +Specifically, the test verifies that: + +* Server groups can be listed successfully. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------- +Test Case 6 - Test show server group details +-------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.servers.test_server_group.ServerGroupTestJSON.test_show_server_group + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Generate a random name N1 +* Test action 2: Create a server group SERG1 with N1 and policy affinity, and stored + the details (D1) returned in the response +* Test action 3: Show the details (D2) of SERG1 +* **Test assertion 1:** All values in D1 are the same as the values in D2 +* Test action 4: Delete SERG1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of showing server group details. +Specifically, the test verifies that: + +* Server groups can be shown successfully. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A diff --git a/docs/testing/user/testspecification/old_files/ipv6/designspecification.rst b/docs/testing/user/testspecification/old_files/ipv6/designspecification.rst deleted file mode 100644 index 9e403472..00000000 --- a/docs/testing/user/testspecification/old_files/ipv6/designspecification.rst +++ /dev/null @@ -1,133 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) Christopher Price (Ericsson AB) and others - -============================== -IPv6 test design specification -============================== - -This document outlines the approach and method for testing IPv6 in the OPNFV compliance test -suite. Providing a brief outline of the features to be tested, the methodology for testing, -schema's and criteria. - -Features to be tested -===================== - -The IPv6 compliance test plan outlines the method for testing IPv6 compliance to the OPNFV -platform behaviours and features of IPv6 enabled VNFi platforms. The specific features to -be tested by the IPv6 compliance test suite is outlined in the following table. - -.. table:: - :class: longtable - -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ -|Features / Requirements |Tests available | Test Cases | -+===========================================================+===================+====================================================================+ -|All topologies work in a multi-tenant environment |No | | -| | | | -| | | | -| | | | -| | | | -| | | | -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ -|IPv6 VM to VM only |No | | -| | | | -| | | | -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ -|IPv6 external L2 VLAN directly attached to a VM |No | | -| | | | -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ -|IPv6 subnet routed via L3 agent to an external IPv6 network|No | | -| | | | -|1. Both VLAN and overlay (e.g. GRE, VXLAN) subnet attached | | | -| to VMs; | | | -|2. Must be able to support multiple L3 agents for a given | | | -| external network to support scaling (neutron scheduler | | | -| to assign vRouters to the L3 agents) | | | -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ -|Ability for a NIC to support both IPv4 and IPv6 (dual |No | | -|stack) address. | | | -| | | | -|1. VM with a single interface associated with a network, | | | -| which is then associated with two subnets. | | | -|2. VM with two different interfaces associated with two | | | -| different networks and two different subnets. | | | -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ -|Support IPv6 Address assignment modes. |No | | -| | | | -|1. SLAAC | | | -|2. DHCPv6 Stateless | | | -|3. DHCPv6 Stateful | | | -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ -|Ability to create a port on an IPv6 DHCPv6 Stateful subnet |No | | -|and assign a specific IPv6 address to the port and have it | | | -|taken out of the DHCP address pool. | | | -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ -|Full support for IPv6 matching (i.e., IPv6, ICMPv6, TCP, |No | | -|UDP) in security groups. Ability to control and manage all | | | -|IPv6 security group capabilities via Neutron/Nova API (REST| | | -|and CLI) as well as via Horizon. | | | -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ -|During network/subnet/router create, there should be an |No | | -|option to allow user to specify the type of address | | | -|management they would like. This includes all options | | | -|including those low priority if implemented (e.g., toggle | | | -|on/off router and address prefix advertisements); It must | | | -|be supported via Neutron API (REST and CLI) as well as via | | | -|Horizon | | | -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ -|Security groups anti-spoofing: Prevent VM from using a |No | | -|source IPv6/MAC address which is not assigned to the VM | | | -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ -|Protect tenant and provider network from rogue RAs |No | | -| | | | -| | | | -| | | | -| | | | -| | | | -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ -|Support the ability to assign multiple IPv6 addresses to |No | | -|an interface; both for Neutron router interfaces and VM | | | -|interfaces. | | | -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ -|Ability for a VM to support a mix of multiple IPv4 and IPv6|No | | -|networks, including multiples of the same type. | | | -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ -|Support for IPv6 Prefix Delegation. |No | | -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ -|IPv6 First-Hop Security, IPv6 ND spoofing |No | | -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ -|IPv6 support in Neutron Layer3 High Availability |No | | -|(keepalived+VRRP). | | | -+-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ - - -Test approach for IPv6 -====================== - -The most common approach for testing IPv6 capabilities in the test suite is through interaction with the SUT control plane. -In this instance the test framework will exercise the NBI provided by the VIM to configure and leverage IPv6 related features -in the platform, instantiate workloads, and invoke behaviours in the platform. The suite may also interact directly with the -data plane to exercise platform capabilities and further invoke helper functions on the platform for the same purpose. - -Test result analysis --------------------- - -All functional tests in the IPv6 test suite will provide a pass/fail result on completion of the test. In addition test logs -and relevant additional information will be provided as part of the test log, available on test suite completion. - -Some tests in the compliance suite measure such metrics as latency and performance. At this time these tests are intended to -provide a feature based pass/fail metric not related to system performance. -These tests may however provide detailed results of performance and latency in the 'test report'_ document. - -Test identification -=================== - -TBD: WE need to identify the test naming scheme we will use in DoveTail in order that we can cross reference to the test -projects and maintain our suite effectively. This naming scheme needs to be externally relevant to non-OPNFV consumers and as -such some consideration is required on the selection. - -Pass Fail Criteria -================== - -This section requires some further work with the test teams to identify how and where we generate, store and provide results. diff --git a/docs/testing/user/testspecification/old_files/ipv6/index.rst b/docs/testing/user/testspecification/old_files/ipv6/index.rst deleted file mode 100644 index a806d644..00000000 --- a/docs/testing/user/testspecification/old_files/ipv6/index.rst +++ /dev/null @@ -1,19 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV - -******************************* -OPNFV IPv6 Compliance Test Plan -******************************* - -.. toctree:: - :maxdepth: 2 - - ./testplan.rst - ./testprocedure.rst - ./testspecification.rst - ./designspecification.rst - ./ipv6.tc001.specification.rst - ./ipv6.tc026.specification.rst - ./ipv6_all_testcases.rst - diff --git a/docs/testing/user/testspecification/old_files/ipv6/ipv6.tc001.specification.rst b/docs/testing/user/testspecification/old_files/ipv6/ipv6.tc001.specification.rst deleted file mode 100644 index 5afb2095..00000000 --- a/docs/testing/user/testspecification/old_files/ipv6/ipv6.tc001.specification.rst +++ /dev/null @@ -1,59 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV - -================================================================================================== -Dovetail IPv6 tc001 specification - Bulk Creation and Deletion of IPv6 Networks, Ports and Subnets -================================================================================================== - - -+-----------------------+----------------------------------------------------------------------------------------------------+ -|test case name |Bulk creation and deletion of IPv6 networks, ports and subnets | -| | | -+-----------------------+----------------------------------------------------------------------------------------------------+ -|id |dovetail.ipv6.tc001 | -+-----------------------+----------------------------------------------------------------------------------------------------+ -|objective |To verify that platform is able to create/delete networks, ports and subnets in bulk operation | -+-----------------------+----------------------------------------------------------------------------------------------------+ -|test items |tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network | -| |{idempotent_id('d4f9024d-1e28-4fc1-a6b1-25dbc6fa11e2')} | -| |tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port | -| |{idempotent_id('48037ff2-e889-4c3b-b86a-8e3f34d2d060')} | -| |tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet | -| |{idempotent_id('8936533b-c0aa-4f29-8e53-6cc873aec489')} | -+-----------------------+----------------------------------------------------------------------------------------------------+ -|environmental | | -|requirements & | environment can be deployed on bare metal of virtualized infrastructure | -|preconditions | deployment can be HA or non-HA | -| | | -+-----------------------+----------------------------------------------------------------------------------------------------+ -|scenario dependencies | NA | -+-----------------------+----------------------------------------------------------------------------------------------------+ -|procedural |Step 1: create/delete network: | -|requirements | create 2 networks in one request | -| | asserting that the networks are found in the list after creation | -| | | -| |Step 2: create/delete subnet: | -| | create 2 subnets in one request | -| | asserting that the subnets are found in the list after creation | -| | | -| |Step 3: create/delete port: | -| | create 2 ports in one request | -| | asserting that the ports are found in the list after creation | -| | | -+-----------------------+----------------------------------------------------------------------------------------------------+ -|input specifications |The parameters needed to execute Neutron network APIs. | -| |Refer to Neutron Networking API v2.0 `[1]`_ `[2]`_ | -+-----------------------+----------------------------------------------------------------------------------------------------+ -|output specifications |The responses after executing Network network APIs. | -| |Refer to Neutron Networking API v2.0 `[1]`_ `[2]`_ | -+-----------------------+----------------------------------------------------------------------------------------------------+ -|pass/fail criteria |If normal response code 200 is returned, the test passes. | -| |Otherwise, the test fails with various error codes. | -| |Refer to Neutron Networking API v2.0 `[1]`_ `[2]`_ | -+-----------------------+----------------------------------------------------------------------------------------------------+ -|test report |TBD | -+-----------------------+----------------------------------------------------------------------------------------------------+ - -.. _`[1]`: http://developer.openstack.org/api-ref/networking/v2/ -.. _`[2]`: http://wiki.openstack.org/wiki/Neutron/APIv2-specification diff --git a/docs/testing/user/testspecification/old_files/ipv6/ipv6.tc026.specification.rst b/docs/testing/user/testspecification/old_files/ipv6/ipv6.tc026.specification.rst deleted file mode 100644 index e7fd82e7..00000000 --- a/docs/testing/user/testspecification/old_files/ipv6/ipv6.tc026.specification.rst +++ /dev/null @@ -1,54 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV - -============================================================== -Dovetail IPv6 tc026 specification - Service VM as IPv6 vRouter -============================================================== - - -+-----------------------+--------------------------------------------------------------------------+ -|test case name |Service VM as IPv6 vRouter | -| | | -+-----------------------+--------------------------------------------------------------------------+ -|id |dovetail.ipv6.tc026 | -+-----------------------+--------------------------------------------------------------------------+ -|objective |IPv6 connnectivity, service VM as IPv6 vRouter | -+-----------------------+--------------------------------------------------------------------------+ -|modules under test |neutron, nova, etc | -+-----------------------+--------------------------------------------------------------------------+ -|dependent test project |yardstick | -+-----------------------+--------------------------------------------------------------------------+ -|test items |yardstick_tc027 | -+-----------------------+--------------------------------------------------------------------------+ -|environmental | OpenStack-only environment | -|requirements & | environment can be deplyed on bare metal of virtualized infrastructure | -|preconditions | deployment can be HA or non-HA | -| | test case image needs to be installed into Glance with ping6 included | -+-----------------------+--------------------------------------------------------------------------+ -|scenario dependencies | nosdn | -+-----------------------+--------------------------------------------------------------------------+ -|procedural |step 1: to setup IPv6 testing environment | -|requirements | 1.1 disable security group | -| | 1.2 create (ipv6, ipv4) router, network and subnet | -| | 1.3 create vRouter, VM1, VM2 | -| |step 2: to run ping6 to verify IPv6 connectivity | -| | 2.1 ssh to VM1 | -| | 2.2 ping6 to ipv6 router from VM1 | -| | 2.3 get the result and store the logs | -| |step 3: to teardown IPv6 testing environment | -| | 3.1 delete vRouter, VM1, VM2 | -| | 3.2 delete (ipv6, ipv4) router, network and subnet | -| | 3.3 enable security group | -+-----------------------+--------------------------------------------------------------------------+ -|input specifications |packetsize: 56 | -| |ping_count: 5 | -| | | -+-----------------------+--------------------------------------------------------------------------+ -|output specifications |output includes max_rtt, min_rtt, average_rtt | -+-----------------------+--------------------------------------------------------------------------+ -|pass/fail criteria |ping6 connectivity success, no SLA | -+-----------------------+--------------------------------------------------------------------------+ -|test report | dovetail dashboard DB here | -+-----------------------+--------------------------------------------------------------------------+ - diff --git a/docs/testing/user/testspecification/old_files/ipv6/ipv6_all_testcases.rst b/docs/testing/user/testspecification/old_files/ipv6/ipv6_all_testcases.rst deleted file mode 100644 index 02115ec3..00000000 --- a/docs/testing/user/testspecification/old_files/ipv6/ipv6_all_testcases.rst +++ /dev/null @@ -1,243 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV - -================================================== -IPv6 Compliance Testing Methodology and Test Cases -================================================== - -IPv6 Compliance Testing focuses on overlay IPv6 capabilities, i.e. to validate that -IPv6 capability is supported in tenant networks, subnets and routers. Both Tempest API -testing and Tempest Scenario testing are reused as much as we can in IPv6 Compliance -Testing. In addition, Yardstick Test Case 027 is also used to validate a specific use case -of using a Service VM as an IPv6 vRouter. - -IPv6 Compliance Testing test cases are described as follows: - ---------------------------------------------------------------- -Test Case 1: Create and Delete an IPv6 Network, Port and Subnet ---------------------------------------------------------------- - -.. code-block:: bash - - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet - ------------------------------------------------------------------ -Test Case 2: Create, Update and Delete an IPv6 Network and Subnet ------------------------------------------------------------------ - -.. code-block:: bash - - tempest.api.network.test_networks.NetworksIpV6Test.test_create_update_delete_network_subnet - ----------------------------------------------- -Test Case 3: Check External Network Visibility ----------------------------------------------- - -.. code-block:: bash - - tempest.api.network.test_networks.NetworksIpV6Test.test_external_network_visibility - -------------------------------------------------------- -Test Case 4: List IPv6 Networks and Subnets of a Tenant -------------------------------------------------------- - -.. code-block:: bash - - tempest.api.network.test_networks.NetworksIpV6Test.test_list_networks - tempest.api.network.test_networks.NetworksIpV6Test.test_list_subnets - ------------------------------------------------------------ -Test Case 5: Show Information of an IPv6 Network and Subnet ------------------------------------------------------------ - -.. code-block:: bash - - tempest.api.network.test_networks.NetworksIpV6Test.test_show_network - tempest.api.network.test_networks.NetworksIpV6Test.test_show_subnet - ------------------------------------------------------------- -Test Case 6: Create an IPv6 Port in Allowed Allocation Pools ------------------------------------------------------------- - -.. code-block:: bash - - tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools - --------------------------------------------------------- -Test Case 7: Create an IPv6 Port without Security Groups --------------------------------------------------------- - -.. code-block:: bash - - tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_with_no_securitygroups - ---------------------------------------------------- -Test Case 8: Create, Update and Delete an IPv6 Port ---------------------------------------------------- - -.. code-block:: bash - - tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_update_delete_port - ----------------------------------------- -Test Case 9: List IPv6 Ports of a Tenant ----------------------------------------- - -.. code-block:: bash - - tempest.api.network.test_ports.PortsIpV6TestJSON.test_list_ports - ----------------------------------------------- -Test Case 10: Show Information of an IPv6 Port ----------------------------------------------- - -.. code-block:: bash - - tempest.api.network.test_ports.PortsIpV6TestJSON.test_show_port - --------------------------------------------------------- -Test Case 11: Add Multiple Interfaces for an IPv6 Router --------------------------------------------------------- - -.. code-block:: bash - - tempest.api.network.test_routers.RoutersIpV6Test.test_add_multiple_router_interfaces - ------------------------------------------------------------------- -Test Case 12: Add and Remove an IPv6 Router Interface with port_id ------------------------------------------------------------------- - -.. code-block:: bash - - tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_port_id - --------------------------------------------------------------------- -Test Case 13: Add and Remove an IPv6 Router Interface with subnet_id --------------------------------------------------------------------- - -.. code-block:: bash - - tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_subnet_id - ------------------------------------------------------------------- -Test Case 14: Create, Update, Delete, List and Show an IPv6 Router ------------------------------------------------------------------- - -.. code-block:: bash - - tempest.api.network.test_routers.RoutersIpV6Test.test_create_show_list_update_delete_router - --------------------------------------------------------------------------- -Test Case 15: Create, Update, Delete, List and Show an IPv6 Security Group --------------------------------------------------------------------------- - -.. code-block:: bash - - tempest.api.network.test_security_groups.SecGroupIPv6Test.test_create_list_update_show_delete_security_group - ----------------------------------------------------------- -Test Case 16: Create, Delete and Show Security Group Rules ----------------------------------------------------------- - -.. code-block:: bash - - tempest.api.network.test_security_groups.SecGroupIPv6Test.test_create_show_delete_security_group_rule - --------------------------------------- -Test Case 17: List All Security Groups --------------------------------------- - -.. code-block:: bash - - tempest.api.network.test_security_groups.SecGroupIPv6Test.test_list_security_groups - --------------------------------------------------------- -Test Case 18: IPv6 Address Assignment - DHCPv6 Stateless --------------------------------------------------------- - -.. code-block:: bash - - tempest.scenario.test_network_v6.TestGettingAddress.test_dhcp6_stateless_from_os - --------------------------------------------------------------------- -Test Case 19: IPv6 Address Assignment - Dual Stack, DHCPv6 Stateless --------------------------------------------------------------------- - -.. code-block:: bash - - tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_dhcp6_stateless_from_os - ---------------------------------------------------------------------------- -Test Case 20: IPv6 Address Assignment - Multiple Prefixes, DHCPv6 Stateless ---------------------------------------------------------------------------- - -.. code-block:: bash - - tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_dhcpv6_stateless - ---------------------------------------------------------------------------------------- -Test Case 21: IPv6 Address Assignment - Dual Stack, Multiple Prefixes, DHCPv6 Stateless ---------------------------------------------------------------------------------------- - -.. code-block:: bash - - tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_dhcpv6_stateless - ---------------------------------------------- -Test Case 22: IPv6 Address Assignment - SLAAC ---------------------------------------------- - -.. code-block:: bash - - tempest.scenario.test_network_v6.TestGettingAddress.test_slaac_from_os - ---------------------------------------------------------- -Test Case 23: IPv6 Address Assignment - Dual Stack, SLAAC ---------------------------------------------------------- - -.. code-block:: bash - - tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_slaac_from_os - ----------------------------------------------------------------- -Test Case 24: IPv6 Address Assignment - Multiple Prefixes, SLAAC ----------------------------------------------------------------- - -.. code-block:: bash - - tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_slaac - ----------------------------------------------------------------------------- -Test Case 25: IPv6 Address Assignment - Dual Stack, Multiple Prefixes, SLAAC ----------------------------------------------------------------------------- - -.. code-block:: bash - - tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_slaac - -------------------------------------------- -Test Case 26: Service VM as an IPv6 vRouter -------------------------------------------- - -.. code-block:: bash - - # Refer to Yardstick Test Case 027 - # Instruction: http://artifacts.opnfv.org/ipv6/docs/configurationguide/index.html - # Step 1: Set up Service VM as an IPv6 vRouter - # 1.1: Install OPNFV and Preparation - # 1.2: Disable Security Groups in OpenStack ML2 Setup - # 1.3: Create IPv4 and IPv6 Neutron routers, networks and subnets - # 1.4: Boot vRouter VM, and Guest VM1 and Guest VM2 - # Step 2: Verify IPv6 Connectivity - # 2.1: ssh to Guest VM1 - # 2.2: Ping6 from Guest VM1 to Guest VM2 - # 2.3: Ping6 from Guest VM1 to vRouter VM - # 2.4: Ping6 from Guest VM1 to Neutron IPv6 Router Namespace - # Step 3: Tear down Setup - # 3.1: Delete Guest VM1, Guest VM2 and vRouter VM - # 3.2: Delete IPv4 and IPv6 Neutron routers, networks and subnets - # 3.3: Enable Security Groups - diff --git a/docs/testing/user/testspecification/old_files/ipv6/testplan.rst b/docs/testing/user/testspecification/old_files/ipv6/testplan.rst deleted file mode 100644 index 3470e7a6..00000000 --- a/docs/testing/user/testspecification/old_files/ipv6/testplan.rst +++ /dev/null @@ -1,34 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV - -=============================== -OPNFV IPv6 Compliance Test Plan -=============================== - -Introduction -============ - -The IPv6 compliance test plan outlines the method for testing IPv6 Tenant Network feature -compliance with the OPNFV platform. - -Scope ------ - -This test, and other tests in the test suite, are designed to verify an entire SUT, -and not any individual component of the system. - -Test suite scope and procedures -=============================== - -The IPv6 compliance test suite will evaluate the ability for a SUT to support IPv6 -Tenant Network features and functionality provided by OPNFV platform. - -Please refer to the complete list of the test cases for details. - -Test suite execution -==================== - -Please refer to each test case for specific setup and execution procedure. - -.._[1]: http://www.opnfv.org diff --git a/docs/testing/user/testspecification/old_files/ipv6/testprocedure.rst b/docs/testing/user/testspecification/old_files/ipv6/testprocedure.rst deleted file mode 100644 index 2119ed61..00000000 --- a/docs/testing/user/testspecification/old_files/ipv6/testprocedure.rst +++ /dev/null @@ -1,9 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) Christopher Price (Ericsson AB) and others - -=================== -IPv6 test procedure -=================== - -Draft to be patched this week, someone feel free to work on this in parallel. diff --git a/docs/testing/user/testspecification/old_files/ipv6/testspecification.rst b/docs/testing/user/testspecification/old_files/ipv6/testspecification.rst deleted file mode 100644 index e51f2a5b..00000000 --- a/docs/testing/user/testspecification/old_files/ipv6/testspecification.rst +++ /dev/null @@ -1,57 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) Christopher Price (Ericsson AB) and others - -=============================================== -Test specification - Service VM as IPv6 vRouter -=============================================== - -Draft to be worked on, this represents the YardStick test but I would suggest we need to break -this into a set of tests which provide more details per action with boundary validation. - -Test Item -========= - -TBD -> IPv6 Ping... - -Identify the items or features to be tested by this test case. The item description and -definition can be referenced from any one of several sources, depending on the level of the -test case specification. It may be a good idea to reference the source documents as well. - -Environmental requirements -========================== - -For ipv6 Test Case 18-25, those test cases are scenario tests, they need to boot virtual -machines and ping6 in addition to test APIs, ping6 to vRouter is not supported by SDN controller -yet, such as Opendaylight (Boron and previous releases), so they are scenario dependent, -i.e., currently ipv6 Test Case 18-25 can only run on scenario os-nosdn-nofeature. - -Preconditions and procedural requirements -========================================= - -TBD - -.. <Start> -.. this section may be iterated over for a set of simillar test cases that would be run as one. - -Input Specifications -==================== - -TBD - -Output Specifications -===================== - -TBD - -.. <End> - -Test Reporting -============== - -The test report for this test case will be generated with links to relevant data sources. -This section can be updated once we have a template for the report in place. - -http://testresults.opnfv.org/grafana/dashboard/db/yardstick-tc027 - - diff --git a/docs/testing/user/testspecification/securitygroup/index.rst b/docs/testing/user/testspecification/securitygroup/index.rst new file mode 100644 index 00000000..0621b84d --- /dev/null +++ b/docs/testing/user/testspecification/securitygroup/index.rst @@ -0,0 +1,450 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Huawei Technologies Co.,Ltd + +=================================================== +Security Group and Port Security test specification +=================================================== + +.. toctree:: + :maxdepth: 2 + +Scope +===== + +The security group and port security test area evaluates the ability of the +system under test to support packet filtering by security group and port security. +The tests in this test area will evaluate preventing MAC spoofing by port security, +basic security group operations including testing cross/in tenant traffic, testing +multiple security groups, using port security to disable security groups and +updating security groups. + +References +========== + +N/A + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test +area + +- API - Application Programming Interface +- ICMP - Internet Control Message Protocol +- MAC - Media Access Control +- NFVi - Network Functions Virtualization infrastructure +- SSH - Secure Shell +- TCP - Transmission Control Protocol +- VIM - Virtual Infrastructure Manager +- VM - Virtual Machine + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + +Test Area Structure +=================== + +The test area is structured based on the basic operations of security group and +port security. Each test case is able to run independently, i.e. irrelevant of +the state created by a previous test. Specifically, every test performs clean-up +operations which return the system to the same state as before the test. + +Test Descriptions +================= + +API Used and Reference +---------------------- + +Security Groups: https://developer.openstack.org/api-ref/network/v2/index.html#security-groups-security-groups + +- create security group +- delete security group + +Networks: https://developer.openstack.org/api-ref/networking/v2/index.html#networks + +- create network +- delete network +- list networks +- create floating ip +- delete floating ip + +Routers and interface: https://developer.openstack.org/api-ref/networking/v2/index.html#routers-routers + +- create router +- delete router +- list routers +- add interface to router + +Subnets: https://developer.openstack.org/api-ref/networking/v2/index.html#subnets + +- create subnet +- list subnets +- delete subnet + +Servers: https://developer.openstack.org/api-ref/compute/ + +- create keypair +- create server +- delete server +- add/assign floating ip + +Ports: https://developer.openstack.org/api-ref/networking/v2/index.html#ports + +- update port +- list ports +- show port details + +-------------------------------------------- +Test Case 1 - Port Security and MAC Spoofing +-------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_port_security_macspoofing_port + +Test preconditions +------------------ + +* Neutron port-security extension API +* Neutron security-group extension API +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a security group SG1, which has rules for allowing incoming + SSH and ICMP traffic +* Test action 2: Create a neutron network NET1 +* Test action 3: Create a tenant router R1 which routes traffic to public network +* Test action 4: Create a subnet SUBNET1 and add it as router interface +* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating ip + FIP1 (via R1) to VM1 +* Test action 6: Verify can ping FIP1 successfully and can SSH to VM1 with FIP1 +* Test action 7: Create a second neutron network NET2 and subnet SUBNET2, and attach VM1 to NET2 +* Test action 8: Get VM1's ethernet interface NIC2 for NET2 +* Test action 9: Create second server VM2 on NET2 +* Test action 10: Verify VM1 is able to communicate with VM2 via NIC2 +* Test action 11: Login to VM1 and spoof the MAC address of NIC2 to "00:00:00:00:00:01" +* Test action 12: Verify VM1 fails to communicate with VM2 via NIC2 +* **Test assertion 1:** The ping operation is failed +* Test action 13: Update 'security_groups' to be none for VM1's NIC2 port +* Test action 14: Update 'port_security_enable' to be False for VM1's NIC2 port +* Test action 15: Verify now VM1 is able to communicate with VM2 via NIC2 +* **Test assertion 2:** The ping operation is successful +* Test action 16: Delete SG1, NET1, NET2, SUBNET1, SUBNET2, R1, VM1, VM2 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to prevent MAC spoofing by using port security. +Specifically, the test verifies that: + +* With port security, the ICMP packets from a spoof server cannot pass the port. + +* Without port security, the ICMP packets from a spoof server can pass the port. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------------------ +Test Case 2 - Test Security Group Cross Tenant Traffic +------------------------------------------------------ + +Test case specification +----------------------- + +tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic + +Test preconditions +------------------ + +* Neutron security-group extension API +* Two tenants +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a neutron network NET1 for primary tenant +* Test action 2: Create a primary tenant router R1 which routes traffic to public network +* Test action 3: Create a subnet SUBNET1 and add it as router interface +* Test action 4: Create 2 empty security groups SG1 and SG2 for primary tenant +* Test action 5: Add a tcp rule to SG1 +* Test action 6: Create a server VM1 with SG1, SG2 and NET1, and assign a floating ip + FIP1 (via R1) to VM1 +* Test action 7: Repeat test action 1 to 6 and create NET2, R2, SUBNET2, SG3, SG4, + FIP2 and VM2 for an alt_tenant +* Test action 8: Verify VM1 fails to communicate with VM2 through FIP2 +* **Test assertion 1:** The ping operation is failed +* Test action 9: Add ICMP rule to SG4 +* Test action 10: Verify VM1 is able to communicate with VM2 through FIP2 +* **Test assertion 2:** The ping operation is successful +* Test action 11: Verify VM2 fails to communicate with VM1 through FIP1 +* **Test assertion 3:** The ping operation is failed +* Test action 12: Add ICMP rule to SG2 +* Test action 13: Verify VM2 is able to communicate with VM1 through FIP1 +* **Test assertion 4:** The ping operation is successful +* Test action 14: Delete SG1, SG2, SG3, SG4, NET1, NET2, SUBNET1, SUBNET2, R1, R2, + VM1, VM2, FIP1 and FIP2 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability of the security group to filter packets cross tenant. +Specifically, the test verifies that: + +* Without ICMP security group rule, the ICMP packets cannot be received by the server + in another tenant which differs from the source server. + +* With ingress ICMP security group rule enabled only at tenant1, the server in tenant2 + can ping server in tenant1 but not the reverse direction. + +* With ingress ICMP security group rule enabled at tenant2 also, the ping works from both directions. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------------------- +Test Case 3 - Test Security Group in Tenant Traffic +--------------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_in_tenant_traffic + +Test preconditions +------------------ + +* Neutron security-group extension API +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a neutron network NET1 +* Test action 2: Create a tenant router R1 which routes traffic to public network +* Test action 3: Create a subnet SUBNET1 and add it as router interface +* Test action 4: Create 2 empty security groups SG1 and SG2 +* Test action 5: Add a tcp rule to SG1 +* Test action 6: Create a server VM1 with SG1, SG2 and NET1, and assign a floating ip + FIP1 (via R1) to VM1 +* Test action 7: Create second server VM2 with default security group and NET1 +* Test action 8: Verify VM1 fails to communicate with VM2 through VM2's fixed ip +* **Test assertion 1:** The ping operation is failed +* Test action 9: Add ICMP security group rule to default security group +* Test action 10: Verify VM1 is able to communicate with VM2 through VM2's fixed ip +* **Test assertion 2:** The ping operation is successful +* Test action 11: Delete SG1, SG2, NET1, SUBNET1, R1, VM1, VM2 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability of the security group to filter packets in one tenant. +Specifically, the test verifies that: + +* Without ICMP security group rule, the ICMP packets cannot be received by the server + in the same tenant. + +* With ICMP security group rule, the ICMP packets can be received by the server + in the same tenant. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------- +Test Case 4 - Test Multiple Security Groups +------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_multiple_security_groups + +Test preconditions +------------------ + +* Neutron security-group extension API +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a neutron network NET1 +* Test action 2: Create a tenant router R1 which routes traffic to public network +* Test action 3: Create a subnet SUBNET1 and add it as router interface +* Test action 4: Create 2 empty security groups SG1 and SG2 +* Test action 5: Add a tcp rule to SG1 +* Test action 6: Create a server VM1 with SG1, SG2 and NET1, and assign a floating ip + FIP1 (via R1) to VM1 +* Test action 7: Verify failed to ping FIP1 +* **Test assertion 1:** The ping operation is failed +* Test action 8: Add ICMP security group rule to SG2 +* Test action 9: Verify can ping FIP1 successfully +* **Test assertion 2:** The ping operation is successful +* Test action 10: Verify can SSH to VM1 with FIP1 +* **Test assertion 3:** Can SSH to VM1 successfully +* Test action 11: Delete SG1, SG2, NET1, SUBNET1, R1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability of multiple security groups to filter packets. +Specifically, the test verifies that: + +* A server with 2 security groups, one with TCP rule and without ICMP rule, + cannot receive the ICMP packets sending from the tempest host machine. + +* A server with 2 security groups, one with TCP rule and the other with ICMP rule, + can receive the ICMP packets sending from the tempest host machine and be connected + via the SSH client. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------------------- +Test Case 5 - Test Port Security Disable Security Group +------------------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_port_security_disable_security_group + +Test preconditions +------------------ + +* Neutron security-group extension API +* Neutron port-security extension API +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a neutron network NET1 +* Test action 2: Create a tenant router R1 which routes traffic to public network +* Test action 3: Create a subnet SUBNET1 and add it as router interface +* Test action 4: Create 2 empty security groups SG1 and SG2 +* Test action 5: Add a tcp rule to SG1 +* Test action 6: Create a server VM1 with SG1, SG2 and NET1, and assign a floating ip + FIP1 (via R1) to VM1 +* Test action 7: Create second server VM2 with default security group and NET1 +* Test action 8: Update 'security_groups' to be none and 'port_security_enabled' to be + True for VM2's port +* Test action 9: Verify VM1 fails to communicate with VM2 through VM2's fixed ip +* **Test assertion 1:** The ping operation is failed +* Test action 10: Update 'security_groups' to be none and 'port_security_enabled' to be + False for VM2's port +* Test action 11: Verify VM1 is able to communicate with VM2 through VM2's fixed ip +* **Test assertion 2:** The ping operation is successful +* Test action 12: Delete SG1, SG2, NET1, SUBNET1, R1, VM1, VM2 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability of port security to disable security group. +Specifically, the test verifies that: + +* The ICMP packets cannot pass the port whose 'port_security_enabled' is True + and security_groups is none. + +* The ICMP packets can pass the port whose 'port_security_enabled' is False + and security_groups is none. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------------- +Test Case 6 - Test Update Port Security Group +--------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_port_update_new_security_group + +Test preconditions +------------------ + +* Neutron security-group extension API +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a neutron network NET1 +* Test action 2: Create a tenant router R1 which routes traffic to public network +* Test action 3: Create a subnet SUBNET1 and add it as router interface +* Test action 4: Create 2 empty security groups SG1 and SG2 +* Test action 5: Add a tcp rule to SG1 +* Test action 6: Create a server VM1 with SG1, SG2 and NET1, and assign a floating ip + FIP1 (via R1) to VM1 +* Test action 7: Create third empty security group SG3 +* Test action 8: Add ICMP rule to SG3 +* Test action 9: Create second server VM2 with default security group and NET1 +* Test action 10: Verify VM1 fails to communicate with VM2 through VM2's fixed ip +* **Test assertion 1:** The ping operation is failed +* Test action 11: Update 'security_groups' to be SG3 for VM2's port +* Test action 12: Verify VM1 is able to communicate with VM2 through VM2's fixed ip +* **Test assertion 2:** The ping operation is successful +* Test action 13: Delete SG1, SG2, SG3, NET1, SUBNET1, R1, VM1, VM2 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to update port with a new security group. +Specifically, the test verifies that: + +* Without ICMP security group rule, the VM cannot receive ICMP packets. + +* Update the port's security group which has ICMP rule, the VM can receive ICMP packets. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A diff --git a/docs/testing/user/testspecification/vimoperationscompute/index.rst b/docs/testing/user/testspecification/vimoperationscompute/index.rst index 4ed37809..42a8e70e 100644 --- a/docs/testing/user/testspecification/vimoperationscompute/index.rst +++ b/docs/testing/user/testspecification/vimoperationscompute/index.rst @@ -1,6 +1,6 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. (c) Ericsson AB +.. (c) Ericsson AB, Huawei Technologies Co.,Ltd ========================================= VIM compute operations test specification @@ -12,47 +12,375 @@ VIM compute operations test specification Scope ===== +The VIM compute operations test area evaluates the ability of the system under +test to support VIM compute operations. The test cases documented here are the +compute API test cases in the OpenStack Interop guideline 2016.8 as implemented +by the RefStack client. These test cases will evaluate basic OpenStack (as a VIM) +compute operations, including: + +- Image management operations +- Basic support operations +- API version support operations +- Quotas management operations +- Basic server operations +- Volume management operations + References ================ +- OpenStack Governance/Interop + + - https://wiki.openstack.org/wiki/Governance/InteropWG + +- Defcore test cases + + - https://github.com/openstack/interop/blob/master/2016.08.json + +- Refstack client + + - https://github.com/openstack/refstack-client Definitions and abbreviations ============================= +The following terms and abbreviations are used in conjunction with this test area -Use case description -==================== - +- API - Application Programming Interface +- NFVi - Network Functions Virtualization infrastructure +- SUT - System Under Test +- UUID - Universally Unique Identifier +- VIM - Virtual Infrastructure Manager +- VM - Virtual Machine System Under Test (SUT) ======================= +The system under test is assumed to be the NFVi and VIM deployed with a Pharos compliant infrastructure. -Test Suite Structure +Test Area Structure ==================== +The test area is structured based on VIM compute API operations. Each test case is +able to run independently, i.e. irrelevant of the state created by a previous test. +Specifically, every test performs clean-up operations which return the system to +the same state as before the test. + +For brevity, the test cases in this test area are summarized together based on +the operations they are testing. Test Descriptions ================= +API Used and Reference +---------------------- + +Servers: https://developer.openstack.org/api-ref/compute/ + +- create server +- delete server +- list servers +- start server +- stop server +- update server +- get server action +- set server metadata +- update server metadata +- rebuild server + +- create image +- delete image + +- create keypair +- delete keypair + +Block storage: https://developer.openstack.org/api-ref/block-storage + +- create volume +- delete volume +- attach volume to server +- detach volume from server + +----------------------------------------------------- +Test Case 1 - Image operations within the Compute API +----------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image +tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_image_specify_multibyte_character_image_name + +Test preconditions +------------------ + +* Compute server extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a server VM1 with an image IMG1 and wait for VM1 to reach 'ACTIVE' status +* Test action 2: Create a new server image IMG2 from VM1, specifying image name + and image metadata. Wait for IMG2 to reach 'ACTIVE' status, and then delete IMG2 +* **Test assertion 1:** Verify IMG2 is created with correct image name and image + metadata; verify IMG1's 'minRam' equals to IMG2's 'minRam' and IMG2's 'minDisk' equals + to IMG1's 'minDisk' or VM1's flavor disk size +* **Test assertion 2:** Verify IMG2 is deleted correctly +* Test action 3: Create another server IMG3 from VM1, specifying image name + with a 3 byte utf-8 character +* **Test assertion 3:** Verify IMG3 is created correctly +* Test action 4: Delete VM1, IMG1 and IMG3 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the Compute API ability of creating image from server, +deleting image, creating server image with multi-byte character name. +Specifically, the test verifies that: + +* Compute server create image and delete image APIs work correctly. +* Compute server image can be created with multi-byte character name. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +----------------------------------------------------- +Test Case 2 - Action operation within the Compute API +----------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_get_instance_action +tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_list_instance_actions + +Test preconditions +------------------ + +* Compute server extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a server VM1 and wait for VM1 to reach 'ACTIVE' status +* Test action 2: Get the action details ACT_DTL of VM1 +* **Test assertion 1:** Verify ACT_DTL's 'instance_uuid' matches VM1's ID and + ACT_DTL's 'action' matched 'create' +* Test action 3: Create a server VM2 and wait for VM2 to reach 'ACTIVE' status +* Test action 4: Delete server VM2 and wait for VM2 to reach termination +* Test action 5: Get the action list ACT_LST of VM2 +* **Test assertion 2:** Verify ACT_LST's length is 2 and two actions are 'create' and 'delete' +* Test action 6: Delete VM1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the Compute API ability of getting the action details +of a provided server and getting the action list of a deleted server. +Specifically, the test verifies that: + +* Get the details of the action in a specified server. +* List the actions that were performed on the specified server. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------------------------- +Test Case 3 - Generate, import and delete SSH keys within Compute services +-------------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_specify_keypair + +Test preconditions +------------------ + +* Compute server extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 and list all existing keypairs +* Test action 2: Create a server VM1 with KEYP1 and wait for VM1 to reach 'ACTIVE' status +* Test action 3: Show details of VM1 +* **Test assertion 1:** Verify value of 'key_name' in the details equals to the name of KEYP1 +* Test action 4: Delete KEYP1 and VM1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the Compute API ability of creating a keypair, listing +keypairs and creating a server with a provided keypair. +Specifically, the test verifies that: + +* Compute create keypair and list keypair APIs work correctly. +* While creating a server, keypair can be specified. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------- +Test Case 4 - List supported versions of the Compute API +-------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.test_versions.TestVersions.test_list_api_versions + +Test preconditions +------------------ + +* Compute versions extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Get a List of versioned endpoints in the SUT +* **Test assertion 1:** Verify endpoints versions start at 'v2.0' + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of listing all available APIs to API consumers. +Specifically, the test verifies that: + +* Compute list API versions API works correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------------- +Test Case 5 - Quotas management in Compute API +---------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.test_quotas.QuotasTestJSON.test_get_default_quotas +tempest.api.compute.test_quotas.QuotasTestJSON.test_get_quotas + +Test preconditions +------------------ + +* Compute quotas extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' +* Test action 1: Get the default quota set using the tenant ID +* **Test assertion 1:** Verify the default quota set ID matches tenant ID and + the default quota set is complete +* Test action 2: Get the quota set using the tenant ID +* **Test assertion 2:** Verify the quota set ID matches tenant ID and the quota + set is complete +* Test action 3: Get the quota set using the user ID +* **Test assertion 3:** Verify the quota set ID matches tenant ID and the quota + set is complete + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of getting quota set. +Specifically, the test verifies that: + +* User can get the default quota set for its tenant. +* User can get the quota set for its tenant. +* User can get the quota set using user ID. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------- +Test Case 6 - Basic server operations in the Compute API +-------------------------------------------------------- + +Test case specification +----------------------- + +This test case evaluates the Compute API ability of basic server operations, including: + +- Create a server with admin password +- Create a server with a name that already exists +- Create a server with a numeric name +- Create a server with a really long metadata +- Create a server with a name whose length exceeding 255 characters +- Create a server with an unknown flavor +- Create a server with an unknown image ID +- Create a server with an invalid network UUID +- Delete a server using a server ID that exceeds length limit +- Delete a server using a negative server ID +- Get a nonexistent server details +- Verify the instance host name is the same as the server name +- Create a server with an invalid access IPv6 address +- List all existent servers +- Filter the (detailed) list of servers by flavor, image, server name, server status or limit +- Lock a server and try server stop, unlock and retry +- Get and delete metadata from a server +- List and set metadata for a server +- Reboot, rebuild, stop and start a server +- Update a server's access addresses and server name + +The reference is, + +tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_server_with_admin_password +tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_with_existing_server_name +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_numeric_server_name +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_server_metadata_exceeds_length_limit +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_server_name_length_exceeds_256 +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_flavor +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_image +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_network_uuid +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_delete_server_pass_id_exceeding_length_limit +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_delete_server_pass_negative_id +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_get_non_existent_server tempest.api.compute.servers.test_create_server.ServersTestJSON.test_host_name_is_same_as_server_name +tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_host_name_is_same_as_server_name +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_invalid_ip_v6_address tempest.api.compute.servers.test_create_server.ServersTestJSON.test_list_servers tempest.api.compute.servers.test_create_server.ServersTestJSON.test_list_servers_with_detail -tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_created_server_vcpus -tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_server_details -tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_host_name_is_same_as_server_name tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_list_servers tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_list_servers_with_detail -tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_created_server_vcpus -tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_server_details -tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_get_instance_action -tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_list_instance_actions tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_flavor tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_image tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_server_name tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_server_status tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_limit_results -tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_active_status tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_flavor tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_image tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_limit @@ -72,41 +400,302 @@ tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJS tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_status_non_existing tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_with_a_deleted_server tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_lock_unlock_server -tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_reboot_server_hard -tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_rebuild_server -tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_stop_start_server tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_delete_server_metadata_item tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_get_server_metadata_item tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_list_server_metadata tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_set_server_metadata tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_set_server_metadata_item tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_update_server_metadata -tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_server_with_admin_password -tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_specify_keypair -tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_with_existing_server_name -tempest.api.compute.servers.test_servers.ServersTestJSON.test_update_access_server_address -tempest.api.compute.servers.test_servers.ServersTestJSON.test_update_server_name -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_numeric_server_name -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_server_metadata_exceeds_length_limit -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_server_name_length_exceeds_256 -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_flavor -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_image -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_network_uuid -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_delete_server_pass_id_exceeding_length_limit -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_delete_server_pass_negative_id -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_get_non_existent_server -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_invalid_ip_v6_address +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_server_name_blank +tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_reboot_server_hard tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_non_existent_server +tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_rebuild_server tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_deleted_server tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_non_existent_server -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_reboot_deleted_server -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_server_name_blank +tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_stop_start_server tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_stop_non_existent_server +tempest.api.compute.servers.test_servers.ServersTestJSON.test_update_access_server_address +tempest.api.compute.servers.test_servers.ServersTestJSON.test_update_server_name tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_update_name_of_non_existent_server tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_update_server_name_length_exceeds_256 tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_update_server_set_empty_name -tempest.api.compute.test_quotas.QuotasTestJSON.test_get_default_quotas -tempest.api.compute.test_quotas.QuotasTestJSON.test_get_quotas -tempest.api.compute.test_versions.TestVersions.test_list_api_versions +tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_created_server_vcpus +tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_server_details +tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_created_server_vcpus +tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_server_details + +tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_active_status +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_reboot_deleted_server + +Note: the last 2 test cases are the alias of another 2 test cases respectively, which are + +tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_server_status +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_deleted_server + +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of OpenStack. + +Test preconditions +------------------ + +* Compute quotas extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a server VM1 with a admin password 'testpassword' +* **Test assertion 1:** Verify the password returned in the response equals to 'testpassword' +* Test action 2: Generate a VM name VM_NAME +* Test action 3: Create 2 servers VM2 and VM3 both with name VM_NAME +* **Test assertion 2:** Verify VM2's ID is not equal to VM3's ID, and VM2's name equal to VM3's name +* Test action 4: Create a server VM4 with a numeric name '12345' +* **Test assertion 3:** Verify creating VM4 failed +* Test action 5: Create a server VM5 with a long metadata '{'a': 'b' * 260}' +* **Test assertion 4:** Verify creating VM5 failed +* Test action 6: Create a server VM6 with name length exceeding 255 characters +* **Test assertion 5:** Verify creating VM6 failed +* Test action 7: Create a server VM7 with an unknown flavor '-1' +* **Test assertion 6:** Verify creating VM7 failed +* Test action 8: Create a server VM8 with an unknown image ID '-1' +* **Test assertion 7:** Verify creating VM8 failed +* Test action 9: Create a server VM9 with an invalid network UUID 'a-b-c-d-e-f-g-h-i-j' +* **Test assertion 8:** Verify creating VM9 failed +* Test action 10: Delete a server using a server ID that exceeds system's max integer limit +* **Test assertion 9:** Verify deleting server failed +* Test action 11: Delete a server using a server ID '-1' +* **Test assertion 10:** Verify deleting server failed +* Test action 12: Get a nonexistent server by using a random generated server ID +* **Test assertion 11:** Verify get server failed +* Test action 13: SSH into a provided server and get server's hostname +* **Test assertion 12:** Verify server's host name is the same as the server name +* Test action 14: SSH into a provided server and get server's hostname (manual disk configuration) +* **Test assertion 13:** Verify server's host name is the same as the server name (manual disk configuration) +* Test action 15: Create a server with an invalid access IPv6 address +* **Test assertion 14:** Verify creating server failed, a bad request error is returned in response +* Test action 16: List all existent servers +* **Test assertion 15:** Verify a provided server is in the server list +* Test action 17: List all existent servers in detail +* **Test assertion 16:** Verify a provided server is in the detailed server list +* Test action 18: List all existent servers (manual disk configuration) +* **Test assertion 17:** Verify a provided server is in the server list (manual disk configuration) +* Test action 19: List all existent servers in detail (manual disk configuration) +* **Test assertion 18:** Verify a provided server is in the detailed server list (manual disk configuration) +* Test action 20: List all existent servers in detail and filter the server list by flavor +* **Test assertion 19:** Verify the filtered server list is correct +* Test action 21: List all existent servers in detail and filter the server list by image +* **Test assertion 20:** Verify the filtered server list is correct +* Test action 22: List all existent servers in detail and filter the server list by server name +* **Test assertion 21:** Verify the filtered server list is correct +* Test action 23: List all existent servers in detail and filter the server list by server status +* **Test assertion 22:** Verify the filtered server list is correct +* Test action 24: List all existent servers in detail and filter the server list by display limit '1' +* **Test assertion 23:** Verify the length of filtered server list is 1 +* Test action 25: List all existent servers and filter the server list by flavor +* **Test assertion 24:** Verify the filtered server list is correct +* Test action 26: List all existent servers and filter the server list by image +* **Test assertion 25:** Verify the filtered server list is correct +* Test action 27: List all existent servers and filter the server list by display limit '1' +* **Test assertion 26:** Verify the length of filtered server list is 1 +* Test action 28: List all existent servers and filter the server list by server name +* **Test assertion 27:** Verify the filtered server list is correct +* Test action 29: List all existent servers and filter the server list by server status +* **Test assertion 28:** Verify the filtered server list is correct +* Test action 30: List all existent servers and filter the server list by server name wildcard +* **Test assertion 29:** Verify the filtered server list is correct +* Test action 31: List all existent servers and filter the server list by part of server name +* **Test assertion 30:** Verify the filtered server list is correct +* Test action 32: List all existent servers and filter the server list by a future change-since date +* **Test assertion 31:** Verify the filtered server list is empty +* Test action 33: List all existent servers and filter the server list by a invalid change-since date format +* **Test assertion 32:** Verify a bad request error is returned in the response +* Test action 34: List all existent servers and filter the server list by display limit '1' +* **Test assertion 33:** Verify the length of filtered server list is 1 +* Test action 35: List all existent servers and filter the server list by a + display limit value greater than the length of the server list +* **Test assertion 34:** Verify the length of filtered server list equals to the length of server list +* Test action 36: List all existent servers and filter the server list by display limit '-1' +* **Test assertion 35:** Verify a bad request error is returned in the response +* Test action 37: List all existent servers and filter the server list by a string type limit value 'testing' +* **Test assertion 36:** Verify a bad request error is returned in the response +* Test action 38: List all existent servers and filter the server list by a nonexistent flavor +* **Test assertion 37:** Verify the filtered server list is empty +* Test action 39: List all existent servers and filter the server list by a nonexistent image +* **Test assertion 38:** Verify the filtered server list is empty +* Test action 40: List all existent servers and filter the server list by a nonexistent server name +* **Test assertion 39:** Verify the filtered server list is empty +* Test action 41: List all existent servers in detail and search the server list for a deleted server +* **Test assertion 40:** Verify the deleted server is not in the server list +* Test action 42: List all existent servers and filter the server list by a nonexistent server status +* **Test assertion 41:** Verify the filtered server list is empty +* Test action 43: List all existent servers in detail +* **Test assertion 42:** Verify a provided deleted server's id is not in the server list +* Test action 44: Lock a provided server VM10 and retrieve the server's status +* **Test assertion 43:** Verify VM10 is in 'ACTIVE' status +* Test action 45: Stop VM10 +* **Test assertion 44:** Verify stop VM10 failed +* Test action 46: Unlock VM10 and stop VM10 again +* **Test assertion 45:** Verify VM10 is stopped and in 'SHUTOFF' status +* Test action 47: Start VM10 +* **Test assertion 46:** Verify VM10 is in 'ACTIVE' status +* Test action 48: Delete metadata item 'key1' from a provided server +* **Test assertion 47:** Verify the metadata item is removed +* Test action 49: Get metadata item 'key2' from a provided server +* **Test assertion 48:** Verify the metadata item is correct +* Test action 50: List all metadata key/value pair for a provided server +* **Test assertion 49:** Verify all metadata are retrieved correctly +* Test action 51: Set metadata {'meta2': 'data2', 'meta3': 'data3'} for a provided server +* **Test assertion 50:** Verify server's metadata are replaced correctly +* Test action 52: Set metadata item nova's value to 'alt' for a provided server +* **Test assertion 51:** Verify server's metadata are set correctly +* Test action 53: Update metadata {'key1': 'alt1', 'key3': 'value3'} for a provided server +* **Test assertion 52:** Verify server's metadata are updated correctly +* Test action 54: Create a server with empty name parameter +* **Test assertion 53:** Verify create server failed +* Test action 55: Hard reboot a provided server +* **Test assertion 54:** Verify server is rebooted successfully +* Test action 56: Soft reboot a nonexistent server +* **Test assertion 55:** Verify reboot failed, an error is returned in the response +* Test action 57: Rebuild a provided server with new image, new server name and metadata +* **Test assertion 56:** Verify server is rebuilt successfully, server image, name and metadata are correct +* Test action 58: Create a server VM11 +* Test action 59: Delete VM11 and wait for VM11 to reach termination +* Test action 60: Rebuild VM11 with another image +* **Test assertion 57:** Verify rebuild server failed, an error is returned in the response +* Test action 61: Rebuild a nonexistent server +* **Test assertion 58:** Verify rebuild server failed, an error is returned in the response +* Test action 62: Stop a provided server +* **Test assertion 59:** Verify server reaches 'SHUTOFF' status +* Test action 63: Start the stopped server +* **Test assertion 60:** Verify server reaches 'ACTIVE' status +* Test action 64: Stop a provided server +* **Test assertion 61:** Verify stop server failed, an error is returned in the response +* Test action 65: Create a server VM12 and wait it to reach 'ACTIVE' status +* Test action 66: Update VM12's IPv4 and IPv6 access addresses +* **Test assertion 62:** Verify VM12's access addresses have been updated correctly +* Test action 67: Create a server VM13 and wait it to reach 'ACTIVE' status +* Test action 68: Update VM13's server name with non-ASCII characters '\u00CD\u00F1st\u00E1\u00F1c\u00E9' +* **Test assertion 63:** Verify VM13's server name has been updated correctly +* Test action 69: Update the server name of a nonexistent server +* **Test assertion 64:** Verify update server name failed, an 'object not found' error is returned in the response +* Test action 70: Update a provided server's name with a 256-character long name +* **Test assertion 65:** Verify update server name failed, a bad request is returned in the response +* Test action 71: Update a provided server's server name with an empty string +* **Test assertion 66:** Verify update server name failed, a bad request error is returned in the response +* Test action 72: Get the number of vcpus of a provided server +* Test action 73: Get the number of vcpus stated by the server's flavor +* **Test assertion 67:** Verify that the number of vcpus reported by the server + matches the amount stated by the server's flavor +* Test action 74: Create a server VM14 +* **Test assertion 68:** Verify VM14's server attributes are set correctly +* Test action 75: Get the number of vcpus of a provided server (manual disk configuration) +* Test action 76: Get the number of vcpus stated by the server's flavor (manual disk configuration) +* **Test assertion 69:** Verify that the number of vcpus reported by the server + matches the amount stated by the server's flavor (manual disk configuration) +* Test action 77: Create a server VM15 (manual disk configuration) +* **Test assertion 70:** Verify VM15's server attributes are set correctly (manual disk configuration) +* Test action 78: Delete all VMs created + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of basic server operations. +Specifically, the test verifies that: + +* If an admin password is provided on server creation, the server's root password should be set to that password +* Create a server with a name that already exists is allowed +* Create a server with a numeric name or a name that exceeds the length limit is not allowed +* Create a server with a metadata that exceeds the length limit is not allowed +* Create a server with an invalid flavor, an invalid image or an invalid network UUID is not allowed +* Delete a server with a server ID that exceeds the length limit or a nonexistent server ID is not allowed +* A provided server's host name is the same as the server name +* Create a server with an invalid IPv6 access address is not allowed +* A created server is in the (detailed) list of servers +* Filter the (detailed) list of servers by flavor, image, server name, server status, + and display limit, respectively. +* Filter the list of servers by a future date +* Filter the list of servers by an invalid date format, a negative display limit or a string type + display limit value is not allowed +* Filter the list of servers by a nonexistent flavor, image, server name or server status is not allowed +* Deleted servers are not in the list of servers +* Deleted servers do not show by default in list of servers +* Locked server is not allowed to be stopped by non-admin user +* Can get and delete metadata from servers +* Can list, set and update server metadata +* Create a server with name parameter empty is not allowed +* Hard reboot a server and the server should be power cycled +* Reboot, rebuild and stop a nonexistent server is not allowed +* Rebuild a server using the provided image and metadata +* Stop and restart a server +* A server's name and access addresses can be updated +* Update the name of a nonexistent server is not allowed +* Update name of a server to a name that exceeds the name length limit is not allowed +* Update name of a server to an empty string is not allowed +* The number of vcpus reported by the server matches the amount stated by the server's flavor +* The specified server attributes are set correctly + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +----------------------------------------------------------------- +Test Case 7 - Retrieve volume information through the Compute API +----------------------------------------------------------------- + +Test case specification +----------------------- + +This test case evaluates the Compute API ability of attaching volume to a +specific server and retrieve volume information, the reference is, + tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_attach_detach_volume tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_list_get_volume_attachments + +Test preconditions +------------------ + +* Compute volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a server VM1 and a volume VOL1 +* Test action 2: Attach VOL1 to VM1 +* **Test assertion 1:** Stop VM1 successfully and wait VM1 to reach 'SHUTOFF' status +* **Test assertion 2:** Start VM1 successfully and wait VM1 to reach 'ACTIVE' status +* **Test assertion 3:** SSH into VM1 and verify VOL1 is in VM1's root disk devices +* Test action 3: Detach VOL1 from VM1 +* **Test assertion 4:** Stop VM1 successfully and wait VM1 to reach 'SHUTOFF' status +* **Test assertion 5:** Start VM1 successfully and wait VM1 to reach 'ACTIVE' status +* **Test assertion 6:** SSH into VM1 and verify VOL1 is not in VM1's root disk devices +* Test action 4: Create a server VM2 and a volume VOL2 +* Test action 5: Attach VOL2 to VM2 +* Test action 6: List VM2's volume attachments +* **Test assertion 7:** Verify the length of the list is 1 and VOL2 attachment is in the list +* Test action 7: Retrieve VM2's volume information +* **Test assertion 8:** Verify volume information is correct +* Test action 8: Delete VM1, VM2, VOL1 and VOL2 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of retrieving volume information. +Specifically, the test verifies that: + +* Stop and start a server with an attached volume work correctly. +* Retrieve a server's volume information correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A diff --git a/docs/testing/user/testspecification/vimoperationsimage/index.rst b/docs/testing/user/testspecification/vimoperationsimage/index.rst index 932c7382..8e5f08ce 100644 --- a/docs/testing/user/testspecification/vimoperationsimage/index.rst +++ b/docs/testing/user/testspecification/vimoperationsimage/index.rst @@ -1,6 +1,6 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. (c) Ericsson AB +.. (c) Ericsson AB, Huawei Technologies Co.,Ltd ======================================= VIM image operations test specification @@ -9,17 +9,191 @@ VIM image operations test specification .. toctree:: :maxdepth: 2 -Each test case requires documentation according to: -* Use case specification -* Test preconditions -* Basic test flow execution descriptor -* Post conditions and pass fail criteria +Scope +===== + +The VIM image test area evaluates the ability of the system under test to support +VIM image operations. The test cases documented here are the Image API test cases +in the Openstack Interop guideline 2016.8 as implemented by the Refstack client. +These test cases will evaluate basic Openstack (as a VIM) image operations including +image creation, image list, image update and image deletion capabilities using Glance v2 API. + +References +========== + +- Defcore test cases + + - https://github.com/openstack/interop/blob/master/2016.08.json + +- Refstack client + + - https://github.com/openstack/refstack-client + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test area + +- API - Application Programming Interface +- CRUD - Create, Read, Update, and Delete +- NFVi - Network Functions Virtualization infrastructure +- VIM - Virtual Infrastructure Manager + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + +Test Area Structure +==================== + +The test area is structured based on VIM image operations. Each test case is able +to run independently, i.e. irrelevant of the state created by a previous test. + +For brevity, the test cases in this test area are summarized together based on +the operations they are testing. + +Test Descriptions +================= + +API Used and Reference +---------------------- + +Images: https://developer.openstack.org/api-ref/image/v2/ + +- create image +- delete image +- show image details +- show images +- show image schema +- show images schema +- upload binary image data +- add image tag +- delete image tag + +--------------------------------------- +Image get tests using the Glance v2 API +--------------------------------------- + +Test case specification +----------------------- -tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_delete_image -tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_update_image tempest.api.image.v2.test_images.ListImagesTest.test_get_image_schema tempest.api.image.v2.test_images.ListImagesTest.test_get_images_schema +tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_delete_deleted_image +tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_image_null_id +tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_non_existent_image + +tempest.api.image.v2.test_images.ListUserImagesTest.test_get_image_schema +tempest.api.image.v2.test_images.ListUserImagesTest.test_get_images_schema + +Note: the latter two test cases are the alias of the former first two, respectively. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +Glance is available. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create 6 images and store their ids in a created images list. +* Test action 2: Use image v2 API to show image schema and check the body of the response. +* **Test assertion 1:** In the body of the response, the value of the key 'name' is 'image'. +* Test action 3: Use image v2 API to show images schema and check the body of the response. +* **Test assertion 2:** In the body of the response, the value of the key 'name' is 'images'. +* Test action 4: Create an image with name 'test', container_formats 'bare' and + disk_formats 'raw'. Delete this image with its id and then try to show it with + its id. Delete this deleted image again with its id and check the API's response code. +* **Test assertion 3:** The operations of showing and deleting a deleted image with its id + both get 404 response code. +* Test action 5: Use a null image id to show a image and check the API's response code. +* **Test assertion 4:** The API's response code is 404. +* Test action 6: Generate a random uuid and use it as the image id to show the image. +* **Test assertion 5:** The API's response code is 404. +* Test action 7: Delete the 6 images with the stored ids. Show all images and check + whether the 6 images' ids are not in the show list. +* **Test assertion 6:** The 6 images' ids are not found in the show list. + +Pass / fail criteria +'''''''''''''''''''' + +The first two test cases evaluate the ability to use Glance v2 API to show image +and images schema. The latter three test cases evaluate the ability to use Glance +v2 API to show images with a deleted image id, a null image id and a non-existing +image id. Specifically it verifies that: + +* Glance image get API can show the image and images schema. +* Glance image get API can't show an image with a deleted image id. +* Glance image get API can't show an image with a null image id. +* Glance image get API can't show an image with a non-existing image id. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +None + +-------------------------------------- +CRUD image operations in Images API v2 +-------------------------------------- + +Test case specification +----------------------- + +tempest.api.image.v2.test_images.ListImagesTest.test_list_no_params + tempest.api.image.v2.test_images.ListImagesTest.test_index_no_params +tempest.api.image.v2.test_images.ListUserImagesTest.test_list_no_params + +Note: the latter two test cases are the alias of the former one. Alias should +always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +Glance is available. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create 6 images and store their ids in a created images list. +* Test action 2: List all images and check whether the ids listed are in the created images list. +* **Test assertion 1:** The ids get from the list images API are in the created images list. + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the ability to use Glance v2 API to list images. +Specifically it verifies that: + +* Glance image API can show the images. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +None + +---------------------------------------- +Image list tests using the Glance v2 API +---------------------------------------- + +Test case specification +----------------------- + tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_container_format tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_disk_format tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_limit @@ -27,9 +201,7 @@ tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_min_max_s tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_size tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_status tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_visibility -tempest.api.image.v2.test_images.ListImagesTest.test_list_no_params -tempest.api.image.v2.test_images.ListUserImagesTest.test_get_image_schema -tempest.api.image.v2.test_images.ListUserImagesTest.test_get_images_schema + tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_container_format tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_disk_format tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_limit @@ -37,12 +209,198 @@ tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_min_m tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_size tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_status tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_visibility -tempest.api.image.v2.test_images.ListUserImagesTest.test_list_no_params + +Note: the latter 7 test cases are the alias of the former 7, respectively. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +Glance is available. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create 6 images with a random size ranging from 1024 to 4096 and + visibility 'private'; set their (container_format, disk_format) pair to be + (ami, ami), (ami, ari), (ami, aki), (ami, vhd), (ami, vmdk) and (ami, raw); + store their ids in a list and upload the binary images data. +* Test action 2: Use Glance v2 API to list all images whose container_format is 'ami' + and store the response details in a list. +* **Test assertion 1:** The list is not empty and all the values of container_format + in the list are 'ami'. +* Test action 3: Use Glance v2 API to list all images whose disk_format is 'raw' + and store the response details in a list. +* **Test assertion 2:** The list is not empty and all the values of disk_format + in the list are 'raw'. +* Test action 4: Use Glance v2 API to list one image by setting limit to be 1 and + store the response details in a list. +* **Test assertion 3:** The length of the list is one. +* Test action 5: Use Glance v2 API to list images by setting size_min and size_max, + and store the response images' sizes in a list. Choose the first image's size as + the median, size_min is median-500 and size_max is median+500. +* **Test assertion 4:** All sizes in the list are no less than size_min and no more + than size_max. +* Test action 6: Use Glance v2 API to show the first created image with its id and + get its size from the response. Use Glance v2 API to list images whose size is equal + to this size and store the response details in a list. +* **Test assertion 5:** All sizes of the images in the list are equal to the size + used to list the images. +* Test action 7: Use Glance v2 API to list the images whose status is active and + store the response details in a list. +* **Test assertion 6:** All status of images in the list are active. +* Test action 8: Use Glance v2 API to list the images whose visibility is private and + store the response details in a list. +* **Test assertion 7:** All images' values of visibility in the list are private. +* Test action 9: Delete the 6 images with the stored ids. Show images and check whether + the 6 ids are not in the show list. +* **Test assertion 8:** The stored 6 ids are not found in the show list. + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the ability to use Glance v2 API to list images with +different parameters. Specifically it verifies that: + +* Glance image API can show the images with the container_format. +* Glance image API can show the images with the disk_format. +* Glance image API can show the images by setting a limit number. +* Glance image API can show the images with the size_min and size_max. +* Glance image API can show the images with the size. +* Glance image API can show the images with the status. +* Glance image API can show the images with the visibility type. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +None + +------------------------------------------ +Image update tests using the Glance v2 API +------------------------------------------ + +Test case specification +----------------------- + +tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_update_image +tempest.api.image.v2.test_images_tags.ImagesTagsTest.test_update_delete_tags_for_image +tempest.api.image.v2.test_images_tags_negative.ImagesTagsNegativeTest.test_update_tags_for_non_existing_image + +Test preconditions +------------------ + +Glance is available. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create an image with container_formats 'ami', disk_formats 'ami' + and visibility 'private' and store its id returned in the response. Check whether + the status of the created image is 'queued'. +* **Test assertion 1:** The status of the created image is 'queued'. +* Test action 2: Use the stored image id to upload the binary image data and update + this image's name. Show this image with the stored id. Check if the stored id and + name used to update the image are equal to the id and name in the show list. +* **Test assertion 2:** The id and name returned in the show list are equal to + the stored id and name used to update the image. +* Test action 3: Create an image with container_formats 'bare', disk_formats 'raw' + and visibility 'private' and store its id returned in the response. +* Test action 4: Use the stored id to add a tag. Show the image with the stored id + and check if the tag used to add is in the image's tags returned in the show list. +* **Test assertion 3:** The tag used to add into the image is in the show list. +* Test action 5: Use the stored id to delete this tag. Show the image with the + stored id and check if the tag used to delete is not in the show list. +* **Test assertion 4:** The tag used to delete from the image is not in the show list. +* Test action 6: Generate a random uuid as the image id. Use the image id to add a tag + into the image's tags. +* **Test assertion 5:** The API's response code is 404. +* Test action 7: Delete the images created in test action 1 and 3. Show the images + and check whether the ids are not in the show list. +* **Test assertion 6:** The two ids are not found in the show list. + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the ability to use Glance v2 API to update images with +different parameters. Specifically it verifies that: + +* Glance image API can update image's name with the existing image id. +* Glance image API can update image's tags with the existing image id. +* Glance image API can't update image's tags with a non-existing image id. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +None + +-------------------------------------------- +Image deletion tests using the Glance v2 API +-------------------------------------------- + +Test case specification +----------------------- + +tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_delete_image tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_delete_image_null_id tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_delete_non_existing_image -tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_delete_deleted_image -tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_image_null_id -tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_non_existent_image -tempest.api.image.v2.test_images_tags.ImagesTagsTest.test_update_delete_tags_for_image tempest.api.image.v2.test_images_tags_negative.ImagesTagsNegativeTest.test_delete_non_existing_tag -tempest.api.image.v2.test_images_tags_negative.ImagesTagsNegativeTest.test_update_tags_for_non_existing_image + +Test preconditions +------------------ + +Glance is available. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create an image with container_formats 'ami', disk_formats 'ami' + and visibility 'private'. Use the id of the created image to delete the image. + List all images and check whether this id is in the list. +* **Test assertion 1:** The id of the created image is not found in the list + of all images after the deletion operation. +* Test action 2: Delete images with a null id and check the API's response code. +* **Test assertion 2:** The API's response code is 404. +* Test action 3: Generate a random uuid and delete images with this uuid as image id. + Check the API's response code. +* **Test assertion 3:** The API's response code is 404. +* Test action 4: Create an image with container_formats 'bare', disk_formats 'raw' + and visibility 'private'. Delete this image's tag with the image id and a random tag + Check the API's response code. +* **Test assertion 4:** The API's response code is 404. +* Test action 5: Delete the images created in test action 1 and 4. List all images + and check whether the ids are in the list. +* **Test assertion 5:** The two ids are not found in the list. + +Pass / fail criteria +'''''''''''''''''''' + +The first three test cases evaluate the ability to use Glance v2 API to delete images +with an existing image id, a null image id and a non-existing image id. The last one +evaluates the ability to use the API to delete a non-existing image tag. +Specifically it verifies that: + +* Glance image deletion API can delete the image with an existing id. +* Glance image deletion API can't delete an image with a null image id. +* Glance image deletion API can't delete an image with a non-existing image id. +* Glance image deletion API can't delete an image tag with a non-existing image tag. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +None diff --git a/docs/testing/user/testspecification/vimoperationsnetwork/index.rst b/docs/testing/user/testspecification/vimoperationsnetwork/index.rst index 252513ee..47c08ef2 100644 --- a/docs/testing/user/testspecification/vimoperationsnetwork/index.rst +++ b/docs/testing/user/testspecification/vimoperationsnetwork/index.rst @@ -1,6 +1,6 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. (c) Ericsson AB +.. (c) Ericsson AB, Huawei Technologies Co.,Ltd ========================================= VIM network operations test specification @@ -9,19 +9,105 @@ VIM network operations test specification .. toctree:: :maxdepth: 2 -Each test case requires documentation according to: -* Use case specification -* Test preconditions -* Basic test flow execution descriptor -* Post conditions and pass fail criteria +Scope +===== + +The VIM network test area evaluates the ability of the system under test to support +VIM network operations. The test cases documented here are the network API test cases +in the Openstack Interop guideline 2016.8 as implemented by the Refstack client. +These test cases will evaluate basic Openstack (as a VIM) network operations including +basic CRUD operations on L2 networks, L2 network ports and security groups. + +References +========== + +- Defcore test cases + + - https://github.com/openstack/interop/blob/master/2016.08.json + +- Refstack client + + - https://github.com/openstack/refstack-client + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test area + +- API - Application Programming Interface +- CRUD - Create, Read, Update and Delete +- NFVi - Network Functions Virtualization infrastructure +- VIM - Virtual Infrastructure Manager + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + +Test Area Structure +==================== + +The test area is structured based on VIM network operations. Each test case is able +to run independently, i.e. irrelevant of the state created by a previous test. +Specifically, every test performs clean-up operations which return the system to +the same state as before the test. + +For brevity, the test cases in this test area are summarized together based on +the operations they are testing. + +Test Descriptions +================= + +API Used and Reference +---------------------- + +Network: http://developer.openstack.org/api-ref/networking/v2/index.html + +- create network +- update network +- list networks +- show network details +- delete network + +- create subnet +- update subnet +- list subnets +- show subnet details +- delete subnet + +- create port +- bulk create ports +- update port +- list ports +- show port details +- delete port + +- create security group +- update security group +- list security groups +- show security group +- delete security group + +- create security group rule +- list security group rules +- show security group rule +- delete security group rule + +--------------------------------------------------------- +Basic CRUD operations on L2 networks and L2 network ports +--------------------------------------------------------- + +Test case specification +----------------------- -tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_all_attributes tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_with_allocation_pools tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_with_dhcp_enabled tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_with_gw tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_with_gw_and_allocation_pools tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_with_host_routes_and_dns_nameservers tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_without_gateway +tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_all_attributes tempest.api.network.test_networks.NetworksTest.test_create_update_delete_network_subnet tempest.api.network.test_networks.NetworksTest.test_delete_network_with_subnet tempest.api.network.test_networks.NetworksTest.test_list_networks @@ -33,13 +119,23 @@ tempest.api.network.test_networks.NetworksTest.test_show_network_fields tempest.api.network.test_networks.NetworksTest.test_show_subnet tempest.api.network.test_networks.NetworksTest.test_show_subnet_fields tempest.api.network.test_networks.NetworksTest.test_update_subnet_gw_dns_host_routes_dhcp -tempest.api.network.test_networks.NetworksTestJSON.test_create_delete_subnet_all_attributes +tempest.api.network.test_ports.PortsTestJSON.test_create_bulk_port +tempest.api.network.test_ports.PortsTestJSON.test_create_port_in_allowed_allocation_pools +tempest.api.network.test_ports.PortsTestJSON.test_create_update_delete_port +tempest.api.network.test_ports.PortsTestJSON.test_list_ports +tempest.api.network.test_ports.PortsTestJSON.test_list_ports_fields +tempest.api.network.test_ports.PortsTestJSON.test_show_port +tempest.api.network.test_ports.PortsTestJSON.test_show_port_fields +tempest.api.network.test_ports.PortsTestJSON.test_update_port_with_security_group_and_extra_attributes +tempest.api.network.test_ports.PortsTestJSON.test_update_port_with_two_security_groups_and_extra_attributes + tempest.api.network.test_networks.NetworksTestJSON.test_create_delete_subnet_with_allocation_pools tempest.api.network.test_networks.NetworksTestJSON.test_create_delete_subnet_with_dhcp_enabled tempest.api.network.test_networks.NetworksTestJSON.test_create_delete_subnet_with_gw tempest.api.network.test_networks.NetworksTestJSON.test_create_delete_subnet_with_gw_and_allocation_pools tempest.api.network.test_networks.NetworksTestJSON.test_create_delete_subnet_with_host_routes_and_dns_nameservers tempest.api.network.test_networks.NetworksTestJSON.test_create_delete_subnet_without_gateway +tempest.api.network.test_networks.NetworksTestJSON.test_create_delete_subnet_all_attributes tempest.api.network.test_networks.NetworksTestJSON.test_create_update_delete_network_subnet tempest.api.network.test_networks.NetworksTestJSON.test_delete_network_with_subnet tempest.api.network.test_networks.NetworksTestJSON.test_list_networks @@ -51,15 +147,143 @@ tempest.api.network.test_networks.NetworksTestJSON.test_show_network_fields tempest.api.network.test_networks.NetworksTestJSON.test_show_subnet tempest.api.network.test_networks.NetworksTestJSON.test_show_subnet_fields tempest.api.network.test_networks.NetworksTestJSON.test_update_subnet_gw_dns_host_routes_dhcp -tempest.api.network.test_ports.PortsTestJSON.test_create_bulk_port -tempest.api.network.test_ports.PortsTestJSON.test_create_port_in_allowed_allocation_pools -tempest.api.network.test_ports.PortsTestJSON.test_create_update_delete_port -tempest.api.network.test_ports.PortsTestJSON.test_list_ports -tempest.api.network.test_ports.PortsTestJSON.test_list_ports_fields -tempest.api.network.test_ports.PortsTestJSON.test_show_port -tempest.api.network.test_ports.PortsTestJSON.test_show_port_fields -tempest.api.network.test_ports.PortsTestJSON.test_update_port_with_security_group_and_extra_attributes -tempest.api.network.test_ports.PortsTestJSON.test_update_port_with_two_security_groups_and_extra_attributes + +Note: the latter 18 test cases are the alias of the former first 18, respectively. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +Neutron is available. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a network and create a subnet of this network by setting + allocation_pools, then check the details of the subnet and delete the subnet and network +* **Test assertion 1:** The allocation_pools returned in the response equals to the one used + to create the subnet, and the network and subnet ids are not found after deletion +* Test action 2: Create a network and create a subnet of this network by setting + enable_dhcp "True", then check the details of the subnet and delete the subnet and network +* **Test assertion 2:** The enable_dhcp returned in the response is "True" and the network + and subnet ids are not found after deletion +* Test action 3: Create a network and create a subnet of this network by setting + gateway_ip, then check the details of the subnet and delete the subnet and network +* **Test assertion 3:** The gateway_ip returned in the response equals to the one used to + create the subnet, and the network and subnet ids are not found after deletion +* Test action 4: Create a network and create a subnet of this network by setting allocation_pools + and gateway_ip, then check the details of the subnet and delete the subnet and network +* **Test assertion 4:** The allocation_pools and gateway_ip returned in the response equal to + the ones used to create the subnet, and the network and subnet ids are not found after deletion +* Test action 5: Create a network and create a subnet of this network by setting host_routes and + dns_nameservers, then check the details of the subnet and delete the subnet and network +* **Test assertion 5:** The host_routes and dns_nameservers returned in the response equal to + the ones used to create the subnet, and the network and subnet ids are not found after deletion +* Test action 6: Create a network and create a subnet of this network without setting + gateway_ip, then delete the subnet and network +* **Test assertion 6:** The network and subnet ids are not found after deletion +* Test action 7: Create a network and create a subnet of this network by setting enable_dhcp "true", + gateway_ip, ip_version, cidr, host_routes, allocation_pools and dns_nameservers, + then check the details of the subnet and delete the subnet and network +* **Test assertion 7:** The values returned in the response equal to the ones used to + create the subnet, and the network and subnet ids are not found after deletion +* Test action 8: Create a network and update this network's name, then create a subnet and update + this subnet's name, delete the subnet and network +* **Test assertion 8:** The network's status and subnet's status are both 'ACTIVE' after creation, + their names equal to the new names used to update, and the network and subnet ids are not + found after deletion +* Test action 9: Create a network and create a subnet of this network, then delete this network +* **Test assertion 9:** The subnet has also been deleted after deleting the network +* Test action 10: Create a network and list all networks +* **Test assertion 10:** The network created is found in the list +* Test action 11: Create a network and list networks with the id and name of the created network +* **Test assertion 11:** The id and name of the list network equal to the created network's id and name +* Test action 12: Create a network and create a subnet of this network, then list all subnets +* **Test assertion 12:** The subnet created is found in the list +* Test action 13: Create a network and create a subnet of this network, then list subnets with + the id and network_id of the created subnet +* **Test assertion 13:** The id and network_id of the list subnet equal to the created subnet +* Test action 14: Create a network and show network's details with the id of the created network +* **Test assertion 14:** The id and name returned in the response equal to the created network's id and name +* Test action 15: Create a network and just show network's id and name info with the id of the created network +* **Test assertion 15:** The keys returned in the response are only id and name, and the values + of all the keys equal to network's id and name +* Test action 16: Create a network and create a subnet of this network, then show subnet's details + with the id of the created subnet +* **Test assertion 16:** The id and cidr info returned in the response equal to the created + subnet's id and cidr +* Test action 17: Create a network and create a subnet of this network, then show subnet's id and + network_id info with the id of the created subnet +* **Test assertion 17:** The keys returned in the response are just id and network_id, and the values + of all the keys equal to subnet's id and network_id +* Test action 18: Create a network and create a subnet of this network, then update subnet's + name, host_routes, dns_nameservers and gateway_ip +* **Test assertion 18:** The name, host_routes, dns_nameservers and gateway_ip returned in the + response equal to the values used to update the subnet +* Test action 19: Create 2 networks and bulk create 2 ports with the ids of the created networks +* **Test assertion 19:** The network_id of each port equals to the one used to create the port and + the admin_state_up of each port is True +* Test action 20: Create a network and create a subnet of this network by setting allocation_pools, + then create a port with the created network's id +* **Test assertion 20:** The ip_address of the created port is in the range of the allocation_pools +* Test action 21: Create a network and create a port with its id, then update the port's name and + set its admin_state_up to be False +* **Test assertion 21:** The name returned in the response equals to the name used to update + the port and the port's admin_state_up is False +* Test action 22: Create a network and create a port with its id, then list all ports +* **Test assertion 22:** The created port is found in the list +* Test action 23: Create a network and create a port with its id, then list ports with the id + and mac_address of the created port +* **Test assertion 23:** The created port is found in the list +* Test action 24: Create a network and create a port with its id, then show the port's details +* **Test assertion 24:** The key 'id' is in the details +* Test action 25: Create a network and create a port with its id, then show the port's id + and mac_address info with the port's id +* **Test assertion 25:** The keys returned in the response are just id and mac_address, + and the values of all the keys equal to port's id and mac_address +* Test action 26: Create a network, 2 subnets (SUBNET1 and SUBNET2) and 2 security groups + (SG1 and SG2), create a port with SG1 and SUBNET1, then update the port's security group to SG2 + and its subnet_id to SUBNET2 +* **Test assertion 26:** The port's subnet_id equals to SUBNET2's id and its security_group_ids + equals to SG2's id +* Test action 27: Create a network, 2 subnets (SUBNET1 and SUBNET2) and 3 security groups + (SG1, SG2 and SG3), create a port with SG1 and SUBNET1, then update the port's security group to + SG2 and SG3 and its subnet_id to SUBNET2 +* **Test assertion 27:** The port's subnet_id equal to SUBNET2's id and its security_group_ids + equals to the ids of SG2 and SG3 + +Pass / fail criteria +'''''''''''''''''''' + +These test cases evaluate the ability of basic CRUD operations on L2 networks and L2 network ports. +Specifically it verifies that: + +* Subnets can be created successfully by setting different parameters. +* Subnets can be updated after being created. +* Ports can be bulk created with network ids. +* Port's security group(s) can be updated after being created. +* Networks/subnets/ports can be listed with their ids and other parameters. +* All details or special fields' info of networks/subnets/ports can be shown with their ids. +* Networks/subnets/ports can be successfully deleted. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------- +Basic CRUD operations on security groups +---------------------------------------- + +Test case specification +----------------------- + tempest.api.network.test_security_groups.SecGroupTest.test_create_list_update_show_delete_security_group tempest.api.network.test_security_groups.SecGroupTest.test_create_security_group_rule_with_additional_args tempest.api.network.test_security_groups.SecGroupTest.test_create_security_group_rule_with_icmp_type_code @@ -79,3 +303,86 @@ tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_crea tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_delete_non_existent_security_group tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_show_non_existent_security_group tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_show_non_existent_security_group_rule + +Test preconditions +------------------ + +Neutron is available. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a security group SG1, list all security groups, update the name and description + of SG1, show details of SG1 and delete SG1 +* **Test assertion 1:** SG1 is in the list, the name and description of SG1 equal to the ones used to + update it, the name and description of SG1 shown in the details equal to the ones used to update it, + and SG1's id is not found after deletion +* Test action 2: Create a security group SG1, and create a rule with protocol 'tcp', + port_range_min and port_range_max +* **Test assertion 2:** The values returned in the response equal to the ones used to create the rule +* Test action 3: Create a security group SG1, and create a rule with protocol 'icmp' and icmp_type_codes +* **Test assertion 3:** The values returned in the response equal to the ones used to create the rule +* Test action 4: Create a security group SG1, and create a rule with protocol '17' +* **Test assertion 4:** The values returned in the response equal to the ones used to create the rule +* Test action 5: Create a security group SG1, and create a rule with protocol 'udp', port_range_min, + port_range_max and remote_group_id +* **Test assertion 5:** The values returned in the response equal to the ones used to create the rule +* Test action 6: Create a security group SG1, and create a rule with protocol 'tcp', port_range_min, + port_range_max and remote_ip_prefix +* **Test assertion 6:** The values returned in the response equal to the ones used to create the rule +* Test action 7: Create a security group SG1, create 3 rules with protocol 'tcp', 'udp' and 'icmp' + respectively, show details of each rule, list all rules and delete all rules +* **Test assertion 7:** The values in the shown details equal to the ones used to create the rule, + all rules are found in the list, and all rules are not found after deletion +* Test action 8: List all security groups +* **Test assertion 8:** There is one default security group in the list +* Test action 9: Create a security group whose name is 'default' +* **Test assertion 9:** Failed to create this security group because of name conflict +* Test action 10: Create a security group SG1, create a rule with protocol 'tcp', port_range_min + and port_range_max, and create another tcp rule with the same parameters +* **Test assertion 10:** Failed to create this security group rule because of duplicate protocol +* Test action 11: Create a security group SG1, and create a rule with ethertype 'bad_ethertype' +* **Test assertion 11:** Failed to create this security group rule because of bad ethertype +* Test action 12: Create a security group SG1, and create a rule with protocol 'bad_protocol_name' +* **Test assertion 12:** Failed to create this security group rule because of bad protocol +* Test action 13: Create a security group SG1, and create a rule with remote_ip_prefix '92.168.1./24', + '192.168.1.1/33', 'bad_prefix' and '256' respectively +* **Test assertion 13:** Failed to create these security group rules because of bad remote_ip_prefix +* Test action 14: Create a security group SG1, and create a tcp rule with (port_range_min, port_range_max) + (-16, 80), (80, 79), (80, 65536), (None, 6) and (-16, 65536) respectively +* **Test assertion 14:** Failed to create these security group rules because of bad ports +* Test action 15: Create a security group SG1, and create a tcp rule with remote_group_id 'bad_group_id' + and a random uuid respectively +* **Test assertion 15:** Failed to create these security group rules because of nonexistent remote_group_id +* Test action 16: Create a security group SG1, and create a rule with a random uuid as security_group_id +* **Test assertion 16:** Failed to create these security group rules because of nonexistent security_group_id +* Test action 17: Generate a random uuid and use this id to delete security group +* **Test assertion 17:** Failed to delete security group because of nonexistent security_group_id +* Test action 18: Generate a random uuid and use this id to show security group +* **Test assertion 18:** Failed to show security group because of nonexistent id of security group +* Test action 19: Generate a random uuid and use this id to show security group rule +* **Test assertion 19:** Failed to show security group rule because of nonexistent id of security group rule + +Pass / fail criteria +'''''''''''''''''''' + +These test cases evaluate the ability of Basic CRUD operations on security groups and security group rules. +Specifically it verifies that: + +* Security groups can be created, list, updated, shown and deleted. +* Security group rules can be created with different parameters, list, shown and deleted. +* Cannot create an additional default security group. +* Cannot create a duplicate security group rules. +* Cannot create security group rules with bad ethertype, protocol, remote_ip_prefix, ports, + remote_group_id and security_group_id. +* Cannot show or delete security groups or security group rules with nonexistent ids. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A diff --git a/docs/testing/user/testspecification/vimoperationsvolume/index.rst b/docs/testing/user/testspecification/vimoperationsvolume/index.rst index ce039a5b..9a76c37c 100644 --- a/docs/testing/user/testspecification/vimoperationsvolume/index.rst +++ b/docs/testing/user/testspecification/vimoperationsvolume/index.rst @@ -1,36 +1,512 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. (c) Ericsson AB +.. (c) Ericsson AB, Huawei Technologies Co.,Ltd -======================================== +========================================= VIM volume operations test specification -======================================== +========================================= .. toctree:: :maxdepth: 2 -Each test case requires documentation according to: -* Use case specification -* Test preconditions -* Basic test flow execution descriptor -* Post conditions and pass fail criteria +Scope +===== + +The VIM volume operations test area evaluates the ability of the system under +test to support VIM volume operations. The test cases documented here are the +volume API test cases in the OpenStack Interop guideline 2016.8 as implemented +by the RefStack client. These test cases will evaluate basic OpenStack (as a VIM) +volume operations, including: + +- Volume attach and detach operations +- Volume service availability zone operations +- Volume cloning operations +- Image copy-to-volume operations +- Volume creation and deletion operations +- Volume service extension listing +- Volume metadata operations +- Volume snapshot operations + +References +================ + +- OpenStack Governance/Interop + + - https://wiki.openstack.org/wiki/Governance/InteropWG + +- Defcore test cases + + - https://github.com/openstack/interop/blob/master/2016.08.json + +- Refstack client + + - https://github.com/openstack/refstack-client + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test area + +- API - Application Programming Interface +- NFVi - Network Functions Virtualization infrastructure +- SUT - System Under Test +- VIM - Virtual Infrastructure Manager +- VM - Virtual Machine + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVI and VIM deployed with a Pharos compliant infrastructure. + +Test Area Structure +==================== + +The test area is structured based on VIM volume API operations. Each test case is +able to run independently, i.e. irrelevant of the state created by a previous test. +Specifically, every test performs clean-up operations which return the system to +the same state as before the test. + +For brevity, the test cases in this test area are summarized together based on +the operations they are testing. + +Test Descriptions +================= + +API Used and Reference +---------------------- + +Block storage: https://developer.openstack.org/api-ref/block-storage + +- create volume +- delete volume +- update volume +- attach volume to server +- detach volume from server +- create volume metadata +- update volume metadata +- delete volume metadata +- list volume + +- create snapshot +- update snapshot +- delete snapshot + +------------------------------------------------------------------------ +Test Case 1 - Volume attach and detach operations with the Cinder v2 API +------------------------------------------------------------------------ + +Test case specification +----------------------- -tempest.api.volume.test_availability_zone.AvailabilityZoneV2TestJSON.test_get_availability_zone_list -tempest.api.volume.test_extensions.ExtensionsV2TestJSON.test_list_extensions -tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_create_get_delete_snapshot_metadata -tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_crud_snapshot_metadata -tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_update_snapshot_metadata_item -tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_create_get_delete_volume_metadata -tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_crud_volume_metadata -tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_update_volume_metadata_item tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_attach_detach_volume_to_instance tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_get_volume_attachment -tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_reserve_unreserve_volume -tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_volume_bootable -tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_volume_readonly_update -tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_attach_volumes_with_nonexistent_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_detach_volumes_with_invalid_volume_id + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' +* Test action 1: Create a server VM1 +* Test action 2: Attach a provided VOL1 to VM1 +* **Test assertion 1:** Verify VOL1 is in 'in-use' status +* Test action 3: Detach VOL1 from VM1 +* **Test assertion 2:** Verify VOL1 is in 'available' status +* Test action 4: Create a server VM2 +* Test action 5: Attach a provided VOL2 to VM2 and wait for VOL2 to reach 'in-use' status +* Test action 6: Retrieve VOL2's attachment information ATTCH_INFO +* **Test assertion 3:** Verify ATTCH_INFO is correct +* Test action 7: Create a server VM3 and wait for VM3 to reach 'ACTIVE' status +* Test action 8: Attach a non-existent volume to VM3 +* **Test assertion 4:** Verify attach volume failed, a 'NOT FOUND' error is returned in the response +* Test action 9: Detach a volume from a server by using an invalid volume ID +* **Test assertion 5:** Verify detach volume failed, a 'NOT FOUND' error is returned in the response + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the volume API ability of attaching a volume to a server +and detaching a volume from a server. Specifically, the test verifies that: + +* Volumes can be attached and detached from servers. +* Volume attachment information can be retrieved. +* Attach and detach a volume using an invalid volume ID is not allowed. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------------------------------- +Test Case 2 - Volume service availability zone operations with the Cinder v2 API +-------------------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_availability_zone.AvailabilityZoneV2TestJSON.test_get_availability_zone_list + +tempest.api.volume.test_availability_zone.AvailabilityZoneTestJSON.test_get_availability_zone_list + +Note: the second test case is the alias of the first one. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' +* Test action 1: List all existent availability zones +* **Test assertion 1:** Verify the availability zone list length is greater than 0 + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of listing availability zones. +Specifically, the test verifies that: + +* Availability zones can be listed. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------------- +Test Case 3 - Volume cloning operations with the Cinder v2 API +-------------------------------------------------------------- + +Test case specification +----------------------- + tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete_as_clone + +tempest.api.volume.test_volumes_get.VolumesGetTest.test_volume_create_get_update_delete_as_clone + +Note: the second test case is the alias of the first one. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API +* Cinder volume clones feature is enabled + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' +* Test action 1: Create a volume VOL1 +* Test action 2: Create a volume VOL2 from source volume VOL1 with a specific name and metadata +* Test action 2: Wait for VOL2 to reach 'available' status +* **Test assertion 1:** Verify the name of VOL2 is correct +* Test action 3: Retrieve VOL2's detail information +* **Test assertion 2:** Verify the retrieved volume name, ID and metadata are the same as VOL2 +* **Test assertion 3:** Verify VOL2's bootable flag is 'False' +* Test action 4: Update the name of VOL2 with the original value +* Test action 5: Update the name of VOL2 with a new value +* **Test assertion 4:** Verify the name of VOL2 is updated successfully +* Test action 6: Create a volume VOL3 with no name specified and a description contains characters '@#$%^*' +* **Test assertion 5:** Verify VOL3 is created successfully +* Test action 7: Update the name of VOL3 and description with the original value +* **Test assertion 6:** Verify VOL3's bootable flag is 'False' + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of creating a cloned volume from a source volume, +getting cloned volume detail information and updating cloned volume attributes. + +Specifically, the test verifies that: + +* Cloned volume can be created from a source volume. +* Cloned volume detail information can be retrieved. +* Cloned volume detail information can be updated. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------------------- +Test Case 4 - Image copy-to-volume operations with the Cinder v2 API +-------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_volume_bootable tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete_from_image + +tempest.api.volume.test_volumes_get.VolumesActionsTest.test_volume_bootable +tempest.api.volume.test_volumes_get.VolumesGetTest.test_volume_create_get_update_delete_from_image + +Note: the last 2 test cases are the alias of the former 2. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' +* Test action 1: Set a provided volume VOL1's bootable flag to 'True' +* Test action 2: Retrieve VOL1's bootable flag +* **Test assertion 1:** Verify VOL1's bootable flag is 'True' +* Test action 3: Set a provided volume VOL1's bootable flag to 'False' +* Test action 4: Retrieve VOL1's bootable flag +* **Test assertion 2:** Verify VOL1's bootable flag is 'False' +* Test action 5: Create a bootable volume VOL2 from one image with a specific name and metadata +* Test action 6: Wait for VOL2 to reach 'available' status +* **Test assertion 3:** Verify the name of VOL2 name is correct +* Test action 7: Retrieve VOL2's information +* **Test assertion 4:** Verify the retrieved volume name, ID and metadata are the same as VOL2 +* **Test assertion 5:** Verify VOL2's bootable flag is 'True' +* Test action 8: Update the name of VOL2 with the original value +* Test action 9: Update the name of VOL2 with a new value +* **Test assertion 6:** Verify the name of VOL2 is updated successfully +* Test action 10: Create a volume VOL3 with no name specified and a description contains characters '@#$%^*' +* **Test assertion 7:** Verify VOL3 is created successfully +* Test action 11: Update the name of VOL3 and description with the original value +* **Test assertion 8:** Verify VOL3's bootable flag is 'True' + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of updating volume's bootable flag and creating +a bootable volume from an image, getting bootable volume detail information and updating bootable volume. + +Specifically, the test verifies that: + +* Volume bootable flag can be set and retrieved. +* Bootable volume can be created from a source volume. +* Bootable volume detail information can be retrieved. +* Bootable volume detail information can be updated. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------------------------------------------- +Test Case 5 - Volume creation and deletion operations with the Cinder v2 API +---------------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_invalid_size +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_source_volid +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_volume_type +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_out_passing_size +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_size_negative +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_size_zero + +tempest.api.volume.test_volumes_get.VolumesGetTest.test_volume_create_get_update_delete +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_invalid_size +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_nonexistent_source_volid +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_nonexistent_volume_type + +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_without_passing_size +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_without_passing_size + +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_size_negative +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_size_zero + +Note: test cases 8 to 11 are the alias of the fist 4 test cases, test cases 12 and 13 are both alias of +test case 5, and test cases 14 and 15 are the alias of the cases 6 and 7, respectively. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of OpenStack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' +* Test action 1: Create a volume VOL1 with a specific name and metadata +* Test action 2: Wait for VOL1 to reach 'available' status +* **Test assertion 1:** Verify the name of VOL1 is correct +* Test action 3: Retrieve VOL1's information +* **Test assertion 2:** Verify the retrieved volume name, ID and metadata are the same as VOL1 +* **Test assertion 3:** Verify VOL1's bootable flag is 'False' +* Test action 4: Update the name of VOL1 with the original value +* Test action 5: Update the name of VOL1 with a new value +* **Test assertion 4:** Verify the name of VOL1 is updated successfully +* Test action 6: Create a volume VOL2 with no name specified and a description contains characters '@#$%^*' +* **Test assertion 5:** Verify VOL2 is created successfully +* Test action 7: Update the name of VOL2 and description with the original value +* **Test assertion 6:** Verify VOL2's bootable flag is 'False' +* Test action 8: Create a volume with an invalid size '#$%' +* **Test assertion 7:** Verify create volume failed, a bad request error is returned in the response +* Test action 9: Create a volume with a nonexistent source volume +* **Test assertion 8:** Verify create volume failed, a 'Not Found' error is returned in the response +* Test action 10: Create a volume with a nonexistent volume type +* **Test assertion 9:** Verify create volume failed, a 'Not Found' error is returned in the response +* Test action 11: Create a volume without passing a volume size +* **Test assertion 10:** Verify create volume failed, a bad request error is returned in the response +* Test action 12: Create a volume with a negative volume size +* **Test assertion 11:** Verify create volume failed, a bad request error is returned in the response +* Test action 13: Create a volume with volume size '0' +* **Test assertion 12:** Verify create volume failed, a bad request error is returned in the response + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of creating a volume, getting volume +detail information and updating volume, the reference is, +Specifically, the test verifies that: + +* Volume can be created from a source volume. +* Volume detail information can be retrieved/updated. +* Create a volume with an invalid size is not allowed. +* Create a volume with a nonexistent source volume or volume type is not allowed. +* Create a volume without passing a volume size is not allowed. +* Create a volume with a negative volume size is not allowed. +* Create a volume with volume size '0' is not allowed. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------------------------------- +Test Case 6 - Volume service extension listing operations with the Cinder v2 API +-------------------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_extensions.ExtensionsV2TestJSON.test_list_extensions + +tempest.api.volume.test_extensions.ExtensionsTestJSON.test_list_extensions + +Note: the second test case is the alias of the first one. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API +* At least one Cinder extension is configured + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: List all cinder service extensions +* **Test assertion 1:** Verify all extensions are list in the extension list + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of listing all existent volume service extensions. + +* Cinder service extensions can be listed. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------------------------- +Test Case 7 - Volume GET operations with the Cinder v2 API +---------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_get_invalid_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_get_volume_without_passing_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_volume_get_nonexistent_volume_id + +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_get_invalid_volume_id +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_get_volume_without_passing_volume_id" +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_volume_get_nonexistent_volume_id + +Note: the latter 3 test cases is the alias of the first 3 ones. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Retrieve a volume with an invalid volume ID +* **Test assertion 1:** Verify retrieve volume failed, a 'Not Found' error is returned in the response +* Test action 2: Retrieve a volume with an empty volume ID +* **Test assertion 2:** Verify retrieve volume failed, a 'Not Found' error is returned in the response +* Test action 3: Retrieve a volume with a nonexistent volume ID +* **Test assertion 3:** Verify retrieve volume failed, a 'Not Found' error is returned in the response + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of getting volumes. +Specifically, the test verifies that: + +* Get a volume with an invalid/an empty/a nonexistent volume ID is not allowed. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------------- +Test Case 8 - Volume listing operations with the Cinder v2 API +-------------------------------------------------------------- + +Test case specification +----------------------- + tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_by_name tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_by_name @@ -43,40 +519,475 @@ tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volumes_list_by_ tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volumes_list_by_status tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volumes_list_details_by_availability_zone tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volumes_list_details_by_status -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_attach_volumes_with_nonexistent_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_invalid_size -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_snapshot_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_source_volid -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_volume_type -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_out_passing_size -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_size_negative -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_size_zero -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_without_passing_size -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_delete_invalid_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_delete_volume_without_passing_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_detach_volumes_with_invalid_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_get_invalid_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_get_volume_without_passing_volume_id tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_detail_with_invalid_status tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_detail_with_nonexistent_name tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_with_invalid_status tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_with_nonexistent_name +tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_pagination +tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_with_multiple_params +tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_pagination + +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_by_name +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_details_by_name +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_param_display_name_and_status +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_with_detail_param_display_name_and_status +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_with_detail_param_metadata +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_with_details +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_with_param_metadata +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_by_availability_zone +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_by_status +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_details_by_availability_zone +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_details_by_status +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_list_volumes_detail_with_invalid_status +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_list_volumes_detail_with_nonexistent_name +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_list_volumes_with_invalid_status +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_list_volumes_with_nonexistent_name +tempest.api.volume.v2.test_volumes_list.VolumesListTestJSON.test_volume_list_details_pagination +tempest.api.volume.v2.test_volumes_list.VolumesListTestJSON.test_volume_list_details_with_multiple_params +tempest.api.volume.v2.test_volumes_list.VolumesListTestJSON.test_volume_list_pagination + +Note: the latter 19 test cases is the alias of the first 19 ones. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API +* The backing file for the volume group that Nova uses has space for at least 3 1G volumes + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: List all existent volumes +* **Test assertion 1:** Verify the volume list is complete +* Test action 2: List existent volumes and filter the volume list by volume name +* **Test assertion 2:** Verify the length of filtered volume list is 1 and the retrieved volume is correct +* Test action 3: List existent volumes in detail and filter the volume list by volume name +* **Test assertion 3:** Verify the length of filtered volume list is 1 and the retrieved volume is correct +* Test action 4: List existent volumes and filter the volume list by volume name and status 'available' +* **Test assertion 4:** Verify the name and status parameters of the fetched volume are correct +* Test action 5: List existent volumes in detail and filter the volume list by volume name and status 'available' +* **Test assertion 5:** Verify the name and status parameters of the fetched volume are correct +* Test action 6: List all existent volumes in detail and filter the volume list by volume metadata +* **Test assertion 6:** Verify the metadata parameter of the fetched volume is correct +* Test action 7: List all existent volumes in detail +* **Test assertion 7:** Verify the volume list is complete +* Test action 8: List all existent volumes and filter the volume list by volume metadata +* **Test assertion 8:** Verify the metadata parameter of the fetched volume is correct +* Test action 9: List existent volumes and filter the volume list by availability zone +* **Test assertion 9:** Verify the availability zone parameter of the fetched volume is correct +* Test action 10: List all existent volumes and filter the volume list by volume status 'available' +* **Test assertion 10:** Verify the status parameter of the fetched volume is correct +* Test action 11: List existent volumes in detail and filter the volume list by availability zone +* **Test assertion 11:** Verify the availability zone parameter of the fetched volume is correct +* Test action 12: List all existent volumes in detail and filter the volume list by volume status 'available' +* **Test assertion 12:** Verify the status parameter of the fetched volume is correct +* Test action 13: List all existent volumes in detail and filter the volume list by an invalid volume status 'null' +* **Test assertion 13:** Verify the filtered volume list is empty +* Test action 14: List all existent volumes in detail and filter the volume list by a non-existent volume name +* **Test assertion 14:** Verify the filtered volume list is empty +* Test action 15: List all existent volumes and filter the volume list by an invalid volume status 'null' +* **Test assertion 15:** Verify the filtered volume list is empty +* Test action 16: List all existent volumes and filter the volume list by a non-existent volume name +* **Test assertion 16:** Verify the filtered volume list is empty +* Test action 17: List all existent volumes in detail and paginate the volume list by desired volume IDs +* **Test assertion 17:** Verify only the desired volumes are listed in the filtered volume list +* Test action 18: List all existent volumes in detail and filter the volume list by volume status 'available' and display limit '2' +* Test action 19: Sort the filtered volume list by IDs in ascending order +* **Test assertion 18:** Verify the length of filtered volume list is 2 +* **Test assertion 19:** Verify the status of retrieved volumes is correct +* **Test assertion 20:** Verify the filtered volume list is sorted correctly +* Test action 20: List all existent volumes in detail and filter the volume list by volume status 'available' and display limit '2' +* Test action 21: Sort the filtered volume list by IDs in descending order +* **Test assertion 21:** Verify the length of filtered volume list is 2 +* **Test assertion 22:** Verify the status of retrieved volumes is correct +* **Test assertion 23:** Verify the filtered volume list is sorted correctly +* Test action 22: List all existent volumes and paginate the volume list by desired volume IDs +* **Test assertion 24:** Verify only the desired volumes are listed in the filtered volume list + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of getting a list of volumes and filtering the volume list. +Specifically, the test verifies that: + +* Get a list of volumes (in detail) successful. +* Get a list of volumes (in detail) and filter volumes by name/status/metadata/availability zone successful. +* Volume list pagination functionality is working. +* Get a list of volumes in detail using combined condition successful. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------------------------------- +Test Case 9 - Volume metadata operations with the Cinder v2 API +--------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_create_get_delete_volume_metadata +tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_update_volume_metadata_item + +tempest.api.volume.test_volume_metadata.VolumesMetadataTest.test_crud_volume_metadata +tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_crud_volume_metadata + +tempest.api.volume.test_volume_metadata.VolumesMetadataTest.test_update_volume_metadata_item +tempest.api.volume.test_volume_metadata.VolumesMetadataTest.test_update_show_volume_metadata_item + +Note: Test case 3 and 4 are the alias of the first test case, and the last 2 test cases +are the alias of the second test case. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of OpenStack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create metadata for a provided volume VOL1 +* Test action 2: Get the metadata of VOL1 +* **Test assertion 1:** Verify the metadata of VOL1 is correct +* Test action 3: Update the metadata of VOL1 +* **Test assertion 2:** Verify the metadata of VOL1 is updated +* Test action 4: Delete one metadata item 'key1' of VOL1 +* **Test assertion 3:** Verify the metadata item 'key1' is deleted +* Test action 5: Create metadata for a provided volume VOL2 +* **Test assertion 4:** Verify the metadata of VOL2 is correct +* Test action 6: Update one metadata item 'key3' of VOL2 +* **Test assertion 5:** Verify the metadata of VOL2 is updated + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of creating metadata for a volume, getting the +metadata of a volume, updating volume metadata and deleting a metadata item of a volume. +Specifically, the test verifies that: + +* Create metadata for volume successfully. +* Get metadata of volume successfully. +* Update volume metadata and metadata item successfully. +* Delete metadata item of a volume successfully. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------------------------------------------------- +Test Case 10 - Verification of read-only status on volumes with the Cinder v2 API +--------------------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_volume_readonly_update + +tempest.api.volume.test_volumes_actions.VolumesActionsTest.test_volume_readonly_update + +Note: the second test case is the alias of the first one. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Update a provided volume VOL1's read-only access mode to 'True' +* **Test assertion 1:** Verify VOL1 is in read-only access mode +* Test action 2: Update a provided volume VOL1's read-only access mode to 'False' +* **Test assertion 2:** Verify VOL1 is not in read-only access mode + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of setting and updating volume read-only access mode. +Specifically, the test verifies that: + +* Volume read-only access mode can be set and updated. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------------------------------- +Test Case 11 - Volume reservation operations with the Cinder v2 API +------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_reserve_unreserve_volume tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_reserve_volume_with_negative_volume_status tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_reserve_volume_with_nonexistent_volume_id tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_unreserve_volume_with_nonexistent_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_empty_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_invalid_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_nonexistent_volume_id + +tempest.api.volume.test_volumes_actions.VolumesActionsTest.test_reserve_unreserve_volume +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_reserve_volume_with_negative_volume_status +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_reserve_volume_with_nonexistent_volume_id +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_unreserve_volume_with_nonexistent_volume_id + +Note: the last 4 test cases are the alias of the first 4 ones. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Update a provided volume VOL1 as reserved +* **Test assertion 1:** Verify VOL1 is in 'attaching' status +* Test action 2: Update VOL1 as un-reserved +* **Test assertion 2:** Verify VOL1 is in 'available' status +* Test action 3: Update a provided volume VOL2 as reserved +* Test action 4: Update VOL2 as reserved again +* **Test assertion 3:** Verify update VOL2 status failed, a bad request error is returned in the response +* Test action 5: Update VOL2 as un-reserved +* Test action 6: Update a non-existent volume as reserved by using an invalid volume ID +* **Test assertion 4:** Verify update non-existent volume as reserved failed, a 'Not Found' error is returned in the response +* Test action 7: Update a non-existent volume as un-reserved by using an invalid volume ID +* **Test assertion 5:** Verify update non-existent volume as un-reserved failed, a 'Not Found' error is returned in the response + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of reserving and un-reserving volumes. +Specifically, the test verifies that: + +* Volume can be reserved and un-reserved. +* Update a non-existent volume as reserved is not allowed. +* Update a non-existent volume as un-reserved is not allowed. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------------------------------------------------- +Test Case 12 - Volume snapshot creation/deletion operations with the Cinder v2 API +---------------------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_create_get_delete_snapshot_metadata +tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_update_snapshot_metadata_item +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_snapshot_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_delete_invalid_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_delete_volume_without_passing_volume_id tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_volume_delete_nonexistent_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_volume_get_nonexistent_volume_id tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshot_create_get_list_update_delete +tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_volume_from_snapshot tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshots_list_details_with_params tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshots_list_with_params -tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_volume_from_snapshot -tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_details_with_params -tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_with_params tempest.api.volume.test_volumes_snapshots_negative.VolumesV2SnapshotNegativeTestJSON.test_create_snapshot_with_nonexistent_volume_id tempest.api.volume.test_volumes_snapshots_negative.VolumesV2SnapshotNegativeTestJSON.test_create_snapshot_without_passing_volume_id -tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_pagination -tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_with_multiple_params -tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_pagination + +tempest.api.volume.test_snapshot_metadata.SnapshotMetadataTestJSON.test_crud_snapshot_metadata +tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_crud_snapshot_metadata + +tempest.api.volume.test_snapshot_metadata.SnapshotMetadataTestJSON.test_update_snapshot_metadata_item +tempest.api.volume.test_snapshot_metadata.SnapshotMetadataTestJSON.test_update_show_snapshot_metadata_item + +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_nonexistent_snapshot_id +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_delete_invalid_volume_id +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_delete_volume_without_passing_volume_id +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_volume_delete_nonexistent_volume_id +tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_snapshot_create_get_list_update_delete +tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_volume_from_snapshot + +tempest.api.volume.test_volumes_snapshots_list.VolumesSnapshotListTestJSON.test_snapshots_list_details_with_params +tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_details_with_params + +tempest.api.volume.test_volumes_snapshots_list.VolumesSnapshotListTestJSON.test_snapshots_list_with_params +tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_with_params + +tempest.api.volume.test_volumes_snapshots_negative.VolumesSnapshotNegativeTestJSON.test_create_snapshot_with_nonexistent_volume_id +tempest.api.volume.test_volumes_snapshots_negative.VolumesSnapshotNegativeTestJSON.test_create_snapshot_without_passing_volume_id + +Note: test case 13 and 14 are the alias of test case 1, test case 15 and 16 are the alias of test case 2, +test case 17 to 22 are the alias of test case 3 to 8 respectively, test case 23 and 24 are the alias of +test case 9, test case 25 and 26 are the alias of test case 10, and test case 27 and 28 are the alias of +test case 11 and 12 respectively. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of OpenStack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create metadata for a provided snapshot SNAP1 +* Test action 2: Get the metadata of SNAP1 +* **Test assertion 1:** Verify the metadata of SNAP1 is correct +* Test action 3: Update the metadata of SNAP1 +* **Test assertion 2:** Verify the metadata of SNAP1 is updated +* Test action 4: Delete one metadata item 'key3' of SNAP1 +* **Test assertion 3:** Verify the metadata item 'key3' is deleted +* Test action 5: Create metadata for a provided snapshot SNAP2 +* **Test assertion 4:** Verify the metadata of SNAP2 is correct +* Test action 6: Update one metadata item 'key3' of SNAP2 +* **Test assertion 5:** Verify the metadata of SNAP2 is updated +* Test action 7: Create a volume with a nonexistent snapshot +* **Test assertion 6:** Verify create volume failed, a 'Not Found' error is returned in the response +* Test action 8: Delete a volume with an invalid volume ID +* **Test assertion 7:** Verify delete volume failed, a 'Not Found' error is returned in the response +* Test action 9: Delete a volume with an empty volume ID +* **Test assertion 8:** Verify delete volume failed, a 'Not Found' error is returned in the response +* Test action 10: Delete a volume with a nonexistent volume ID +* **Test assertion 9:** Verify delete volume failed, a 'Not Found' error is returned in the response +* Test action 11: Create a snapshot SNAP2 from a provided volume VOL1 +* Test action 12: Retrieve SNAP2's detail information +* **Test assertion 10:** Verify SNAP2 is created from VOL1 +* Test action 13: Update the name and description of SNAP2 +* **Test assertion 11:** Verify the name and description of SNAP2 are updated in the response body of update snapshot API +* Test action 14: Retrieve SNAP2's detail information +* **Test assertion 12:** Verify the name and description of SNAP2 are correct +* Test action 15: Delete SNAP2 +* Test action 16: Create a volume VOL2 with a volume size +* Test action 17: Create a snapshot SNAP3 from VOL2 +* Test action 18: Create a volume VOL3 from SNAP3 with a bigger volume size +* Test action 19: Retrieve VOL3's detail information +* **Test assertion 13:** Verify volume size and source snapshot of VOL3 are correct +* Test action 20: List all snapshots in detail and filter the snapshot list by name +* **Test assertion 14:** Verify the filtered snapshot list is correct +* Test action 21: List all snapshots in detail and filter the snapshot list by status +* **Test assertion 15:** Verify the filtered snapshot list is correct +* Test action 22: List all snapshots in detail and filter the snapshot list by name and status +* **Test assertion 16:** Verify the filtered snapshot list is correct +* Test action 23: List all snapshots and filter the snapshot list by name +* **Test assertion 17:** Verify the filtered snapshot list is correct +* Test action 24: List all snapshots and filter the snapshot list by status +* **Test assertion 18:** Verify the filtered snapshot list is correct +* Test action 25: List all snapshots and filter the snapshot list by name and status +* **Test assertion 19:** Verify the filtered snapshot list is correct +* Test action 26: Create a snapshot from a nonexistent volume by using an invalid volume ID +* **Test assertion 20:** Verify create snapshot failed, a 'Not Found' error is returned in the response +* Test action 27: Create a snapshot from a volume by using an empty volume ID +* **Test assertion 21:** Verify create snapshot failed, a 'Not Found' error is returned in the response + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of managing snapshot and snapshot metadata. +Specifically, the test verifies that: + +* Create metadata for snapshot successfully. +* Get metadata of snapshot successfully. +* Update snapshot metadata and metadata item successfully. +* Delete metadata item of a snapshot successfully. +* Create a volume from a nonexistent snapshot is not allowed. +* Delete a volume using an invalid volume ID is not allowed. +* Delete a volume without passing the volume ID is not allowed. +* Delete a non-existent volume is not allowed. +* Create snapshot successfully. +* Get snapshot's detail information successfully. +* Update snapshot attributes successfully. +* Delete snapshot successfully. +* Creates a volume and a snapshot passing a size different from the source successfully. +* List snapshot details by display_name and status filters successfully. +* Create a snapshot from a nonexistent volume is not allowed. +* Create a snapshot from a volume without passing the volume ID is not allowed. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------------- +Test Case 13 - Volume update operations with the Cinder v2 API +-------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_empty_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_invalid_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_nonexistent_volume_id + +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_update_volume_with_empty_volume_id +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_update_volume_with_invalid_volume_id +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_update_volume_with_nonexistent_volume_id + +Note: the last 3 test cases are the alias of the first 3 ones. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Update a volume by using an empty volume ID +* **Test assertion 1:** Verify update volume failed, a 'Not Found' error is returned in the response +* Test action 2: Update a volume by using an invalid volume ID +* **Test assertion 2:** Verify update volume failed, a 'Not Found' error is returned in the response +* Test action 3: Update a non-existent volume by using a random generated volume ID +* **Test assertion 3:** Verify update volume failed, a 'Not Found' error is returned in the response + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of updating volume attributes. +Specifically, the test verifies that: + +* Update a volume without passing the volume ID is not allowed. +* Update a volume using an invalid volume ID is not allowed. +* Update a non-existent volume is not allowed. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A diff --git a/docs/testing/user/testspecification/vping/index.rst b/docs/testing/user/testspecification/vping/index.rst new file mode 100644 index 00000000..d7a207c0 --- /dev/null +++ b/docs/testing/user/testspecification/vping/index.rst @@ -0,0 +1,279 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Ericsson AB + +======================== +Vping test specification +======================== + +.. toctree:: + :maxdepth: 2 + +Scope +===== + +The vping test area evaluates basic NFVi capabilities of the system under test. +These capabilities include creating a small number of virtual machines, +establishing basic L3 connectivity between them and verifying connectivity by +means of ICMP packets. + + +References +========== + +- Neutron Client + + - https://docs.openstack.org/developer/python-neutronclient/usage/library.html + +- Nova Client + + - https://docs.openstack.org/developer/python-novaclient/ref/v2/servers.html + +- SSHClient + + - http://docs.paramiko.org/en/2.2/ + +- SCPClient + + - https://pypi.python.org/pypi/scp + + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test +area + +- ICMP - Internet Control Message Protocol +- L3 - Layer 3 +- NFVi - Network functions virtualization infrastructure +- SCP - Secure Copy +- SSH - Secure Shell +- VM - Virtual machine + + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + + +Test Area Structure +=================== + +The test area is structured in two separate tests which are executed +sequentially. The order of the tests is arbitrary as there are no dependencies +across the tests. + + +Test Descriptions +================= + +-------------------------------------------------------------------- +Test Case 1 - vPing using userdata provided by nova metadata service +-------------------------------------------------------------------- + +Short name +---------- + +opnfv.vping.userdata + + +Use case specification +---------------------- + +This test evaluates the use case where an NFVi tenant boots up two VMs and +requires L3 connectivity between those VMs. The target IP is passed to the VM +that will initiate pings by using a custom userdata script provided by nova metadata service. + + +Test preconditions +------------------ + +At least one compute node is available. No further pre-configuration needed. + + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for verifying connectivity +'''''''''''''''''''''''''''''''''''''' + +Connectivity between VMs is tested by sending ICMP ping packets between +selected VMs. The target IP is passed to the VM sending pings by using a +custom userdata script by means of the config driver mechanism provided by +Nova metadata service. Whether or not a ping was successful is determined by +checking the console output of the source VMs. + + +Test execution +'''''''''''''' + +* Test action 1: + * Create a private tenant network by using neutron client + * Create one subnet and one router in the network by neutron client + * Add one interface between the subnet and router + * Add one gateway route to the router by neutron client + * Store the network id in the response +* **Test assertion 1:** The network id, subnet id and router id can be found in the response +* Test action 2: + * Create an security group by using neutron client + * Store the security group id parameter in the response +* **Test assertion 2:** The security group id can be found in the response +* Test action 3: boot VM1 by using nova client with configured name, image, flavor, private tenant + network created in test action 1, security group created in test action 2 +* **Test assertion 3:** The VM1 object can be found in the response +* Test action 4: Generate ping script with the IP of VM1 to be passed as userdata provided by + the **nova metadata service**. +* Test action 5: Boot VM2 by using nova client with configured name, image, flavor, private tenant + network created in test action 1, security group created in test action 2, userdata created + in test action 4 +* **Test assertion 4:** The VM2 object can be found in the response +* Test action 6: Inside VM2, the ping script is executed automatically when booted and it contains a + loop doing the ping until the return code is 0 or timeout reached. For each ping, when the return + code is 0, "vPing OK" is printed in the VM2 console-log, otherwise, "vPing KO" is printed. + Monitoring the console-log of VM2 to see the response generated by the script. +* **Test assertion 5:** "vPing OK" is detected, when monitoring the console-log in VM2 +* Test action 7: delete VM1, VM2 +* **Test assertion 6:** VM1 and VM2 are not present in the VM list +* Test action 8: delete security group, gateway, interface, router, subnet and network +* **Test assertion 7:** The security group, gateway, interface, router, subnet and network are + no longer present in the lists after deleting + + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates basic NFVi capabilities of the system under test. +Specifically, the test verifies that: + +* Neutron client network, subnet, router, interface create commands return valid "id" parameters + which are shown in the create response message +* Neutron client interface add command to add between subnet and router returns success code +* Neutron client gateway add command to add to router returns success code +* Neutron client security group create command returns valid "id" parameter which is shown in + the response message +* Nova client VM create command returns valid VM attributes response message +* Nova metadata server can transfer userdata configuration at nova client VM booting time +* Ping command from one VM to the other in same private tenant network returns valid code +* All items created using neutron client or nova client create commands are able to be removed by + using the returned identifiers + +In order to pass this test, all test assertions listed in the test execution +above need to pass. + + +Post conditions +--------------- + +None + + +---------------------------------------------- +Test Case 2 - vPing using SSH to a floating IP +---------------------------------------------- + +Short name +---------- + +opnfv.vping.ssh + + +Use case specification +---------------------- + +This test evaluates the use case where an NFVi tenant boots up two VMs and requires +L3 connectivity between those VMs. An SSH connection is establised from the host to +a floating IP associated with VM2 and ``ping`` is executed on VM2 with the IP of VM1 as target. + + +Test preconditions +------------------ + +At least one compute node is available. There should exist an OpenStack external network +and can assign floating IP. + + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for verifying connectivity +'''''''''''''''''''''''''''''''''''''' + +Connectivity between VMs is tested by sending ICMP ping packets between +selected VMs. To this end, the test establishes an SSH connection from the host +running the test suite to a floating IP associated with VM2 and executes ``ping`` +on VM2 with the IP of VM1 as target. + + +Test execution +'''''''''''''' + + +* Test action 1: + * Create a private tenant network by neutron client + * Create one subnet and one router are created in the network by using neutron client + * Create one interface between the subnet and router + * Add one gateway route to the router by neutron client + * Store the network id in the response +* **Test assertion 1:** The network id, subnet id and router id can be found in the response +* Test action 2: + * Create an security group by using neutron client + * Store the security group id parameter in the response +* **Test assertion 2:** The security group id can be found in the response +* Test action 3: Boot VM1 by using nova client with configured name, image, flavor, private tenant + network created in test action 1, security group created in test action 2 +* **Test assertion 3:** The VM1 object can be found in the response +* Test action 4: Boot VM2 by using nova client with configured name, image, flavor, private tenant + network created in test action 1, security group created in test action 2 +* **Test assertion 4:** The VM2 object can be found in the response +* Test action 5: create one floating IP by using neutron client, storing the floating IP address + returned in the response +* **Test assertion 5:** Floating IP address can be found in the response +* Test action 6: Assign the floating IP address created in test action 5 to VM2 by using nova client +* **Test assertion 6:** The assigned floating IP can be found in the VM2 console log file +* Test action 7: Establish SSH connection between the test host and VM2 through the floating IP +* **Test assertion 7:** SSH connection between the test host and VM2 is established within + 300 seconds +* Test action 8: Copy the Ping script from the test host to VM2 by using SCPClient +* **Test assertion 8:** The Ping script can be found inside VM2 +* Test action 9: Inside VM2, to execute the Ping script to ping VM1, the Ping script contains a + loop doing the ping until the return code is 0 or timeout reached, for each ping, when the return + code is 0, "vPing OK" is printed in the VM2 console-log, otherwise, "vPing KO" is printed. + Monitoring the console-log of VM2 to see the response generated by the script. +* **Test assertion 9:** "vPing OK" is detected, when monitoring the console-log in VM2 +* Test action 10: delete VM1, VM2 +* **Test assertion 10:** VM1 and VM2 are not present in the VM list +* Test action 11: delete floating IP, security group, gateway, interface, router, subnet and network +* **Test assertion 11:** The security group, gateway, interface, router, subnet and network are + no longer present in the lists after deleting + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates basic NFVi capabilities of the system under test. +Specifically, the test verifies that: + +* Neutron client network, subnet, router, interface create commands return valid "id" parameters + which are shown in the create response message +* Neutron client interface add command to add between subnet and router return success code +* Neutron client gateway add command to add to router return success code +* Neutron client security group create command returns valid "id" parameter which is shown in the + response message +* Nova client VM create command returns valid VM attributes response message +* Neutron client floating IP create command return valid floating IP address +* Nova client add floating IP command returns valid response message +* SSH connection can be established using a floating IP +* Ping command from one VM to another in same private tenant network returns valid code +* All items created using neutron client or nova client create commands are able to be removed by + using the returned identifiers + +In order to pass this test, all test assertions listed in the test execution +above need to pass. + + +Post conditions +--------------- + +None diff --git a/docs/testing/user/testspecification/vpn/index.rst b/docs/testing/user/testspecification/vpn/index.rst index 1b5fe439..0a8a8d17 100644 --- a/docs/testing/user/testspecification/vpn/index.rst +++ b/docs/testing/user/testspecification/vpn/index.rst @@ -12,14 +12,17 @@ VPN test specification Scope ===== -The VPN test area evaluates the ability of the system under test to support VPN networking -for virtual workdloads. The tests in this suite will evaluate establishing VPN networks, -publishing and communication between endpoints using BGP and tear down of the networks. +The VPN test area evaluates the ability of the system under test to support VPN +networking for virtual workloads. The tests in this test area will evaluate +establishing VPN networks, publishing and communication between endpoints using +BGP and tear down of the networks. References -================ +========== -This test suite assumes support for the following specifications: +This test area evaluates the ability of the system to perform selected actions +defined in the following specifications. Details of specific features evaluated +are described in the test descriptions. - RFC 4364 - BGP/MPLS IP Virtual Private Networks @@ -33,10 +36,12 @@ This test suite assumes support for the following specifications: - https://tools.ietf.org/html/rfc2547 + Definitions and abbreviations ============================= -The following terms and abreviations are used in conunction with this test suite +The following terms and abbreviations are used in conjunction with this test +area - BGP - Border gateway protocol - eRT - Export route target @@ -48,15 +53,27 @@ The following terms and abreviations are used in conunction with this test suite - VPN - Virtual private network - VLAN - Virtual local area network + System Under Test (SUT) ======================= -The system under test is assumed to be the NFVi in operation on an Pharos compliant infrastructure. +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + -Test Suite Structure -==================== +Test Area Structure +=================== + +The test area is structured in four separate tests which are executed +sequentially. The order of the tests is arbitrary as there are no dependencies +across the tests. Specifially, every test performs clean-up operations which +return the system to the same state as before the test. + +The test area evaluates the ability of the SUT to establish connectivity +between Virtual Machines using an appropriate route target configuration, +reconfigure the route targets to remove connectivity between the VMs, then +reestablish connectivity by re-association. -The test suite is structured in some way that I am unable to articulate at this time. Test Descriptions ================= @@ -65,43 +82,451 @@ Test Descriptions Test Case 1 - VPN provides connectivity between Neutron subnets ---------------------------------------------------------------- +Short name +---------- + +opnfv.sdnvpn.subnet_connectivity + + Use case specification ---------------------- -This test evaluate the instance where an NFVi tenant wants to use a BGPVPN to provide -connectivity between VMs on different Neutron networks and Subnets that reside on different hosts. +This test evaluates the use case where an NFVi tenant uses a BGPVPN to provide +connectivity between VMs on different Neutron networks and subnets that reside +on different hosts. + Test preconditions ------------------ -2 compute nodes are available, denoted Node1 and Node 2 in the following. +2 compute nodes are available, denoted Node1 and Node2 in the following. + Basic test flow execution description and pass/fail criteria ------------------------------------------------------------ -Set up VM1 and VM2 on Node1 and VM3 on Node2, all having ports in the same Neutron Network N1 -and all having 10.10.10/24 addresses (this subnet is denoted SN1 in the following). +Methodology for verifying connectivity +'''''''''''''''''''''''''''''''''''''' + +Connectivity between VMs is tested by sending ICMP ping packets between +selected VMs. The target IPs are passed to the VMs sending pings by means of a +custom user data script. Whether or not a ping was successful is determined by +checking the console output of the source VMs. + + +Test execution +'''''''''''''' + +* Create Neutron network N1 and subnet SN1 with IP range 10.10.10.0/24 +* Create Neutron network N2 and subnet SN2 with IP range 10.10.11.0/24 + +* Create VM1 on Node1 with a port in network N1 +* Create VM2 on Node1 with a port in network N1 +* Create VM3 on Node2 with a port in network N1 +* Create VM4 on Node1 with a port in network N2 +* Create VM5 on Node2 with a port in network N2 + +* Create VPN1 with eRT<>iRT +* Create network association between network N1 and VPN1 + +* VM1 sends ICMP packets to VM2 using ``ping`` + +* **Test assertion 1:** Ping from VM1 to VM2 succeeds: ``ping`` exits with return code 0 + +* VM1 sends ICMP packets to VM3 using ``ping`` + +* **Test assertion 2:** Ping from VM1 to VM3 succeeds: ``ping`` exits with return code 0 + +* VM1 sends ICMP packets to VM4 using ``ping`` + +* **Test assertion 3:** Ping from VM1 to VM4 fails: ``ping`` exits with a non-zero return code + +* Create network association between network N2 and VPN1 + +* VM4 sends ICMP packets to VM5 using ``ping`` -Set up VM4 on Node1 and VM5 on Node2, both having ports in Neutron Network N2 -and having 10.10.11/24 addresses (this subnet is denoted SN2 in the following). +* **Test assertion 4:** Ping from VM4 to VM5 succeeds: ``ping`` exits with return code 0 -* Create VPN1 with eRT<>iRT and associate SN1 to it -* Test action 1: SSH into VM1 and ping VM2, test passes if ping works -* Test action 2: SSH into VM1 and ping VM3, test passes is ping works -* Test action 3: SSH into VM1 and ping VM4, test passes if ping does not work -* Associate SN2 to VPN1 -* Test action 4: Ping from VM4 to VM5 should work -* Test action 5: Ping from VM1 to VM4 should not work -* Test action 6: Ping from VM1 to VM5 should not work * Configure iRT=eRT in VPN1 -* Test action 7: Ping from VM1 to VM4 should work -* Test action 8: Ping from VM1 to VM5 should work -The pass criteria for this test case is that all instructions are able to be carried out -according to the described behaviour without deviation. -A negative result will be generated if the above is not met in completion. +* VM1 sends ICMP packets to VM4 using ``ping`` + +* **Test assertion 5:** Ping from VM1 to VM4 succeeds: ``ping`` exits with return code 0 + +* VM1 sends ICMP packets to VM5 using ``ping`` + +* **Test assertion 6:** Ping from VM1 to VM5 succeeds: ``ping`` exits with return code 0 + +* Delete all instances: VM1, VM2, VM3, VM4 and VM5 + +* Delete all networks and subnets: networks N1 and N2 including subnets SN1 and SN2 + +* Delete all network associations and VPN1 + + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the capability of the NFVi and VIM to provide routed IP +connectivity between VMs by means of BGP/MPLS VPNs. Specifically, the test +verifies that: + +* VMs in the same Neutron subnet have IP connectivity regardless of BGP/MPLS + VPNs (test assertion 1, 2, 4) + +* VMs in different Neutron subnets do not have IP connectivity by default - in + this case without associating VPNs with the same import and export route + targets to the Neutron networks (test assertion 3) + +* VMs in different Neutron subnets have routed IP connectivity after + associating both networks with BGP/MPLS VPNs which have been configured with + the same import and export route targets (test assertion 5, 6). Hence, + adjusting the ingress and egress route targets enables as well as prohibits + routing. + +In order to pass this test, all test assertions listed in the test execution +above need to pass. + + +Post conditions +--------------- + +N/A + +------------------------------------------------------------ +Test Case 2 - VPNs ensure traffic separation between tenants +------------------------------------------------------------ + +Short Name +---------- + +opnfv.sdnvpn.tenant_separation + + +Use case specification +---------------------- + +This test evaluates if VPNs provide separation of traffic such that overlapping +IP ranges can be used. + + +Test preconditions +------------------ + +2 compute nodes are available, denoted Node1 and Node2 in the following. + + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for verifying connectivity +'''''''''''''''''''''''''''''''''''''' + +Connectivity between VMs is tested by establishing an SSH connection. Moreover, +the command "hostname" is executed at the remote VM in order to retrieve the +hostname of the remote VM. The retrieved hostname is furthermore compared +against an expected value. This is used to verify tenant traffic separation, +i.e., despite overlapping IPs, a connection is made to the correct VM as +determined by means of the hostname of the target VM. + + + +Test execution +'''''''''''''' + +* Create Neutron network N1 +* Create subnet SN1a of network N1 with IP range 10.10.10.0/24 +* Create subnet SN1b of network N1 with IP range 10.10.11.0/24 + +* Create Neutron network N2 +* Create subnet SN2a of network N2 with IP range 10.10.10.0/24 +* Create subnet SN2b of network N2 with IP range 10.10.11.0/24 + +* Create VM1 on Node1 with a port in network N1 and IP 10.10.10.11. +* Create VM2 on Node1 with a port in network N1 and IP 10.10.10.12. +* Create VM3 on Node2 with a port in network N1 and IP 10.10.11.13. +* Create VM4 on Node1 with a port in network N2 and IP 10.10.10.12. +* Create VM5 on Node2 with a port in network N2 and IP 10.10.11.13. + +* Create VPN1 with iRT=eRT=RT1 +* Create network association between network N1 and VPN1 + +* VM1 attempts to execute the command ``hostname`` on the VM with IP 10.10.10.12 via SSH. + +* **Test assertion 1:** VM1 can successfully connect to the VM with IP + 10.10.10.12. via SSH and execute the remote command ``hostname``. The + retrieved hostname equals the hostname of VM2. + +* VM1 attempts to execute the command ``hostname`` on the VM with IP 10.10.11.13 via SSH. + +* **Test assertion 2:** VM1 can successfully connect to the VM with IP + 10.10.11.13 via SSH and execute the remote command ``hostname``. The + retrieved hostname equals the hostname of VM3. + +* Create VPN2 with iRT=eRT=RT2 +* Create network association between network N2 and VPN2 + +* VM4 attempts to execute the command ``hostname`` on the VM with IP 10.10.11.13 via SSH. + +* **Test assertion 3:** VM4 can successfully connect to the VM with IP + 10.10.11.13 via SSH and execute the remote command ``hostname``. The + retrieved hostname equals the hostname of VM5. + +* VM4 attempts to execute the command ``hostname`` on the VM with IP 10.10.11.11 via SSH. + +* **Test assertion 4:** VM4 cannot connect to the VM with IP 10.10.11.11 via SSH. + +* Delete all instances: VM1, VM2, VM3, VM4 and VM5 + +* Delete all networks and subnets: networks N1 and N2 including subnets SN1a, SN1b, SN2a and SN2b + +* Delete all network associations, VPN1 and VPN2 + + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the capability of the NFVi and VIM to provide routed IP +connectivity between VMs by means of BGP/MPLS VPNs. Specifically, the test +verifies that: + +* VMs in the same Neutron subnet (still) have IP connectivity between each + other when a BGP/MPLS VPN is associated with the network (test assertion 1). + +* VMs in different Neutron subnets have routed IP connectivity between each + other when BGP/MPLS VPNs with the same import and expert route targets are + associated with both networks (assertion 2). + +* VMs in different Neutron networks and BGP/MPLS VPNs with different import and + export route targets can have overlapping IP ranges. The BGP/MPLS VPNs + provide traffic separation (assertion 3 and 4). + +In order to pass this test, all test assertions listed in the test execution +above need to pass. + + +Post conditions +--------------- + +N/A + +-------------------------------------------------------------------------------- +Test Case 3 - VPN provides connectivity between subnets using router association +-------------------------------------------------------------------------------- + +Short Name +---------- + +opnfv.sdnvpn.router_association + + +Use case specification +---------------------- + +This test evaluates if a VPN provides connectivity between two subnets by +utilizing two different VPN association mechanisms: a router association and a +network association. + +Specifically, the test network topology comprises two networks N1 and N2 with +corresponding subnets. Additionally, network N1 is connected to a router R1. +This test verifies that a VPN V1 provides connectivity between both networks +when applying a router association to router R1 and a network association to +network N2. + + +Test preconditions +------------------ + +2 compute nodes are available, denoted Node1 and Node2 in the following. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for verifying connectivity +'''''''''''''''''''''''''''''''''''''' + +Connectivity between VMs is tested by sending ICMP ping packets between +selected VMs. The target IPs are passed to the VMs sending pings by means of a +custom user data script. Whether or not a ping was successful is determined by +checking the console output of the source VMs. + + +Test execution +'''''''''''''' + +* Create a network N1, a subnet SN1 with IP range 10.10.10.0/24 and a connected router R1 +* Create a network N2, a subnet SN2 with IP range 10.10.11.0/24 + +* Create VM1 on Node1 with a port in network N1 +* Create VM2 on Node1 with a port in network N1 +* Create VM3 on Node2 with a port in network N1 +* Create VM4 on Node1 with a port in network N2 +* Create VM5 on Node2 with a port in network N2 + +* Create VPN1 with eRT<>iRT so that connected subnets should not reach each other + +* Create route association between router R1 and VPN1 + +* VM1 sends ICMP packets to VM2 using ``ping`` + +* **Test assertion 1:** Ping from VM1 to VM2 succeeds: ``ping`` exits with return code 0 + +* VM1 sends ICMP packets to VM3 using ``ping`` + +* **Test assertion 2:** Ping from VM1 to VM3 succeeds: ``ping`` exits with return code 0 + +* VM1 sends ICMP packets to VM4 using ``ping`` + +* **Test assertion 3:** Ping from VM1 to VM4 fails: ``ping`` exits with a non-zero return code + +* Create network association between network N2 and VPN1 + +* VM4 sends ICMP packets to VM5 using ``ping`` + +* **Test assertion 4:** Ping from VM4 to VM5 succeeds: ``ping`` exits with return code 0 + +* Change VPN1 so that iRT=eRT + +* VM1 sends ICMP packets to VM4 using ``ping`` + +* **Test assertion 5:** Ping from VM1 to VM4 succeeds: ``ping`` exits with return code 0 + +* VM1 sends ICMP packets to VM5 using ``ping`` + +* **Test assertion 6:** Ping from VM1 to VM5 succeeds: ``ping`` exits with return code 0 + +* Delete all instances: VM1, VM2, VM3, VM4 and VM5 + +* Delete all networks, subnets and routers: networks N1 and N2 including subnets SN1 and SN2, router R1 + +* Delete all network and router associations and VPN1 + + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the capability of the NFVi and VIM to provide routed IP +connectivity between VMs by means of BGP/MPLS VPNs. Specifically, the test +verifies that: + +* VMs in the same Neutron subnet have IP connectivity regardless of the import + and export route target configuration of BGP/MPLS VPNs (test assertion 1, 2, 4) + +* VMs in different Neutron subnets do not have IP connectivity by default - in + this case without associating VPNs with the same import and export route + targets to the Neutron networks or connected Neutron routers (test assertion 3). + +* VMs in two different Neutron subnets have routed IP connectivity after + associating the first network and a router connected to the second network + with BGP/MPLS VPNs which have been configured with the same import and export + route targets (test assertion 5, 6). Hence, adjusting the ingress and egress + route targets enables as well as prohibits routing. + +* Network and router associations are equivalent methods for binding Neutron networks + to VPN. + +In order to pass this test, all test assertions listed in the test execution +above need to pass. + + +Post conditions +--------------- + +N/A + +--------------------------------------------------------------------------------------------------- +Test Case 4 - Verify interworking of router and network associations with floating IP functionality +--------------------------------------------------------------------------------------------------- + +Short Name +---------- + +opnfv.sdnvpn.router_association_floating_ip + + +Use case specification +---------------------- + +This test evaluates if both the router association and network association +mechanisms interwork with floating IP functionality. + +Specifically, the test network topology comprises two networks N1 and N2 with +corresponding subnets. Additionally, network N1 is connected to a router R1. +This test verifies that i) a VPN V1 provides connectivity between both networks +when applying a router association to router R1 and a network association to +network N2 and ii) a VM in network N1 is reachable externally by means of a +floating IP. + + +Test preconditions +------------------ + +At least one compute node is available. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for verifying connectivity +'''''''''''''''''''''''''''''''''''''' + +Connectivity between VMs is tested by sending ICMP ping packets between +selected VMs. The target IPs are passed to the VMs sending pings by means of a +custom user data script. Whether or not a ping was successful is determined by +checking the console output of the source VMs. + + +Test execution +'''''''''''''' + +* Create a network N1, a subnet SN1 with IP range 10.10.10.0/24 and a connected router R1 +* Create a network N2 with IP range 10.10.20.0/24 + +* Create VM1 with a port in network N1 +* Create VM2 with a port in network N2 + +* Create VPN1 +* Create a router association between router R1 and VPN1 +* Create a network association between network N2 and VPN1 + + +* VM1 sends ICMP packets to VM2 using ``ping`` + +* **Test assertion 1:** Ping from VM1 to VM2 succeeds: ``ping`` exits with return code 0 + +* Assign a floating IP to VM1 + +* The host running the test framework sends ICMP packets to VM1 using ``ping`` + +* **Test assertion 2:** Ping from the host running the test framework to the + floating IP of VM1 succeeds: ``ping`` exits with return code 0 + +* Delete floating IP assigned to VM1 + +* Delete all instances: VM1, VM2 + +* Delete all networks, subnets and routers: networks N1 and N2 including subnets SN1 and SN2, router R1 + +* Delete all network and router associations as well as VPN1 + + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the capability of the NFVi and VIM to provide routed IP +connectivity between VMs by means of BGP/MPLS VPNs. Specifically, the test +verifies that: + +* VMs in the same Neutron subnet have IP connectivity regardless of the import + and export route target configuration of BGP/MPLS VPNs (test assertion 1) + +* VMs connected to a network which has been associated with a BGP/MPLS VPN are + reachable through floating IPs. + +In order to pass this test, all test assertions listed in the test execution +above need to pass. + Post conditions --------------- -TBD - should there be any other than the system is in the same state it started out as? +N/A |