diff options
Diffstat (limited to 'docs/testing')
| -rw-r--r-- | docs/testing/developer/testscope/index.rst | 18 | ||||
| -rw-r--r-- | docs/testing/user/testspecification/machinelifecycle/index.rst | 757 |
2 files changed, 775 insertions, 0 deletions
diff --git a/docs/testing/developer/testscope/index.rst b/docs/testing/developer/testscope/index.rst index 35ff60af..7ae466be 100644 --- a/docs/testing/developer/testscope/index.rst +++ b/docs/testing/developer/testscope/index.rst @@ -654,3 +654,21 @@ Dynamic Network Runtime Operations Through the Life of a VNF | tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details | tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_instance_port_admin_state | tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state + +---------------------------------------------------------------- +Correct Behavior after Common Virtual Machine Life Cycles Events +---------------------------------------------------------------- + +| tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario +| tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration +| tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration_revert +| tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_pause_unpause +| tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_reboot +| tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_rebuild +| tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_resize +| tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_stop_start +| tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_suspend_resume +| tempest.scenario.test_server_advanced_ops.TestServerAdvancedOps.test_server_sequence_suspend_resume +| tempest.scenario.test_server_advanced_ops.TestServerAdvancedOps.test_resize_volume_backed_server_confirm +| tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_instance +| tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_volume_backed_instance diff --git a/docs/testing/user/testspecification/machinelifecycle/index.rst b/docs/testing/user/testspecification/machinelifecycle/index.rst new file mode 100644 index 00000000..598812f5 --- /dev/null +++ b/docs/testing/user/testspecification/machinelifecycle/index.rst @@ -0,0 +1,757 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Huawei Technologies Co.,Ltd + +=========================================================== +Common virtual machine life cycle events test specification +=========================================================== + +.. toctree:: + :maxdepth: 2 + +Scope +===== + +The common virtual machine life cycle events test area evaluates the ability of +the system under test to behave correctly after common virtual machine life +cycle events. The tests in this test area will evaluate: + +- Stop/Start a server +- Reboot a server +- Rebuild a server +- Pause/Unpause a server +- Suspend/Resume a server +- Resize a server +- Resizing a volume-backed server +- Sequence suspend resume +- Shelve/Unshelve a server +- Cold migrate a server +- Live migrate a server + +References +========== + +- iSCSI + + - https://docs.openstack.org/liberty/config-reference/content/config-iscsi-storage.html + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test area + +- API - Application Programming Interface +- NFVi - Network Functions Virtualization infrastructure +- VIM - Virtual Infrastructure Manager +- VM - Virtual Machine + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + +Test Area Structure +=================== + +The test area is structured based on common virtual machine life cycle events. +Each test case is able to run independently, i.e. irrelevant of the state +created by a previous test. Specifically, every test performs clean-up +operations which return the system to the same state as before the test. + +Test Descriptions +================= + +API Used and Reference +---------------------- + +Block storage: https://developer.openstack.org/api-ref/block-storage + +- create volume +- delete volume +- attach volume to server +- detach volume from server + +Security Groups: https://developer.openstack.org/api-ref/network/v2/index.html#security-groups-security-groups + +- create security group +- delete security group + +Networks: https://developer.openstack.org/api-ref/networking/v2/index.html#networks + +- create network +- delete network + +Routers and interface: https://developer.openstack.org/api-ref/networking/v2/index.html#routers-routers + +- create router +- delete router +- add interface to router + +Subnets: https://developer.openstack.org/api-ref/networking/v2/index.html#subnets + +- create subnet +- delete subnet + +Servers: https://developer.openstack.org/api-ref/compute/ + +- create keypair +- create server +- show server +- delete server +- add/assign floating IP +- resize server +- revert resized server +- confirm resized server +- pause server +- unpause server +- start server +- stop server +- reboot server +- rebuild server +- suspend server +- resume suspended server +- shelve server +- unshelve server +- migrate server +- live-migrate server + +Ports: https://developer.openstack.org/api-ref/networking/v2/index.html#ports + +- create port +- delete port + +Floating IPs: https://developer.openstack.org/api-ref/networking/v2/index.html#floating-ips-floatingips + +- create floating IP +- delete floating IP + +Availability zone: https://developer.openstack.org/api-ref/compute/ + +- get availability zone + +------------------------------------ +Test Case 1 - Minimum basic scenario +------------------------------------ + +Test case specification +----------------------- + +tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario + +Test preconditions +------------------ + +* Nova, cinder, glance, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create an image IMG1 +* Test action 2: Create a keypair KEYP1 +* Test action 3: Create a server VM1 with IMG1 and KEYP1 +* **Test assertion 1:** Verify VM1 is created successfully +* Test action 4: Create a volume VOL1 +* **Test assertion 2:** Verify VOL1 is created successfully +* Test action 5: Attach VOL1 to VM1 +* **Test assertion 3:** Verify VOL1's status has been updated after attached to VM1 +* Test action 6: Create a floating IP FIP1 and assign FIP1 to VM1 +* **Test assertion 4:** Verify VM1's addresses have been refreshed after associating FIP1 +* Test action 7: Create and add security group SG1 to VM1 +* **Test assertion 5:** Verify can SSH to VM1 via FIP1 +* Test action 8: Reboot VM1 +* **Test assertion 6:** Verify can SSH to VM1 via FIP1 +* **Test assertion 7:** Verify VM1's disk count equals to 1 +* Test action 9: Delete the floating IP FIP1 from VM1 +* **Test assertion 8:** Verify VM1's addresses have been refreshed after disassociating FIP1 +* Test action 10: Delete SG1, IMG1, KEYP1, VOL1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates a minimum basic scenario. Specifically, the test verifies that: + +* The server can be connected before reboot. + +* The server can be connected after reboot. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------- +Test Case 2 - Cold migration +---------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration + +Test preconditions +------------------ + +* At least 2 compute nodes +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Get VM1's host info SRC_HOST +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 6: Cold migrate VM1 +* Test action 7: Wait for VM1 to reach 'VERIFY_RESIZE' status +* Test action 8: Confirm resize VM1 +* Test action 9: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 2:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 10: Get VM1's host info DST_HOST +* **Test assertion 3:** Verify SRC_HOST does not equal to DST_HOST +* Test action 11: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to cold migrate VMs. Specifically, the test verifies that: + +* Servers can be cold migrated from one compute node to another computer node. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +----------------------------------- +Test Case 3 - Cold migration revert +----------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration_revert + +Test preconditions +------------------ + +* At least 2 compute nodes +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Get VM1's host info SRC_HOST +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 6: Cold migrate VM1 +* Test action 7: Wait for VM1 to reach 'VERIFY_RESIZE' status +* Test action 8: Revert resize VM1 +* Test action 9: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 2:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 10: Get VM1's host info DST_HOST +* **Test assertion 3:** Verify SRC_HOST equals to DST_HOST +* Test action 11: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to revert cold-migrated VMs. Specifically, the test verifies that: + +* Cold migrate operation can be reverted. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------- +Test Case 4 - Pause and unpause server +-------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_pause_unpause + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Pause VM1 +* Test action 5: Wait for VM1 to reach 'PAUSED' status +* **Test assertion 1:** Verify FIP1 status is 'ACTIVE' +* **Test assertion 2:** Verify ping FIP1 failed and SSH to VM1 via FIP1 failed +* Test action 6: Unpause VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 3:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 8: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to pause and unpause VMs. Specifically, the test verifies that: + +* When paused, servers cannot be reached. + +* When unpaused, servers can recover its reachability. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------- +Test Case 5 - Reboot server +--------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_reboot + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Soft reboot VM1 +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 6: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to reboot servers. Specifically, the test verifies that: + +* After reboot, servers can still be connected. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------- +Test Case 6 - Rebuild server +---------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_rebuild + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Rebuild VM1 with another image +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 6: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to rebuild servers. Specifically, the test verifies that: + +* Servers can be rebuilt with specific image correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------- +Test Case 7 - Resize server +--------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_resize + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Resize VM1 with another flavor +* Test action 5: Wait for VM1 to reach 'VERIFY_RESIZE' status +* Test action 6: Confirm resize VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 8: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to resize servers. Specifically, the test verifies that: + +* Servers can be resized with specific flavor correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +----------------------------------- +Test Case 8 - Stop and start server +----------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_stop_start + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Stop VM1 +* Test action 5: Wait for VM1 to reach 'SHUTOFF' status +* **Test assertion 1:** Verify ping FIP1 failed and SSH to VM1 via FIP1 failed +* Test action 6: Start VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 2:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 8: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to stop and start servers. Specifically, the test verifies that: + +* When stopped, servers cannot be reached. + +* When started, servers can recover its reachability. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------- +Test Case 9 - Suspend and resume server +--------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_suspend_resume + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Suspend VM1 +* Test action 5: Wait for VM1 to reach 'SUSPENDED' status +* **Test assertion 1:** Verify ping FIP1 failed and SSH to VM1 via FIP1 failed +* Test action 6: Resume VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 2:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 8: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to suspend and resume servers. Specifically, the test verifies that: + +* When suspended, servers cannot be reached. + +* When resumed, servers can recover its reachability. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------------------- +Test Case 10 - Suspend and resume server in sequence +---------------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_server_advanced_ops.TestServerAdvancedOps.test_server_sequence_suspend_resume + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a server VM1 +* Test action 2: Suspend VM1 +* Test action 3: Wait for VM1 to reach 'SUSPENDED' status +* **Test assertion 1:** Verify VM1's status is 'SUSPENDED' +* Test action 4: Resume VM1 +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 2:** Verify VM1's status is 'ACTIVE' +* Test action 6: Suspend VM1 +* Test action 7: Wait for VM1 to reach 'SUSPENDED' status +* **Test assertion 3:** Verify VM1 status is 'SUSPENDED' +* Test action 8: Resume VM1 +* Test action 9: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 4:** Verify VM1 status is 'ACTIVE' +* Test action 10: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to suspend and resume servers in sequence. +Specifically, the test verifies that: + +* Servers can be suspend and resume in sequence correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------ +Test Case 11 - Resize volume backed server +------------------------------------------ + +Test case specification +----------------------- + +tempest.scenario.test_server_advanced_ops.TestServerAdvancedOps.test_resize_volume_backed_server_confirm + +Test preconditions +------------------ + +* Nova, neutron, cinder services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a volume backed server VM1 +* Test action 2: Resize VM1 with another flavor +* Test action 3: Wait for VM1 to reach 'VERIFY_RESIZE' status +* Test action 4: Confirm resize VM1 +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** VM1's status is 'ACTIVE' +* Test action 6: Delete VM1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to resize volume backed servers. +Specifically, the test verifies that: + +* Volume backed servers can be resized with specific flavor correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +----------------------------------------- +Test Case 12 - Shelve and unshelve server +----------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_instance + +Test preconditions +------------------ + +* Nova, neutron, image services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a security group SG1, which has rules for allowing + incoming SSH and ICMP traffic +* Test action 3: Create a server with SG1 and KEYP1 +* Test action 4: Create a timestamp and store it in a file F1 inside VM1 +* Test action 5: Shelve VM1 +* Test action 6: Unshelve VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* Test action 8: Read F1 and compare if the read value and the previously written value + are the same or not +* **Test assertion 1:** Verify the values written and read are the same +* Test action 9: Delete SG1, KEYP1 and VM1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to shelve and unshelve servers. +Specifically, the test verifies that: + +* Servers can be shelved and unshelved correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------------------- +Test Case 13 - Shelve and unshelve volume backed server +------------------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_volume_backed_instance + +Test preconditions +------------------ + +* Nova, neutron, image, cinder services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a security group SG1, which has rules for allowing + incoming and outgoing SSH and ICMP traffic +* Test action 3: Create a volume backed server VM1 with SG1 and KEYP1 +* Test action 4: SSH to VM1 to create a timestamp T_STAMP1 and store it in a file F1 inside VM1 +* Test action 5: Shelve VM1 +* Test action 6: Unshelve VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* Test action 8: SSH to VM1 to read the timestamp T_STAMP2 stored in F1 +* **Test assertion 1:** Verify T_STAMP1 equals to T_STAMP2 +* Test action 9: Delete SG1, KEYP1 and VM1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to shelve and unshelve volume backed servers. +Specifically, the test verifies that: + +* Volume backed servers can be shelved and unshelved correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A |