Merge "Add test area 'correct behavior after common virtual machine life cycles events'"

1 files changed, 757 insertions, 0 deletions

diff --git a/docs/testing/user/testspecification/machinelifecycle/index.rst b/docs/testing/user/testspecification/machinelifecycle/index.rst
new file mode 100644
index 00000000..598812f5
--- /dev/null
+++ b/docs/testing/user/testspecification/machinelifecycle/index.rst
@@ -0,0 +1,757 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) Huawei Technologies Co.,Ltd
+
+===========================================================
+Common virtual machine life cycle events test specification
+===========================================================
+
+.. toctree::
+   :maxdepth: 2
+
+Scope
+=====
+
+The common virtual machine life cycle events test area evaluates the ability of
+the system under test to behave correctly after common virtual machine life
+cycle events. The tests in this test area will evaluate:
+
+- Stop/Start a server
+- Reboot a server
+- Rebuild a server
+- Pause/Unpause a server
+- Suspend/Resume a server
+- Resize a server
+- Resizing a volume-backed server
+- Sequence suspend resume
+- Shelve/Unshelve a server
+- Cold migrate a server
+- Live migrate a server
+
+References
+==========
+
+- iSCSI
+
+  - https://docs.openstack.org/liberty/config-reference/content/config-iscsi-storage.html
+
+Definitions and abbreviations
+=============================
+
+The following terms and abbreviations are used in conjunction with this test area
+
+- API - Application Programming Interface
+- NFVi - Network Functions Virtualization infrastructure
+- VIM - Virtual Infrastructure Manager
+- VM - Virtual Machine
+
+System Under Test (SUT)
+=======================
+
+The system under test is assumed to be the NFVi and VIM in operation on a
+Pharos compliant infrastructure.
+
+Test Area Structure
+===================
+
+The test area is structured based on common virtual machine life cycle events.
+Each test case is able to run independently, i.e. irrelevant of the state
+created by a previous test. Specifically, every test performs clean-up
+operations which return the system to the same state as before the test.
+
+Test Descriptions
+=================
+
+API Used and Reference
+----------------------
+
+Block storage: https://developer.openstack.org/api-ref/block-storage
+
+- create volume
+- delete volume
+- attach volume to server
+- detach volume from server
+
+Security Groups: https://developer.openstack.org/api-ref/network/v2/index.html#security-groups-security-groups
+
+- create security group
+- delete security group
+
+Networks: https://developer.openstack.org/api-ref/networking/v2/index.html#networks
+
+- create network
+- delete network
+
+Routers and interface: https://developer.openstack.org/api-ref/networking/v2/index.html#routers-routers
+
+- create router
+- delete router
+- add interface to router
+
+Subnets: https://developer.openstack.org/api-ref/networking/v2/index.html#subnets
+
+- create subnet
+- delete subnet
+
+Servers: https://developer.openstack.org/api-ref/compute/
+
+- create keypair
+- create server
+- show server
+- delete server
+- add/assign floating IP
+- resize server
+- revert resized server
+- confirm resized server
+- pause server
+- unpause server
+- start server
+- stop server
+- reboot server
+- rebuild server
+- suspend server
+- resume suspended server
+- shelve server
+- unshelve server
+- migrate server
+- live-migrate server
+
+Ports: https://developer.openstack.org/api-ref/networking/v2/index.html#ports
+
+- create port
+- delete port
+
+Floating IPs: https://developer.openstack.org/api-ref/networking/v2/index.html#floating-ips-floatingips
+
+- create floating IP
+- delete floating IP
+
+Availability zone: https://developer.openstack.org/api-ref/compute/
+
+- get availability zone
+
+------------------------------------
+Test Case 1 - Minimum basic scenario
+------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario
+
+Test preconditions
+------------------
+
+* Nova, cinder, glance, neutron services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create an image IMG1
+* Test action 2: Create a keypair KEYP1
+* Test action 3: Create a server VM1 with IMG1 and KEYP1
+* **Test assertion 1:** Verify VM1 is created successfully
+* Test action 4: Create a volume VOL1
+* **Test assertion 2:** Verify VOL1 is created successfully
+* Test action 5: Attach VOL1 to VM1
+* **Test assertion 3:** Verify VOL1's status has been updated after attached to VM1
+* Test action 6: Create a floating IP FIP1 and assign FIP1 to VM1
+* **Test assertion 4:** Verify VM1's addresses have been refreshed after associating FIP1
+* Test action 7: Create and add security group SG1 to VM1
+* **Test assertion 5:** Verify can SSH to VM1 via FIP1
+* Test action 8: Reboot VM1
+* **Test assertion 6:** Verify can SSH to VM1 via FIP1
+* **Test assertion 7:** Verify VM1's disk count equals to 1
+* Test action 9: Delete the floating IP FIP1 from VM1
+* **Test assertion 8:** Verify VM1's addresses have been refreshed after disassociating FIP1
+* Test action 10: Delete SG1, IMG1, KEYP1, VOL1, VM1 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates a minimum basic scenario. Specifically, the test verifies that:
+
+* The server can be connected before reboot.
+
+* The server can be connected after reboot.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+----------------------------
+Test Case 2 - Cold migration
+----------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration
+
+Test preconditions
+------------------
+
+* At least 2 compute nodes
+* Nova, neutron services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a keypair KEYP1
+* Test action 2: Create a server VM1 with KEYP1
+* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1
+* Test action 4: Get VM1's host info SRC_HOST
+* Test action 5: Wait for VM1 to reach 'ACTIVE' status
+* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1
+* Test action 6: Cold migrate VM1
+* Test action 7: Wait for VM1 to reach 'VERIFY_RESIZE' status
+* Test action 8: Confirm resize VM1
+* Test action 9: Wait for VM1 to reach 'ACTIVE' status
+* **Test assertion 2:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1
+* Test action 10: Get VM1's host info DST_HOST
+* **Test assertion 3:** Verify SRC_HOST does not equal to DST_HOST
+* Test action 11: Delete KEYP1, VM1 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability to cold migrate VMs. Specifically, the test verifies that:
+
+* Servers can be cold migrated from one compute node to another computer node.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+-----------------------------------
+Test Case 3 - Cold migration revert
+-----------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration_revert
+
+Test preconditions
+------------------
+
+* At least 2 compute nodes
+* Nova, neutron services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a keypair KEYP1
+* Test action 2: Create a server VM1 with KEYP1
+* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1
+* Test action 4: Get VM1's host info SRC_HOST
+* Test action 5: Wait for VM1 to reach 'ACTIVE' status
+* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1
+* Test action 6: Cold migrate VM1
+* Test action 7: Wait for VM1 to reach 'VERIFY_RESIZE' status
+* Test action 8: Revert resize VM1
+* Test action 9: Wait for VM1 to reach 'ACTIVE' status
+* **Test assertion 2:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1
+* Test action 10: Get VM1's host info DST_HOST
+* **Test assertion 3:** Verify SRC_HOST equals to DST_HOST
+* Test action 11: Delete KEYP1, VM1 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability to revert cold-migrated VMs. Specifically, the test verifies that:
+
+* Cold migrate operation can be reverted.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+--------------------------------------
+Test Case 4 - Pause and unpause server
+--------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_pause_unpause
+
+Test preconditions
+------------------
+
+* Nova, neutron services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a keypair KEYP1
+* Test action 2: Create a server VM1 with KEYP1
+* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1
+* Test action 4: Pause VM1
+* Test action 5: Wait for VM1 to reach 'PAUSED' status
+* **Test assertion 1:** Verify FIP1 status is 'ACTIVE'
+* **Test assertion 2:** Verify ping FIP1 failed and SSH to VM1 via FIP1 failed
+* Test action 6: Unpause VM1
+* Test action 7: Wait for VM1 to reach 'ACTIVE' status
+* **Test assertion 3:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1
+* Test action 8: Delete KEYP1, VM1 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability to pause and unpause VMs. Specifically, the test verifies that:
+
+* When paused, servers cannot be reached.
+
+* When unpaused, servers can recover its reachability.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+---------------------------
+Test Case 5 - Reboot server
+---------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_reboot
+
+Test preconditions
+------------------
+
+* Nova, neutron services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a keypair KEYP1
+* Test action 2: Create a server VM1 with KEYP1
+* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1
+* Test action 4: Soft reboot VM1
+* Test action 5: Wait for VM1 to reach 'ACTIVE' status
+* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1
+* Test action 6: Delete KEYP1, VM1 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability to reboot servers. Specifically, the test verifies that:
+
+* After reboot, servers can still be connected.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+----------------------------
+Test Case 6 - Rebuild server
+----------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_rebuild
+
+Test preconditions
+------------------
+
+* Nova, neutron services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a keypair KEYP1
+* Test action 2: Create a server VM1 with KEYP1
+* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1
+* Test action 4: Rebuild VM1 with another image
+* Test action 5: Wait for VM1 to reach 'ACTIVE' status
+* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1
+* Test action 6: Delete KEYP1, VM1 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability to rebuild servers. Specifically, the test verifies that:
+
+* Servers can be rebuilt with specific image correctly.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+---------------------------
+Test Case 7 - Resize server
+---------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_resize
+
+Test preconditions
+------------------
+
+* Nova, neutron services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a keypair KEYP1
+* Test action 2: Create a server VM1 with KEYP1
+* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1
+* Test action 4: Resize VM1 with another flavor
+* Test action 5: Wait for VM1 to reach 'VERIFY_RESIZE' status
+* Test action 6: Confirm resize VM1
+* Test action 7: Wait for VM1 to reach 'ACTIVE' status
+* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1
+* Test action 8: Delete KEYP1, VM1 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability to resize servers. Specifically, the test verifies that:
+
+* Servers can be resized with specific flavor correctly.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+-----------------------------------
+Test Case 8 - Stop and start server
+-----------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_stop_start
+
+Test preconditions
+------------------
+
+* Nova, neutron services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a keypair KEYP1
+* Test action 2: Create a server VM1 with KEYP1
+* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1
+* Test action 4: Stop VM1
+* Test action 5: Wait for VM1 to reach 'SHUTOFF' status
+* **Test assertion 1:** Verify ping FIP1 failed and SSH to VM1 via FIP1 failed
+* Test action 6: Start VM1
+* Test action 7: Wait for VM1 to reach 'ACTIVE' status
+* **Test assertion 2:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1
+* Test action 8: Delete KEYP1, VM1 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability to stop and start servers. Specifically, the test verifies that:
+
+* When stopped, servers cannot be reached.
+
+* When started, servers can recover its reachability.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+---------------------------------------
+Test Case 9 - Suspend and resume server
+---------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_suspend_resume
+
+Test preconditions
+------------------
+
+* Nova, neutron services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a keypair KEYP1
+* Test action 2: Create a server VM1 with KEYP1
+* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1
+* Test action 4: Suspend VM1
+* Test action 5: Wait for VM1 to reach 'SUSPENDED' status
+* **Test assertion 1:** Verify ping FIP1 failed and SSH to VM1 via FIP1 failed
+* Test action 6: Resume VM1
+* Test action 7: Wait for VM1 to reach 'ACTIVE' status
+* **Test assertion 2:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1
+* Test action 8: Delete KEYP1, VM1 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability to suspend and resume servers. Specifically, the test verifies that:
+
+* When suspended, servers cannot be reached.
+
+* When resumed, servers can recover its reachability.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+----------------------------------------------------
+Test Case 10 - Suspend and resume server in sequence
+----------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_server_advanced_ops.TestServerAdvancedOps.test_server_sequence_suspend_resume
+
+Test preconditions
+------------------
+
+* Nova, neutron services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a server VM1
+* Test action 2: Suspend VM1
+* Test action 3: Wait for VM1 to reach 'SUSPENDED' status
+* **Test assertion 1:** Verify VM1's status is 'SUSPENDED'
+* Test action 4: Resume VM1
+* Test action 5: Wait for VM1 to reach 'ACTIVE' status
+* **Test assertion 2:** Verify VM1's status is 'ACTIVE'
+* Test action 6: Suspend VM1
+* Test action 7: Wait for VM1 to reach 'SUSPENDED' status
+* **Test assertion 3:** Verify VM1 status is 'SUSPENDED'
+* Test action 8: Resume VM1
+* Test action 9: Wait for VM1 to reach 'ACTIVE' status
+* **Test assertion 4:** Verify VM1 status is 'ACTIVE'
+* Test action 10: Delete KEYP1, VM1 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability to suspend and resume servers in sequence.
+Specifically, the test verifies that:
+
+* Servers can be suspend and resume in sequence correctly.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+------------------------------------------
+Test Case 11 - Resize volume backed server
+------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_server_advanced_ops.TestServerAdvancedOps.test_resize_volume_backed_server_confirm
+
+Test preconditions
+------------------
+
+* Nova, neutron, cinder services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a volume backed server VM1
+* Test action 2: Resize VM1 with another flavor
+* Test action 3: Wait for VM1 to reach 'VERIFY_RESIZE' status
+* Test action 4: Confirm resize VM1
+* Test action 5: Wait for VM1 to reach 'ACTIVE' status
+* **Test assertion 1:** VM1's status is 'ACTIVE'
+* Test action 6: Delete VM1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability to resize volume backed servers.
+Specifically, the test verifies that:
+
+* Volume backed servers can be resized with specific flavor correctly.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+-----------------------------------------
+Test Case 12 - Shelve and unshelve server
+-----------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_instance
+
+Test preconditions
+------------------
+
+* Nova, neutron, image services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a keypair KEYP1
+* Test action 2: Create a security group SG1, which has rules for allowing
+  incoming SSH and ICMP traffic
+* Test action 3: Create a server with SG1 and KEYP1
+* Test action 4: Create a timestamp and store it in a file F1 inside VM1
+* Test action 5: Shelve VM1
+* Test action 6: Unshelve VM1
+* Test action 7: Wait for VM1 to reach 'ACTIVE' status
+* Test action 8: Read F1 and compare if the read value and the previously written value
+  are the same or not
+* **Test assertion 1:** Verify the values written and read are the same
+* Test action 9: Delete SG1, KEYP1 and VM1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability to shelve and unshelve servers.
+Specifically, the test verifies that:
+
+* Servers can be shelved and unshelved correctly.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+-------------------------------------------------------
+Test Case 13 - Shelve and unshelve volume backed server
+-------------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_volume_backed_instance
+
+Test preconditions
+------------------
+
+* Nova, neutron, image, cinder services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a keypair KEYP1
+* Test action 2: Create a security group SG1, which has rules for allowing
+  incoming and outgoing SSH and ICMP traffic
+* Test action 3: Create a volume backed server VM1 with SG1 and KEYP1
+* Test action 4: SSH to VM1 to create a timestamp T_STAMP1 and store it in a file F1 inside VM1
+* Test action 5: Shelve VM1
+* Test action 6: Unshelve VM1
+* Test action 7: Wait for VM1 to reach 'ACTIVE' status
+* Test action 8: SSH to VM1 to read the timestamp T_STAMP2 stored in F1
+* **Test assertion 1:** Verify T_STAMP1 equals to T_STAMP2
+* Test action 9: Delete SG1, KEYP1 and VM1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability to shelve and unshelve volume backed servers.
+Specifically, the test verifies that:
+
+* Volume backed servers can be shelved and unshelved correctly.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A