diff options
Diffstat (limited to 'docs/testing/user')
10 files changed, 4465 insertions, 97 deletions
diff --git a/docs/testing/user/certificationworkflow/ApplicationForm.rst b/docs/testing/user/certificationworkflow/ApplicationForm.rst index e69de29b..9199d71e 100644 --- a/docs/testing/user/certificationworkflow/ApplicationForm.rst +++ b/docs/testing/user/certificationworkflow/ApplicationForm.rst @@ -0,0 +1,10 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Intel Corporation and others. + +.._cvp-application-form: + +====================================================== +OPNFV COMPLIANCE VERIFICATION PROGRAM APPLICATION FORM +====================================================== + diff --git a/docs/testing/user/certificationworkflow/index.rst b/docs/testing/user/certificationworkflow/index.rst index e69de29b..d26fe5d8 100644 --- a/docs/testing/user/certificationworkflow/index.rst +++ b/docs/testing/user/certificationworkflow/index.rst @@ -0,0 +1,122 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Intel Corporation, Ericsson AB, Huawei, and others + +================================================================ +OPNFV Compliance Verification Program certification workflow +================================================================ + +.. toctree:: + :maxdepth: 2 + +Introduction +============ + +This document provides guidance for testers on how to obtain OPNFV compliance +certification. The OPNFV Compliance Verification Program (CVP) is administered by +the OPNFV Compliance and Certification (C&C) committee. + +For further information about the workflow and general inquiries about the +program, please check out the CVP web portal https://cvp.opnfv.org, or contact +the C&C committee by email address cvp@opnfv.org. This email address should be used +for all communication with the CVP. + +Step 1: Applying +================ + +A tester should start the process by completing an application. +The application form can found on the CVP portal and the following +information should be provided: + + - Organization name + - Organization website (if public) + - Product name and/or identifier + - Product specifications + - Product public documentation + - Product categories, choose one: (i) software and hardware (ii) software + and third party hardware (please specify) + - Primary contact name, business email, postal address and phone number + Only the primary email address should be used for + official communication with OPNFV CVP. + - User ID for CVP web portal + The CVP web portal supports (i) Linux Foundation user ID, and (ii) Openstack user + ID, in the current release. Specify either one as user ID for the CVP web portal. + If a new user ID is needed, visit https://identity.linuxfoundation.org, or + https://www.openstack.org/join/register. + - Location where the verification testing is to be conducted. Choose one: + (internal vendor lab, third-party lab) + - If the test is to be conducted by a third-party lab, please specify + name and contact information of the third-party lab, including email, address and + phone number. + +Please email the completed application using the primary contact email +account in order to establish identity. + +Once the application information is received and in order, an email response will be +sent to the primary contact with confirmation and information to proceed. + +[Editor's note: +No fee has been established at this time for CVP applications. Recommend +we skip fee for the initial release of CVP.] + +Step 2: Testing +=============== + +The following documents guide testers to prepare the test environment and run tests: + + - System Requirements and Preparation Guide for OPNFV CVP [2] + - OPNFV CVP Test Case Specifications [3] + - Dovetail Test Tool User Guide [4] + +[Editor's note: The above three documents are still work in progress. Names and +references will be corrected when they are merged.] + +A unique Test ID is generated by the Dovetail tool for each test run. Please take a note +of this ID for future reference. + +Step 3: Submitting Test Results +=============================== + +Testers can upload the test results to the CVP web portal https://cvp.opnfv.org. +By default, the results are visible only to the tester who uploaded the data. + +Testers can self-review the test results through the portal until they are ready to ask +for CVP review. They may also update with or add new test results as needed. + +Once the tester is satisfied with the test result, the tester grants access to the test result +for CVP review via the portal. The test result is identified by the unique Test ID. + +When a test result is made visible to the reviewers, the web portal will notify +cvp@opnfv.org and Cc the primary contact email that a review request has been made and reference +the Test ID. This will alert the C&C Committee to start the CVP review process. + +Step 4: CVP Review +=================== + +Upon receiving the email notification and the Test ID, the C&C Committee conducts a +peer based review of the test result. Persons employed by the same organization that submitted +the test results or by affiliated organizations will not be part of the reviewers. + +The primary contact may be asked via email for any missing information or +clarification of the application. The reviewers will make a determination and recommend +compliance or non-compliance to +the C&C Committee. Normally, the outcome of the review should be communicated +to the tester within 10 business days after all required information is in order. + +If an application is denied, an appeal can be made to the C&C Committee or ultimately to the +Board of Directors of OPNFV. + +Step 5: Grant of Use of Logo +============================ + +If an application is approved, further information will be communicated to the tester +on the guidelines of using OPNFV CVP logos and the status of compliance for promotional purposes. + +References +========== + +[1] The OPNFV CVP Guidelines v.16 [Editor's note: link to be provided.] +[2] System Requirements and Preparation Guide for OPNFV CVP [Editor's note: link to be provided.] +[3] OPNFV CVP Test Case Specifications v1.0 [Editor's note: link to be provided.] +[4] Dovetail Test Tool User Guide v1.0 [Editor's note: link to be provided.] + diff --git a/docs/testing/user/cvpaddendum/index.rst b/docs/testing/user/cvpaddendum/index.rst new file mode 100644 index 00000000..ee356ff9 --- /dev/null +++ b/docs/testing/user/cvpaddendum/index.rst @@ -0,0 +1,388 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Intel and others + +==================================================================== +Compliance Verification Program - Guidelines Addendum for Danube +==================================================================== + +.. toctree:: + :maxdepth: 2 + + +Introduction +============ + +This addendum provides a high-level description of the testing scope and +pass/fail criteria used in the Compliance Verification Program (CVP) for the +OPNFV Danube release. This information is intended as an overview for CVP +testers and for the Dovetail Project to help guide test-tool and test-case +development for the OPNFV Danube release. The Dovetail project is responsible for documenting +test-case specifications as well as implementing the CVP tool-chain through collaboration +with the OPNFV testing community. CVP testing focuses on establishing the +ability of the System Under Test (SUT) to perform NFVI and VIM operations and support +Service Provider oriented features that ensure manageable, resilient and secure +networks. + + +Meaning of Compliance +===================== + +OPNFV Compliance indicates adherence to NFV platform behavior defined as +various platform capabilities or features to prepare, instantiate, operate and remove +VNFs running on the NFVI. Danube compliance evaluates the ability of a platform +to support Service Provider network capabilities and workloads that are +supported in the OPNFV platform as of this release. Compliance test cases shall +be designated as compulsory or optional based on the maturity of OPNFV +capabilities as well as industry expectations. Compulsory test cases may for +example include NFVI management capabilities whereas tests for certain +high-availability features may be deemed as optional. + +Test coverage and pass/fail criteria are +designed to ensure an acceptable level of compliance but not be so restrictive +as to disqualify variations in platform implementations, capabilities and features. + + +SUT Assumptions +=============== + +Assumptions about the System Under Test (SUT) include ... + + - The minimal specification of physical infrastructure, including controller + nodes, compute nodes and networks, is defined by the Pharos specification + [2]. + - The SUT is fully deployed and operational, i.e. SUT deployment tools are + out of scope of testing. + + +Scope of Testing +================ + +The OPNFV CVP Guidelines [1], as approved by the Board of Directors, outlines +the key objectives of the CVP as follows: + - Help build the market for + - OPNFV based infrastructure + - applications designed to run on that infrastructure + - Reduce adoption risks for end-users + - Decrease testing costs by verifying hardware and software platform + interfaces and components + - Enhance interoperability + +The guidelines further directs the scope to be constrained to "features, +capabilities, components, and interfaces included in an OPNFV release that are +generally available in the industry (e.g., through adoption by an upstream +community)", and that compliance verification is evaluated using "functional tests +that focus on defined interfaces and/or behaviors without regard to the +the implementation of the underlying system under test". + +OPNFV provides a broad range of capabilities, including the reference platform itself +as well as tools-chains and methodologies for building infrastructures, and +deploying and testing the platform. +Not all these aspects are in scope for CVP and not all functions and +components are tested in the initial version of CVP. For example, the deployment tools +for the SUT and CI/CD toolchain are currently out of scope. +Similarly, performance benchmarking related testing is also out of scope or +for further study. Newer functional areas such as MANO (outside of APIs in the NFVI and +VIM) are still developing and are for future considerations. + +General Approach +---------------- + +In order to meet the above objectives for CVP, we aim to follow a general approach +by first identifying the overall requirements for all stake-holders, +then analyzing what OPNFV and the upstream communities can effectively test and verify +presently to derive an initial working scope for CVP, and to recommend what the +community should strive to achieve in future releases. + +The overall requirements for CVP can be cateorized by the basic cloud +capabilities representing common operations needed by basic VNFs, and additional +requirements for VNFs that go beyond the common cloud capabilities including +functional extensions, operational capabilities and additional carrier grade +requirements. + +For the basic NFV requirements, we will analyze the required test cases, +leverage or improve upon existing test cases in OPNFV projects +and upstream projects whenever we can, and bridge the gaps when we must, to meet +these basic requirements. + +We are not yet ready to support requirements +such as hardware portability, carrier grade performance, fault management and other +operational features, security, MANO and VNF verification. +These areas are being studied for future considerations. + +In some areas, we will start with a limited level of verification +initially, constrained by what community resources are able to support at this +time, but still serve a basic need that is not being fulfilled elsewhere. +In these areas, we bring significant value to the community we +serve by starting a new area of verification, breaking new ground and +expanding it in the future. + +In other areas, the functions being verified have yet to reach +wide adoption but are seen as important requirements in NFV, +or features are only needed for specific NFV use cases but +an industry consensus about the APIs and behaviors is still deemed beneficial. In such +cases, we plan to incorporate the test areas as optional. An optional test +area will not have to be run or passed in order to achieve compliance. +Optional tests provide an opportunity for vendors to demonstrate compliance with specific OPNFV +features beyond the mandatory test scope. + +Analysis of Scope +----------------- + +Following on these high level objectives and the identified general approach, +we seek to define the initial verification scope by the analysis summarized +in the following categories: + +1. Basic Cloud Capabilities + +The intent of these tests is to verify that the SUT has the required +capabilities that a basic VNF needs, and these capabilities are implemented +in a way that enables this basic VNF to run on any OPNFV compliant +deployment. + +A basic VNF can be thought of as a single virtual machine that is networked +and can perform the simplest network functions, for example, a simple forwarding +gateway, or a set of such virtual machines connected only by simple virtual network +services. Running such basic VNF leads to a set of common requirements, including: + - image management (Refstack testing Glance API) + - identity management (Refstack testing Keystone Identity API) + - virtual compute (Refstack testing Nova Compute API) + - virtual storage (Refstack testing Cinder API) + - virtual networks (Refstack testing Neutron Network API) + - forwarding packets through virtual networks in data path + - filtering packets based on security rules and port security in data path + - dynamic network runtime operations through the life of a VNF (e.g. attach/detach, + enable/disable, read stats) + - correct behavior after common virtual machine life cycles events (e.g. + suspend/resume, reboot, migrate) + - simple virtual machine resource scheduling on multiple nodes + +OPNFV mainly supports Openstack as the VIM up to the Danube release. The +VNFs used in the CVP program, and features in scope for the program which are +considered to be basic to all VNFs, require commercial Openstack distributions +to support a common basic level of cloud capabilities, and to be compliant +to a common specification for these capabilities. This requirement significantly +overlaps with Openstack community's Interop working group's goals, but they are not +identical. The CVP runs the Openstack Refstack-Compute test cases to verify +compliance to the basic common API requirements of cloud +management functions and VNF (as a VM) management for OPNFV. +Additional NFV specific requirements are added in network data path validation, +packet filtering by security group rules and port security, life cycle runtime events of +virtual networks, multiple networks in a topology, validation +of VNF's functional state after common lifecylce events including reboot, pause, +suspense, stop/start and cold migration. In addition, the +basic requirement also verifies that the SUT can allocate VNF resources based +on simple anti-affinity rules. + +The combined test cases help to ensure that these basic operations are always +supported by a compliant platform and they adhere to +a common standard to enable portability across OPNFV compliant platforms. + +2. NFV specific functional requirements + +NFV has functional requirements beyond the basic common cloud +capabilities, esp. in the networking area. Examples like SDNVPN, IPv6, SFC may +be considered additional NFV requirements beyond general purpose cloud +computing. These feature requirements expand beyond common Openstack (or other +VIM) requirements. OPNFV CVP will incorporate test cases to verify +compliance in these areas as they become mature. Because these extensions +may impose new API demands, maturity and industry adoption is a prerequisite for +making them a mandatory requirement for OPNFV compliance. At the time of Danube, +we have not identified a new functional area that is mandatory for CVP. +In the meantime, CVP +intends to offer tests in some of these areas as an optional extension of the test +report to be submitted for review, noting that passing these tests will not be +required to pass OPNFV compliance verification. + +SDNVPN is relevant due to the wide adoption of MPLS/BGP based VPNs in wide area +networks, which makes it necessary for data centers hosting VNFs to be able to +seamlessly interconnect with such networks. IPv6 is a high priority service provider +requirement to ease IP addressing and operational issues. SFC is also an important +NFV requirement, however its implementation has not yet been accepted or adopted +in the upstream at the time of Danube. + +3. High availability + +High availability is a common carrier grade requirement. Availability of a +platform involves many aspects of the SUT, for example hardware or lower layer +system failures or system overloads, and is also highly dependent on +configurations. The current OPNFV high availability verification focuses on +Openstack control service failures and resource overloads, and verifies service +continuity when the system encounters such failures or resource overloads, and +also verifies the system heals after a failure episode within a reasonable time +window. These service HA capabilities are commonly adopted in the industry +and should be a mandatory requirement. + +The current test cases in HA cover the basic area of failure and resource +overload conditions for a cloud platform's service availability, including all +of the basic cloud capability services, and basic compute and storage loads, +so it is a meaningful first step for CVP. We expect additional high availability +scenarios be extended in future releases. + +4. Resiliency + +Resiliency testing involves stressing the SUT and verifying its ability +to absorb stress conditions and still provide an acceptable level of service. +Resiliency is an important requirement for end-users. + +The OPNFV testing projects have started testing +OPNFV system resiliency in +the Danube release that can be used to provide limited coverage in this area. +However, this is a relatively new test methodology in OPNFV, additional study +and testing experiences are still needed. We defer the resilency testing to +future CVP releases. + +5. Security + +Security is among the top priorities as a carrier grade requirement by the +end-users. Some of the basic common functions, including virtual network isolation, +security groups, port security and role based access control are already covered as +part of the basic cloud capabilities that are verified in CVP. These test cases +however do not yet cover the basic required security capabilities expected of an end-user +deployment. It is an area that we should address in the near future, to define +a common set of requirements and develop test cases for verifying those +requirements. + +Another common requirement is security vulnerability scanning. +While the OPNFV security project integrated tools for security vulnerability +scanning, this has not been fully analyzed or exercised in Danube release. +This area needs further work to identify the required level of security for the +purpose of OPNFV in order to be integrated into the CVP. End-user inputs on +specific requirements in security is needed. + +6. Service assurance + +Service assurance (SA) is a broad area of concern for reliability of the NFVI/VIM +and VNFs, and depends upon multiple subsystems of an NFV platform for essential +information and control mechanisms. These subsystems include telemetry, fault management +(e.g. alarms), performance management, audits, and control mechanisms such as security +and configuration policies. + +The current Danube release implements some enabling capabilities in NFVI/VIM +such as telemetry, policy, and fault management. However, the specification of expected +system components, behavior and the test cases to verify them have not yet +been adequately developed. We will therefore not be testing this area at this time +but defer to future study. + +7. Use case testing + +More complex interactions among multiple VNFs and between VNFs and the cloud +platform can be better exercised through selected more realistic use cases. + +End-users see use case level testing as a significant tool in +verifying OPNFV compliance because it validates design patterns and support +for the types of NFVI features that users +care about. There are a lot of projects in OPNFV developing use cases +and sample VNFs, however most are still in early phase and require further +enhancements to become useful additions to the CVP. + +Many of these use case test cases do not add new API requirements to the SUT per se, but +exercise aspects of the SUT's functional capabilities in more complex ways. +Other use cases, such as SDNVPN, will require additional API level extension, +and to clearly separate the two, we will categorize the latter as +NFV specific functional requirements and not simply as use cases. + +Examples such as vIMS, or those which are not yet available +in Danube release e.g. vCPE, +will be valuable additions to the CVP. These use cases need to +be widely accepted, and since they are more complex, using these VNFs for CVP demands +higher level of community resources to implement, analyze and document these VNFs. + +Use case testing is not ready for CVP at the time of Danube, but can be incorporated +in Euphrates or as a future roadmap area. + +Finally, we take a preliminary look at future roadmap ideas that may be helpful +for the community to plan and pull resources around. + +8. Future CVP scope enhancements + +Some possible areas of enhancement for the CVP scope in subsequent releases include: + + - service assurance, as discussed above + - use case testing, as discussed above + - platform in-place upgrade + - API backward compatibility / micro-versioning + - workload migration + - multisite federation + - platform operational insights, e.g. telemetry, logging + - efficiency, e.g. hardware and energy footprint of the platform + - interoperability with workload automation platforms e.g. ONAP + +And enhancements may also be made to the existing test areas as well, +particularly those with limited coverage in this release. + +Summary of Test Scope +--------------------- + +The above analysis concludes with the following scope summarized below. +These tested areas represent significant advancement in the +direction to meet the CVP's objectives and end-user expectations, and is a +good basis for the initial phase of CVP. + +- Test Area: common cloud capabilities + - Openstack Refstack-compute test cases + Image, Identity, Compute, Network, Storage + - OPNFV-Functest/vPing, including both user data and ssh + - Port security and security groups + - VM lifecycle events + - VM networking + - VM resource scheduling + - Mandatory + +- Test Area: SDNVPN + - OPNFV-SDNVPN + - Optional + +- Test Area: IPv6 + - OPNFV-IPv6 + - Limited to overlay tests, v6Ping + - Optional + +- Test Area: High Availability + - OPNFV-HA + - OPNFV-Yardstick + - Limited to service continuity verification on control services + - Mandatory + +- Test Area: Resilency with Load + - OPNFV-Bottlenecks + - OPNFV-Yardstick + - Limited to compute resource load testing + - Optional + +Note 1: While the current scope of compliance includes functional verification +of certain performance-enhancing NFVI features, no performance measurements or +assessment of performance capabilities are included as of this release. + +Note 2: The SUT is limited to NFVI and VIM functions. While testing MANO +component capabilities is out of scope, certain APIs exposed towards MANO are +used by the current OPNFV compliance testing suite. MANO and other operational +elements may be part of the test infrastructure; for example used for workload +deployment and provisioning. + +Criteria for Awarding Compliance +================================ + +This section provides guidance on compliance criteia for each test area. The +criteria described here are high-level, detailed pass/fail metrics are +documented in Dovetail test specifications. + +1. All mandatory test cases must pass. + +Exceptions to this rule may be legitimate, e.g. due to imperfect test tools or +reasonable circumstances that we can not foresee. These exceptions must be +documented and accepted by the reviewers. + +2. Optional test cases are optional to run. Its test results, pass or fail, + do not impact compliance. + +Applicants who choose to run the optional test cases can include the results +of the optional test cases to highlight the additional compliance. + +References +========== + +[1] The OPNFV CVP Guidelines v.16 [Editor's note: link to be provided.] +[2] Pharos specification xxx [Editor's note: link to be provided.] + diff --git a/docs/testing/user/testspecification/dynamicnetwork/index.rst b/docs/testing/user/testspecification/dynamicnetwork/index.rst new file mode 100644 index 00000000..8fa084c7 --- /dev/null +++ b/docs/testing/user/testspecification/dynamicnetwork/index.rst @@ -0,0 +1,413 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Huawei Technologies Co.,Ltd + +===================================================== +Dynamic Network Runtime Operations test specification +===================================================== + +.. toctree:: + :maxdepth: 2 + +Scope +===== + +The dynamic network runtime operations test area evaluates the ability of the +system under test to support dynamic network runtime operations through the +life of a VNF (e.g. attach/detach, enable/disable, read stats). +The tests in this test area will evaluate IPv4 network runtime operations +functionality. These runtime operations includes hotpluging network interface, +detaching floating-ip from VM, attaching floating-ip to VM, updating subnet's +DNS, updating VM instance port admin state and updating router admin state. + +References +========== + +- DNS + + - https://docs.openstack.org/newton/networking-guide/config-dns-res.html + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test +area + +- API - Application Programming Interface +- DNS - Domain Name System +- ICMP - Internet Control Message Protocol +- MAC - Media Access Control +- NIC - Network Interface Controller +- NFVi - Network Functions Virtualization infrastructure +- SSH - Secure Shell +- TCP - Transmission Control Protocol +- VIM - Virtual Infrastructure Manager +- VM - Virtual Machine + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + +Test Area Structure +=================== + +The test area is structured based on dynamic network runtime operations. Each +test case is able to run independently, i.e. irrelevant of the state created by +a previous test. Specifically, every test performs clean-up operations which +return the system to the same state as before the test. + +Test Descriptions +================= + +API Used and Reference +---------------------- + +Security Groups: https://developer.openstack.org/api-ref/network/v2/index.html#security-groups-security-groups + +- create security group +- delete security group + +Networks: https://developer.openstack.org/api-ref/networking/v2/index.html#networks + +- create network +- delete network + +Routers and interface: https://developer.openstack.org/api-ref/networking/v2/index.html#routers-routers + +- create router +- update router +- delete router +- add interface to router + +Subnets: https://developer.openstack.org/api-ref/networking/v2/index.html#subnets + +- create subnet +- update subnet +- delete subnet + +Servers: https://developer.openstack.org/api-ref/compute/ + +- create keypair +- create server +- delete server +- add/assign floating IP +- disassociate floating IP + +Ports: https://developer.openstack.org/api-ref/networking/v2/index.html#ports + +- create port +- update port +- delete port + +Floating IPs: https://developer.openstack.org/api-ref/networking/v2/index.html#floating-ips-floatingips + +- create floating IP +- delete floating IP + +-------------------------------------- +Test Case 1 - Basic network operations +-------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops + +Test preconditions +------------------ + +* Nova has been configured to boot VMs with Neutron-managed networking +* Openstack nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a security group SG1, which has rules for allowing + incoming SSH and ICMP traffic +* Test action 2: Create a neutron network NET1 +* Test action 3: Create a tenant router R1 which routes traffic to public network +* Test action 4: Create a subnet SUBNET1 and add it as router interface +* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating + IP FIP1 (via R1) to VM1 +* **Test assertion 1:** Ping FIP1 and SSH to VM1 via FIP1 successfully +* **Test assertion 2:** Ping the internal gateway from VM1 successfully +* **Test assertion 3:** Ping the default gateway from VM1 using its floating IP + FIP1 successfully +* Test action 6: Detach FIP1 from VM1 +* **Test assertion 4:** VM1 becomes unreachable after FIP1 disassociated +* Test action 7: Create a new server VM2 with NET1, and associate floating IP FIP1 to VM2 +* **Test assertion 5:** Ping FIP1 and SSH to VM2 via FIP1 successfully +* Test action 8: Delete SG1, NET1, SUBNET1, R1, VM1, VM2 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of basic network operations. +Specifically, the test verifies that: + +* The Tempest host can ping VM's IP address. This implies, but does not + guarantee (see the ssh check that follows), that the VM has been assigned the + correct IP address and has connectivity to the Tempest host. + +* The Tempest host can perform key-based authentication to an ssh server hosted + at VM's IP address. This check guarantees that the IP address is associated + with the target VM. + +* The Tempest host can ssh into the VM via the IP address and successfully ping + the internal gateway address, implying connectivity to another VM on the same network. + +* The Tempest host can ssh into the VM via the IP address and successfully ping + the default gateway, implying external connectivity. + +* Detach the floating-ip from the VM and VM becomes unreachable. + +* Associate attached floating ip to a new VM and the new VM is reachable. + +* Floating IP status is updated correctly after each change. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------- +Test Case 2 - Hotplug network interface +--------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic + +Test preconditions +------------------ + +* Nova has been configured to boot VMs with Neutron-managed networking +* Compute interface_attach feature is enabled +* VM vnic_type is not set to 'direct' or 'macvtap' +* Openstack nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a security group SG1, which has rules for allowing + incoming SSH and ICMP traffic +* Test action 2: Create a neutron network NET1 +* Test action 3: Create a tenant router R1 which routes traffic to public network +* Test action 4: Create a subnet SUBNET1 and add it as router interface +* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating + IP FIP1 (via R1) to VM1 +* **Test assertion 1:** Ping FIP1 and SSH to VM1 with FIP1 successfully +* Test action 6: Create a second neutron network NET2 and subnet SUBNET2, and + attach VM1 to NET2 +* Test action 7: Get VM1's ethernet interface NIC2 for NET2 +* **Test assertion 2:** Ping NET2's internal gateway successfully +* Test action 8: Delete SG1, NET1, NET2, SUBNET1, SUBNET2, R1, NIC2, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of adding network to an active VM. +Specifically, the test verifies that: + +* New network interface can be added to an existing VM successfully. + +* The Tempest host can ssh into the VM via the IP address and successfully ping + the new network's internal gateway address, implying connectivity to the new network. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------- +Test Case 3 - Update subnet's configuration +------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details + +Test preconditions +------------------ + +* Nova has been configured to boot VMs with Neutron-managed networking +* DHCP client is available +* Tenant networks should be non-shared and isolated +* Openstack nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a security group SG1, which has rules for allowing + incoming SSH and ICMP traffic +* Test action 2: Create a neutron network NET1 +* Test action 3: Create a tenant router R1 which routes traffic to public network +* Test action 4: Create a subnet SUBNET1 and add it as router interface, + configure SUBNET1 with dns nameserver '1.2.3.4' +* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating + IP FIP1 (via R1) to VM1 +* **Test assertion 1:** Ping FIP1 and SSH to VM1 with FIP1 successfully +* **Test assertion 2:** Retrieve the VM1's configured dns and verify it matches + the one configured for SUBNET1 +* Test action 6: Update SUBNET1's dns to '9.8.7.6' +* **Test assertion 3:** After triggering the DHCP renew from the VM manually, + retrieve the VM1's configured dns and verify it has been successfully updated +* Test action 7: Delete SG1, NET1, SUBNET1, R1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of updating subnet's configurations. +Specifically, the test verifies that: + +* Updating subnet's DNS server configurations are affecting the VMs. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------- +Test Case 4 - Update VM port admin state +---------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_instance_port_admin_state + +Test preconditions +------------------ + +* Nova has been configured to boot VMs with Neutron-managed networking +* Network port_admin_state_change feature is enabled +* Openstack nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a security group SG1, which has rules for allowing + incoming SSH and ICMP traffic +* Test action 2: Create a neutron network NET1 +* Test action 3: Create a tenant router R1 which routes traffic to public network +* Test action 4: Create a subnet SUBNET1 and add it as router interface +* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating + IP FIP1 (via R1) to VM1 +* Test action 6: Create a server VM2 with SG1 and NET1, and assign a floating + IP FIP2 to VM2 +* Test action 7: Get a SSH client SSHCLNT1 to VM2 +* **Test assertion 1:** Ping FIP1 and SSH to VM1 with FIP1 successfully +* **Test assertion 2:** Ping FIP1 via SSHCLNT1 successfully +* Test action 8: Update admin_state_up attribute of VM1 port to False +* **Test assertion 3:** Ping FIP1 and SSH to VM1 with FIP1 failed +* **Test assertion 4:** Ping FIP1 via SSHCLNT1 failed +* Test action 9: Update admin_state_up attribute of VM1 port to True +* **Test assertion 5:** Ping FIP1 and SSH to VM1 with FIP1 successfully +* **Test assertion 6:** Ping FIP1 via SSHCLNT1 successfully +* Test action 10: Delete SG1, NET1, SUBNET1, R1, SSHCLNT1, VM1, VM2 and FIP1, FIP2 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the VM public and project connectivity status by changing VM port +admin_state_up to True & False. Specifically, the test verifies that: + +* Public and project connectivity is reachable before updating admin_state_up + attribute of VM port to False. + +* Public and project connectivity is unreachable after updating admin_state_up + attribute of VM port to False. + +* Public and project connectivity is reachable after updating admin_state_up + attribute of VM port from False to True. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------- +Test Case 5 - Update router admin state +--------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state + +Test preconditions +------------------ + +* Nova has been configured to boot VMs with Neutron-managed networking +* Multi-tenant networks capabilities +* Openstack nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a security group SG1, which has rules for allowing + incoming SSH and ICMP traffic +* Test action 2: Create a neutron network NET1 +* Test action 3: Create a tenant router R1 which routes traffic to public network +* Test action 4: Create a subnet SUBNET1 and add it as router interface +* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating + IP FIP1 (via R1) to VM1 +* **Test assertion 1:** Ping FIP1 and SSH to VM1 with FIP1 successfully +* Test action 6: Update admin_state_up attribute of R1 to False +* **Test assertion 2:** Ping FIP1 and SSH to VM1 with FIP1 failed +* Test action 7: Update admin_state_up attribute of R1 to True +* **Test assertion 3:** Ping FIP1 and SSH to VM1 with FIP1 successfully +* Test action 8: Delete SG1, NET1, SUBNET1, R1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the router public connectivity status by changing +router admin_state_up to True & False. +Specifically, the test verifies that: + +* Public connectivity is reachable before updating admin_state_up attribute of router to False. + +* Public connectivity is unreachable after updating admin_state_up attribute of router to False. + +* Public connectivity is reachable after updating admin_state_up attribute of router. + from False to True + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A diff --git a/docs/testing/user/testspecification/machinelifecycle/index.rst b/docs/testing/user/testspecification/machinelifecycle/index.rst new file mode 100644 index 00000000..598812f5 --- /dev/null +++ b/docs/testing/user/testspecification/machinelifecycle/index.rst @@ -0,0 +1,757 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Huawei Technologies Co.,Ltd + +=========================================================== +Common virtual machine life cycle events test specification +=========================================================== + +.. toctree:: + :maxdepth: 2 + +Scope +===== + +The common virtual machine life cycle events test area evaluates the ability of +the system under test to behave correctly after common virtual machine life +cycle events. The tests in this test area will evaluate: + +- Stop/Start a server +- Reboot a server +- Rebuild a server +- Pause/Unpause a server +- Suspend/Resume a server +- Resize a server +- Resizing a volume-backed server +- Sequence suspend resume +- Shelve/Unshelve a server +- Cold migrate a server +- Live migrate a server + +References +========== + +- iSCSI + + - https://docs.openstack.org/liberty/config-reference/content/config-iscsi-storage.html + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test area + +- API - Application Programming Interface +- NFVi - Network Functions Virtualization infrastructure +- VIM - Virtual Infrastructure Manager +- VM - Virtual Machine + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + +Test Area Structure +=================== + +The test area is structured based on common virtual machine life cycle events. +Each test case is able to run independently, i.e. irrelevant of the state +created by a previous test. Specifically, every test performs clean-up +operations which return the system to the same state as before the test. + +Test Descriptions +================= + +API Used and Reference +---------------------- + +Block storage: https://developer.openstack.org/api-ref/block-storage + +- create volume +- delete volume +- attach volume to server +- detach volume from server + +Security Groups: https://developer.openstack.org/api-ref/network/v2/index.html#security-groups-security-groups + +- create security group +- delete security group + +Networks: https://developer.openstack.org/api-ref/networking/v2/index.html#networks + +- create network +- delete network + +Routers and interface: https://developer.openstack.org/api-ref/networking/v2/index.html#routers-routers + +- create router +- delete router +- add interface to router + +Subnets: https://developer.openstack.org/api-ref/networking/v2/index.html#subnets + +- create subnet +- delete subnet + +Servers: https://developer.openstack.org/api-ref/compute/ + +- create keypair +- create server +- show server +- delete server +- add/assign floating IP +- resize server +- revert resized server +- confirm resized server +- pause server +- unpause server +- start server +- stop server +- reboot server +- rebuild server +- suspend server +- resume suspended server +- shelve server +- unshelve server +- migrate server +- live-migrate server + +Ports: https://developer.openstack.org/api-ref/networking/v2/index.html#ports + +- create port +- delete port + +Floating IPs: https://developer.openstack.org/api-ref/networking/v2/index.html#floating-ips-floatingips + +- create floating IP +- delete floating IP + +Availability zone: https://developer.openstack.org/api-ref/compute/ + +- get availability zone + +------------------------------------ +Test Case 1 - Minimum basic scenario +------------------------------------ + +Test case specification +----------------------- + +tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario + +Test preconditions +------------------ + +* Nova, cinder, glance, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create an image IMG1 +* Test action 2: Create a keypair KEYP1 +* Test action 3: Create a server VM1 with IMG1 and KEYP1 +* **Test assertion 1:** Verify VM1 is created successfully +* Test action 4: Create a volume VOL1 +* **Test assertion 2:** Verify VOL1 is created successfully +* Test action 5: Attach VOL1 to VM1 +* **Test assertion 3:** Verify VOL1's status has been updated after attached to VM1 +* Test action 6: Create a floating IP FIP1 and assign FIP1 to VM1 +* **Test assertion 4:** Verify VM1's addresses have been refreshed after associating FIP1 +* Test action 7: Create and add security group SG1 to VM1 +* **Test assertion 5:** Verify can SSH to VM1 via FIP1 +* Test action 8: Reboot VM1 +* **Test assertion 6:** Verify can SSH to VM1 via FIP1 +* **Test assertion 7:** Verify VM1's disk count equals to 1 +* Test action 9: Delete the floating IP FIP1 from VM1 +* **Test assertion 8:** Verify VM1's addresses have been refreshed after disassociating FIP1 +* Test action 10: Delete SG1, IMG1, KEYP1, VOL1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates a minimum basic scenario. Specifically, the test verifies that: + +* The server can be connected before reboot. + +* The server can be connected after reboot. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------- +Test Case 2 - Cold migration +---------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration + +Test preconditions +------------------ + +* At least 2 compute nodes +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Get VM1's host info SRC_HOST +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 6: Cold migrate VM1 +* Test action 7: Wait for VM1 to reach 'VERIFY_RESIZE' status +* Test action 8: Confirm resize VM1 +* Test action 9: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 2:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 10: Get VM1's host info DST_HOST +* **Test assertion 3:** Verify SRC_HOST does not equal to DST_HOST +* Test action 11: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to cold migrate VMs. Specifically, the test verifies that: + +* Servers can be cold migrated from one compute node to another computer node. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +----------------------------------- +Test Case 3 - Cold migration revert +----------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration_revert + +Test preconditions +------------------ + +* At least 2 compute nodes +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Get VM1's host info SRC_HOST +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 6: Cold migrate VM1 +* Test action 7: Wait for VM1 to reach 'VERIFY_RESIZE' status +* Test action 8: Revert resize VM1 +* Test action 9: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 2:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 10: Get VM1's host info DST_HOST +* **Test assertion 3:** Verify SRC_HOST equals to DST_HOST +* Test action 11: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to revert cold-migrated VMs. Specifically, the test verifies that: + +* Cold migrate operation can be reverted. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------- +Test Case 4 - Pause and unpause server +-------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_pause_unpause + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Pause VM1 +* Test action 5: Wait for VM1 to reach 'PAUSED' status +* **Test assertion 1:** Verify FIP1 status is 'ACTIVE' +* **Test assertion 2:** Verify ping FIP1 failed and SSH to VM1 via FIP1 failed +* Test action 6: Unpause VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 3:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 8: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to pause and unpause VMs. Specifically, the test verifies that: + +* When paused, servers cannot be reached. + +* When unpaused, servers can recover its reachability. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------- +Test Case 5 - Reboot server +--------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_reboot + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Soft reboot VM1 +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 6: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to reboot servers. Specifically, the test verifies that: + +* After reboot, servers can still be connected. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------- +Test Case 6 - Rebuild server +---------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_rebuild + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Rebuild VM1 with another image +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 6: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to rebuild servers. Specifically, the test verifies that: + +* Servers can be rebuilt with specific image correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------- +Test Case 7 - Resize server +--------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_resize + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Resize VM1 with another flavor +* Test action 5: Wait for VM1 to reach 'VERIFY_RESIZE' status +* Test action 6: Confirm resize VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 8: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to resize servers. Specifically, the test verifies that: + +* Servers can be resized with specific flavor correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +----------------------------------- +Test Case 8 - Stop and start server +----------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_stop_start + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Stop VM1 +* Test action 5: Wait for VM1 to reach 'SHUTOFF' status +* **Test assertion 1:** Verify ping FIP1 failed and SSH to VM1 via FIP1 failed +* Test action 6: Start VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 2:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 8: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to stop and start servers. Specifically, the test verifies that: + +* When stopped, servers cannot be reached. + +* When started, servers can recover its reachability. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------- +Test Case 9 - Suspend and resume server +--------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_suspend_resume + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a server VM1 with KEYP1 +* Test action 3: Create a floating IP FIP1 and assign FIP1 to VM1 +* Test action 4: Suspend VM1 +* Test action 5: Wait for VM1 to reach 'SUSPENDED' status +* **Test assertion 1:** Verify ping FIP1 failed and SSH to VM1 via FIP1 failed +* Test action 6: Resume VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 2:** Verify can ping FIP1 successfully and can SSH to VM1 via FIP1 +* Test action 8: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to suspend and resume servers. Specifically, the test verifies that: + +* When suspended, servers cannot be reached. + +* When resumed, servers can recover its reachability. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------------------- +Test Case 10 - Suspend and resume server in sequence +---------------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_server_advanced_ops.TestServerAdvancedOps.test_server_sequence_suspend_resume + +Test preconditions +------------------ + +* Nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a server VM1 +* Test action 2: Suspend VM1 +* Test action 3: Wait for VM1 to reach 'SUSPENDED' status +* **Test assertion 1:** Verify VM1's status is 'SUSPENDED' +* Test action 4: Resume VM1 +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 2:** Verify VM1's status is 'ACTIVE' +* Test action 6: Suspend VM1 +* Test action 7: Wait for VM1 to reach 'SUSPENDED' status +* **Test assertion 3:** Verify VM1 status is 'SUSPENDED' +* Test action 8: Resume VM1 +* Test action 9: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 4:** Verify VM1 status is 'ACTIVE' +* Test action 10: Delete KEYP1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to suspend and resume servers in sequence. +Specifically, the test verifies that: + +* Servers can be suspend and resume in sequence correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------ +Test Case 11 - Resize volume backed server +------------------------------------------ + +Test case specification +----------------------- + +tempest.scenario.test_server_advanced_ops.TestServerAdvancedOps.test_resize_volume_backed_server_confirm + +Test preconditions +------------------ + +* Nova, neutron, cinder services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a volume backed server VM1 +* Test action 2: Resize VM1 with another flavor +* Test action 3: Wait for VM1 to reach 'VERIFY_RESIZE' status +* Test action 4: Confirm resize VM1 +* Test action 5: Wait for VM1 to reach 'ACTIVE' status +* **Test assertion 1:** VM1's status is 'ACTIVE' +* Test action 6: Delete VM1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to resize volume backed servers. +Specifically, the test verifies that: + +* Volume backed servers can be resized with specific flavor correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +----------------------------------------- +Test Case 12 - Shelve and unshelve server +----------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_instance + +Test preconditions +------------------ + +* Nova, neutron, image services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a security group SG1, which has rules for allowing + incoming SSH and ICMP traffic +* Test action 3: Create a server with SG1 and KEYP1 +* Test action 4: Create a timestamp and store it in a file F1 inside VM1 +* Test action 5: Shelve VM1 +* Test action 6: Unshelve VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* Test action 8: Read F1 and compare if the read value and the previously written value + are the same or not +* **Test assertion 1:** Verify the values written and read are the same +* Test action 9: Delete SG1, KEYP1 and VM1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to shelve and unshelve servers. +Specifically, the test verifies that: + +* Servers can be shelved and unshelved correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------------------- +Test Case 13 - Shelve and unshelve volume backed server +------------------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_volume_backed_instance + +Test preconditions +------------------ + +* Nova, neutron, image, cinder services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 +* Test action 2: Create a security group SG1, which has rules for allowing + incoming and outgoing SSH and ICMP traffic +* Test action 3: Create a volume backed server VM1 with SG1 and KEYP1 +* Test action 4: SSH to VM1 to create a timestamp T_STAMP1 and store it in a file F1 inside VM1 +* Test action 5: Shelve VM1 +* Test action 6: Unshelve VM1 +* Test action 7: Wait for VM1 to reach 'ACTIVE' status +* Test action 8: SSH to VM1 to read the timestamp T_STAMP2 stored in F1 +* **Test assertion 1:** Verify T_STAMP1 equals to T_STAMP2 +* Test action 9: Delete SG1, KEYP1 and VM1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to shelve and unshelve volume backed servers. +Specifically, the test verifies that: + +* Volume backed servers can be shelved and unshelved correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A diff --git a/docs/testing/user/testspecification/multiplenodes/index.rst b/docs/testing/user/testspecification/multiplenodes/index.rst new file mode 100644 index 00000000..72a131a3 --- /dev/null +++ b/docs/testing/user/testspecification/multiplenodes/index.rst @@ -0,0 +1,370 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Huawei Technologies Co.,Ltd + +=========================================================== +VM Resource Scheduling on Multiple Nodes test specification +=========================================================== + +.. toctree:: + :maxdepth: 2 + +Scope +===== + +The VM resource scheduling test area evaluates the ability of the system under +test to support VM resource scheduling on multiple nodes. +The tests in this test area will evaluate capabilities to schedule VM to multiple +compute nodes directly with scheduler hints, and create server groups with policy +affinity and anti-affinity. + +References +========== + +- Availability zone + + - https://docs.openstack.org/newton/networking-guide/config-az.html + +- Scheduling + + - https://docs.openstack.org/kilo/config-reference/content/section_compute-scheduler.html + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test area + +- API - Application Programming Interface +- NFVi - Network Functions Virtualization infrastructure +- VIM - Virtual Infrastructure Manager +- VM - Virtual Machine + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + +Test Area Structure +=================== + +The test area is structured based on server group operations and server operations +on multiple nodes. Each test case is able to run independently, i.e. irrelevant of +the state created by a previous test. Specifically, every test performs clean-up +operations which return the system to the same state as before the test. + +Test Descriptions +================= + +API Used and Reference +---------------------- + +Security Groups: https://developer.openstack.org/api-ref/network/v2/index.html#security-groups-security-groups + +- create security group +- delete security group + +Networks: https://developer.openstack.org/api-ref/networking/v2/index.html#networks + +- create network +- delete network + +Routers and interface: https://developer.openstack.org/api-ref/networking/v2/index.html#routers-routers + +- create router +- delete router +- add interface to router + +Subnets: https://developer.openstack.org/api-ref/networking/v2/index.html#subnets + +- create subnet +- delete subnet + +Servers: https://developer.openstack.org/api-ref/compute/ + +- create keypair +- create server +- show server +- delete server +- add/assign floating IP +- create server group +- delete server group +- list server groups +- show server group details + +Ports: https://developer.openstack.org/api-ref/networking/v2/index.html#ports + +- create port +- delete port + +Floating IPs: https://developer.openstack.org/api-ref/networking/v2/index.html#floating-ips-floatingips + +- create floating IP +- delete floating IP + +Availability zone: https://developer.openstack.org/api-ref/compute/ + +- get availability zone + +------------------------------------------ +Test Case 1 - Schedule VM to compute nodes +------------------------------------------ + +Test case specification +----------------------- + +tempest.scenario.test_server_multinode.TestServerMultinode.test_schedule_to_all_nodes + +Test preconditions +------------------ + +* At least 2 compute nodes +* Openstack nova, neutron services are available +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Get all availability zones AZONES1 in the SUT +* Test action 2: Get all compute nodes in AZONES1 +* Test action 3: Get the value of 'min_compute_nodes' which is set by user in tempest + config file and means the minimum number of compute nodes expected +* **Test assertion 1:** Verify that SUT has at least as many compute nodes as + specified by the 'min_compute_nodes' threshold +* Test action 4: Create one server for each compute node, up to the 'min_compute_nodes' threshold +* **Test assertion 2:** Verify the number of servers matches the 'min_compute_nodes' threshold +* Test action 5: Get every server's 'hostId' and store them in a set which has no duplicate values +* **Test assertion 3:** Verify the length of the set equals to the number of servers to ensure + that every server ended up on a different host +* Test action 6: Delete the created servers + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of VM resource scheduling. +Specifically, the test verifies that: + +* VMs are scheduled to the requested compute nodes correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------------------------------------------------- +Test Case 2 - Test create and delete multiple server groups with same name and policy +------------------------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.servers.test_server_group.ServerGroupTestJSON.test_create_delete_multiple_server_groups_with_same_name_policy + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Generate a random name N1 +* Test action 2: Create a server group SERG1 with N1 and policy affinity +* Test action 3: Create another server group SERG2 with N1 and policy affinity +* **Test assertion 1:** The names of SERG1 and SERG2 are the same +* **Test assertion 2:** The 'policies' of SERG1 and SERG2 are the same +* **Test assertion 3:** The ids of SERG1 and SERG2 are different +* Test action 4: Delete SERG1 and SERG2 +* Test action 5: List all server groups +* **Test assertion 4:** SERG1 and SERG2 are not in the list + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of creating and deleting server groups with the same name and policy. +Specifically, the test verifies that: + +* Server groups can be created with the same name and policy. +* Server groups with the same name and policy can be deleted successfully. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------------------------------------- +Test Case 3 - Test create and delete server group with affinity policy +---------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.servers.test_server_group.ServerGroupTestJSON.test_create_delete_server_group_with_affinity_policy + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Generate a random name N1 +* Test action 2: Create a server group SERG1 with N1 and policy affinity +* **Test assertion 1:** The name of SERG1 returned in the response is the same as N1 +* **Test assertion 2:** The 'policies' of SERG1 returned in the response is affinity +* Test action 3: Delete SERG1 and list all server groups +* **Test assertion 3:** SERG1 is not in the list + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of creating and deleting server group with affinity policy. +Specifically, the test verifies that: + +* Server group can be created with affinity policy and deleted successfully. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------------------------------------------- +Test Case 4 - Test create and delete server group with anti-affinity policy +--------------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.servers.test_server_group.ServerGroupTestJSON.test_create_delete_server_group_with_anti_affinity_policy + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Generate a random name N1 +* Test action 2: Create a server group SERG1 with N1 and policy anti-affinity +* **Test assertion 1:** The name of SERG1 returned in the response is the same as N1 +* **Test assertion 2:** The 'policies' of SERG1 returned in the response is anti-affinity +* Test action 3: Delete SERG1 and list all server groups +* **Test assertion 3:** SERG1 is not in the list + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of creating and deleting server group with anti-affinity policy. +Specifically, the test verifies that: + +* Server group can be created with anti-affinity policy and deleted successfully. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------- +Test Case 5 - Test list server groups +------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.servers.test_server_group.ServerGroupTestJSON.test_list_server_groups + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Generate a random name N1 +* Test action 2: Create a server group SERG1 with N1 and policy affinity +* Test action 3: List all server groups +* **Test assertion 1:** SERG1 is in the list +* Test action 4: Delete SERG1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of listing server groups. +Specifically, the test verifies that: + +* Server groups can be listed successfully. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------- +Test Case 6 - Test show server group details +-------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.servers.test_server_group.ServerGroupTestJSON.test_show_server_group + +Test preconditions +------------------ + +None + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Generate a random name N1 +* Test action 2: Create a server group SERG1 with N1 and policy affinity, and stored + the details (D1) returned in the response +* Test action 3: Show the details (D2) of SERG1 +* **Test assertion 1:** All values in D1 are the same as the values in D2 +* Test action 4: Delete SERG1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of showing server group details. +Specifically, the test verifies that: + +* Server groups can be shown successfully. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A diff --git a/docs/testing/user/testspecification/securitygroup/index.rst b/docs/testing/user/testspecification/securitygroup/index.rst new file mode 100644 index 00000000..0621b84d --- /dev/null +++ b/docs/testing/user/testspecification/securitygroup/index.rst @@ -0,0 +1,450 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Huawei Technologies Co.,Ltd + +=================================================== +Security Group and Port Security test specification +=================================================== + +.. toctree:: + :maxdepth: 2 + +Scope +===== + +The security group and port security test area evaluates the ability of the +system under test to support packet filtering by security group and port security. +The tests in this test area will evaluate preventing MAC spoofing by port security, +basic security group operations including testing cross/in tenant traffic, testing +multiple security groups, using port security to disable security groups and +updating security groups. + +References +========== + +N/A + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test +area + +- API - Application Programming Interface +- ICMP - Internet Control Message Protocol +- MAC - Media Access Control +- NFVi - Network Functions Virtualization infrastructure +- SSH - Secure Shell +- TCP - Transmission Control Protocol +- VIM - Virtual Infrastructure Manager +- VM - Virtual Machine + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + +Test Area Structure +=================== + +The test area is structured based on the basic operations of security group and +port security. Each test case is able to run independently, i.e. irrelevant of +the state created by a previous test. Specifically, every test performs clean-up +operations which return the system to the same state as before the test. + +Test Descriptions +================= + +API Used and Reference +---------------------- + +Security Groups: https://developer.openstack.org/api-ref/network/v2/index.html#security-groups-security-groups + +- create security group +- delete security group + +Networks: https://developer.openstack.org/api-ref/networking/v2/index.html#networks + +- create network +- delete network +- list networks +- create floating ip +- delete floating ip + +Routers and interface: https://developer.openstack.org/api-ref/networking/v2/index.html#routers-routers + +- create router +- delete router +- list routers +- add interface to router + +Subnets: https://developer.openstack.org/api-ref/networking/v2/index.html#subnets + +- create subnet +- list subnets +- delete subnet + +Servers: https://developer.openstack.org/api-ref/compute/ + +- create keypair +- create server +- delete server +- add/assign floating ip + +Ports: https://developer.openstack.org/api-ref/networking/v2/index.html#ports + +- update port +- list ports +- show port details + +-------------------------------------------- +Test Case 1 - Port Security and MAC Spoofing +-------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_port_security_macspoofing_port + +Test preconditions +------------------ + +* Neutron port-security extension API +* Neutron security-group extension API +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a security group SG1, which has rules for allowing incoming + SSH and ICMP traffic +* Test action 2: Create a neutron network NET1 +* Test action 3: Create a tenant router R1 which routes traffic to public network +* Test action 4: Create a subnet SUBNET1 and add it as router interface +* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating ip + FIP1 (via R1) to VM1 +* Test action 6: Verify can ping FIP1 successfully and can SSH to VM1 with FIP1 +* Test action 7: Create a second neutron network NET2 and subnet SUBNET2, and attach VM1 to NET2 +* Test action 8: Get VM1's ethernet interface NIC2 for NET2 +* Test action 9: Create second server VM2 on NET2 +* Test action 10: Verify VM1 is able to communicate with VM2 via NIC2 +* Test action 11: Login to VM1 and spoof the MAC address of NIC2 to "00:00:00:00:00:01" +* Test action 12: Verify VM1 fails to communicate with VM2 via NIC2 +* **Test assertion 1:** The ping operation is failed +* Test action 13: Update 'security_groups' to be none for VM1's NIC2 port +* Test action 14: Update 'port_security_enable' to be False for VM1's NIC2 port +* Test action 15: Verify now VM1 is able to communicate with VM2 via NIC2 +* **Test assertion 2:** The ping operation is successful +* Test action 16: Delete SG1, NET1, NET2, SUBNET1, SUBNET2, R1, VM1, VM2 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to prevent MAC spoofing by using port security. +Specifically, the test verifies that: + +* With port security, the ICMP packets from a spoof server cannot pass the port. + +* Without port security, the ICMP packets from a spoof server can pass the port. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------------------ +Test Case 2 - Test Security Group Cross Tenant Traffic +------------------------------------------------------ + +Test case specification +----------------------- + +tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic + +Test preconditions +------------------ + +* Neutron security-group extension API +* Two tenants +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a neutron network NET1 for primary tenant +* Test action 2: Create a primary tenant router R1 which routes traffic to public network +* Test action 3: Create a subnet SUBNET1 and add it as router interface +* Test action 4: Create 2 empty security groups SG1 and SG2 for primary tenant +* Test action 5: Add a tcp rule to SG1 +* Test action 6: Create a server VM1 with SG1, SG2 and NET1, and assign a floating ip + FIP1 (via R1) to VM1 +* Test action 7: Repeat test action 1 to 6 and create NET2, R2, SUBNET2, SG3, SG4, + FIP2 and VM2 for an alt_tenant +* Test action 8: Verify VM1 fails to communicate with VM2 through FIP2 +* **Test assertion 1:** The ping operation is failed +* Test action 9: Add ICMP rule to SG4 +* Test action 10: Verify VM1 is able to communicate with VM2 through FIP2 +* **Test assertion 2:** The ping operation is successful +* Test action 11: Verify VM2 fails to communicate with VM1 through FIP1 +* **Test assertion 3:** The ping operation is failed +* Test action 12: Add ICMP rule to SG2 +* Test action 13: Verify VM2 is able to communicate with VM1 through FIP1 +* **Test assertion 4:** The ping operation is successful +* Test action 14: Delete SG1, SG2, SG3, SG4, NET1, NET2, SUBNET1, SUBNET2, R1, R2, + VM1, VM2, FIP1 and FIP2 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability of the security group to filter packets cross tenant. +Specifically, the test verifies that: + +* Without ICMP security group rule, the ICMP packets cannot be received by the server + in another tenant which differs from the source server. + +* With ingress ICMP security group rule enabled only at tenant1, the server in tenant2 + can ping server in tenant1 but not the reverse direction. + +* With ingress ICMP security group rule enabled at tenant2 also, the ping works from both directions. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------------------- +Test Case 3 - Test Security Group in Tenant Traffic +--------------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_in_tenant_traffic + +Test preconditions +------------------ + +* Neutron security-group extension API +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a neutron network NET1 +* Test action 2: Create a tenant router R1 which routes traffic to public network +* Test action 3: Create a subnet SUBNET1 and add it as router interface +* Test action 4: Create 2 empty security groups SG1 and SG2 +* Test action 5: Add a tcp rule to SG1 +* Test action 6: Create a server VM1 with SG1, SG2 and NET1, and assign a floating ip + FIP1 (via R1) to VM1 +* Test action 7: Create second server VM2 with default security group and NET1 +* Test action 8: Verify VM1 fails to communicate with VM2 through VM2's fixed ip +* **Test assertion 1:** The ping operation is failed +* Test action 9: Add ICMP security group rule to default security group +* Test action 10: Verify VM1 is able to communicate with VM2 through VM2's fixed ip +* **Test assertion 2:** The ping operation is successful +* Test action 11: Delete SG1, SG2, NET1, SUBNET1, R1, VM1, VM2 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability of the security group to filter packets in one tenant. +Specifically, the test verifies that: + +* Without ICMP security group rule, the ICMP packets cannot be received by the server + in the same tenant. + +* With ICMP security group rule, the ICMP packets can be received by the server + in the same tenant. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------- +Test Case 4 - Test Multiple Security Groups +------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_multiple_security_groups + +Test preconditions +------------------ + +* Neutron security-group extension API +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a neutron network NET1 +* Test action 2: Create a tenant router R1 which routes traffic to public network +* Test action 3: Create a subnet SUBNET1 and add it as router interface +* Test action 4: Create 2 empty security groups SG1 and SG2 +* Test action 5: Add a tcp rule to SG1 +* Test action 6: Create a server VM1 with SG1, SG2 and NET1, and assign a floating ip + FIP1 (via R1) to VM1 +* Test action 7: Verify failed to ping FIP1 +* **Test assertion 1:** The ping operation is failed +* Test action 8: Add ICMP security group rule to SG2 +* Test action 9: Verify can ping FIP1 successfully +* **Test assertion 2:** The ping operation is successful +* Test action 10: Verify can SSH to VM1 with FIP1 +* **Test assertion 3:** Can SSH to VM1 successfully +* Test action 11: Delete SG1, SG2, NET1, SUBNET1, R1, VM1 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability of multiple security groups to filter packets. +Specifically, the test verifies that: + +* A server with 2 security groups, one with TCP rule and without ICMP rule, + cannot receive the ICMP packets sending from the tempest host machine. + +* A server with 2 security groups, one with TCP rule and the other with ICMP rule, + can receive the ICMP packets sending from the tempest host machine and be connected + via the SSH client. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------------------- +Test Case 5 - Test Port Security Disable Security Group +------------------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_port_security_disable_security_group + +Test preconditions +------------------ + +* Neutron security-group extension API +* Neutron port-security extension API +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a neutron network NET1 +* Test action 2: Create a tenant router R1 which routes traffic to public network +* Test action 3: Create a subnet SUBNET1 and add it as router interface +* Test action 4: Create 2 empty security groups SG1 and SG2 +* Test action 5: Add a tcp rule to SG1 +* Test action 6: Create a server VM1 with SG1, SG2 and NET1, and assign a floating ip + FIP1 (via R1) to VM1 +* Test action 7: Create second server VM2 with default security group and NET1 +* Test action 8: Update 'security_groups' to be none and 'port_security_enabled' to be + True for VM2's port +* Test action 9: Verify VM1 fails to communicate with VM2 through VM2's fixed ip +* **Test assertion 1:** The ping operation is failed +* Test action 10: Update 'security_groups' to be none and 'port_security_enabled' to be + False for VM2's port +* Test action 11: Verify VM1 is able to communicate with VM2 through VM2's fixed ip +* **Test assertion 2:** The ping operation is successful +* Test action 12: Delete SG1, SG2, NET1, SUBNET1, R1, VM1, VM2 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability of port security to disable security group. +Specifically, the test verifies that: + +* The ICMP packets cannot pass the port whose 'port_security_enabled' is True + and security_groups is none. + +* The ICMP packets can pass the port whose 'port_security_enabled' is False + and security_groups is none. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------------- +Test Case 6 - Test Update Port Security Group +--------------------------------------------- + +Test case specification +----------------------- + +tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_port_update_new_security_group + +Test preconditions +------------------ + +* Neutron security-group extension API +* One public network + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a neutron network NET1 +* Test action 2: Create a tenant router R1 which routes traffic to public network +* Test action 3: Create a subnet SUBNET1 and add it as router interface +* Test action 4: Create 2 empty security groups SG1 and SG2 +* Test action 5: Add a tcp rule to SG1 +* Test action 6: Create a server VM1 with SG1, SG2 and NET1, and assign a floating ip + FIP1 (via R1) to VM1 +* Test action 7: Create third empty security group SG3 +* Test action 8: Add ICMP rule to SG3 +* Test action 9: Create second server VM2 with default security group and NET1 +* Test action 10: Verify VM1 fails to communicate with VM2 through VM2's fixed ip +* **Test assertion 1:** The ping operation is failed +* Test action 11: Update 'security_groups' to be SG3 for VM2's port +* Test action 12: Verify VM1 is able to communicate with VM2 through VM2's fixed ip +* **Test assertion 2:** The ping operation is successful +* Test action 13: Delete SG1, SG2, SG3, NET1, SUBNET1, R1, VM1, VM2 and FIP1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the ability to update port with a new security group. +Specifically, the test verifies that: + +* Without ICMP security group rule, the VM cannot receive ICMP packets. + +* Update the port's security group which has ICMP rule, the VM can receive ICMP packets. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A diff --git a/docs/testing/user/testspecification/vimoperationscompute/index.rst b/docs/testing/user/testspecification/vimoperationscompute/index.rst index 4ed37809..42a8e70e 100644 --- a/docs/testing/user/testspecification/vimoperationscompute/index.rst +++ b/docs/testing/user/testspecification/vimoperationscompute/index.rst @@ -1,6 +1,6 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. (c) Ericsson AB +.. (c) Ericsson AB, Huawei Technologies Co.,Ltd ========================================= VIM compute operations test specification @@ -12,47 +12,375 @@ VIM compute operations test specification Scope ===== +The VIM compute operations test area evaluates the ability of the system under +test to support VIM compute operations. The test cases documented here are the +compute API test cases in the OpenStack Interop guideline 2016.8 as implemented +by the RefStack client. These test cases will evaluate basic OpenStack (as a VIM) +compute operations, including: + +- Image management operations +- Basic support operations +- API version support operations +- Quotas management operations +- Basic server operations +- Volume management operations + References ================ +- OpenStack Governance/Interop + + - https://wiki.openstack.org/wiki/Governance/InteropWG + +- Defcore test cases + + - https://github.com/openstack/interop/blob/master/2016.08.json + +- Refstack client + + - https://github.com/openstack/refstack-client Definitions and abbreviations ============================= +The following terms and abbreviations are used in conjunction with this test area -Use case description -==================== - +- API - Application Programming Interface +- NFVi - Network Functions Virtualization infrastructure +- SUT - System Under Test +- UUID - Universally Unique Identifier +- VIM - Virtual Infrastructure Manager +- VM - Virtual Machine System Under Test (SUT) ======================= +The system under test is assumed to be the NFVi and VIM deployed with a Pharos compliant infrastructure. -Test Suite Structure +Test Area Structure ==================== +The test area is structured based on VIM compute API operations. Each test case is +able to run independently, i.e. irrelevant of the state created by a previous test. +Specifically, every test performs clean-up operations which return the system to +the same state as before the test. + +For brevity, the test cases in this test area are summarized together based on +the operations they are testing. Test Descriptions ================= +API Used and Reference +---------------------- + +Servers: https://developer.openstack.org/api-ref/compute/ + +- create server +- delete server +- list servers +- start server +- stop server +- update server +- get server action +- set server metadata +- update server metadata +- rebuild server + +- create image +- delete image + +- create keypair +- delete keypair + +Block storage: https://developer.openstack.org/api-ref/block-storage + +- create volume +- delete volume +- attach volume to server +- detach volume from server + +----------------------------------------------------- +Test Case 1 - Image operations within the Compute API +----------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image +tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_image_specify_multibyte_character_image_name + +Test preconditions +------------------ + +* Compute server extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a server VM1 with an image IMG1 and wait for VM1 to reach 'ACTIVE' status +* Test action 2: Create a new server image IMG2 from VM1, specifying image name + and image metadata. Wait for IMG2 to reach 'ACTIVE' status, and then delete IMG2 +* **Test assertion 1:** Verify IMG2 is created with correct image name and image + metadata; verify IMG1's 'minRam' equals to IMG2's 'minRam' and IMG2's 'minDisk' equals + to IMG1's 'minDisk' or VM1's flavor disk size +* **Test assertion 2:** Verify IMG2 is deleted correctly +* Test action 3: Create another server IMG3 from VM1, specifying image name + with a 3 byte utf-8 character +* **Test assertion 3:** Verify IMG3 is created correctly +* Test action 4: Delete VM1, IMG1 and IMG3 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the Compute API ability of creating image from server, +deleting image, creating server image with multi-byte character name. +Specifically, the test verifies that: + +* Compute server create image and delete image APIs work correctly. +* Compute server image can be created with multi-byte character name. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +----------------------------------------------------- +Test Case 2 - Action operation within the Compute API +----------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_get_instance_action +tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_list_instance_actions + +Test preconditions +------------------ + +* Compute server extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a server VM1 and wait for VM1 to reach 'ACTIVE' status +* Test action 2: Get the action details ACT_DTL of VM1 +* **Test assertion 1:** Verify ACT_DTL's 'instance_uuid' matches VM1's ID and + ACT_DTL's 'action' matched 'create' +* Test action 3: Create a server VM2 and wait for VM2 to reach 'ACTIVE' status +* Test action 4: Delete server VM2 and wait for VM2 to reach termination +* Test action 5: Get the action list ACT_LST of VM2 +* **Test assertion 2:** Verify ACT_LST's length is 2 and two actions are 'create' and 'delete' +* Test action 6: Delete VM1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the Compute API ability of getting the action details +of a provided server and getting the action list of a deleted server. +Specifically, the test verifies that: + +* Get the details of the action in a specified server. +* List the actions that were performed on the specified server. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------------------------- +Test Case 3 - Generate, import and delete SSH keys within Compute services +-------------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_specify_keypair + +Test preconditions +------------------ + +* Compute server extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a keypair KEYP1 and list all existing keypairs +* Test action 2: Create a server VM1 with KEYP1 and wait for VM1 to reach 'ACTIVE' status +* Test action 3: Show details of VM1 +* **Test assertion 1:** Verify value of 'key_name' in the details equals to the name of KEYP1 +* Test action 4: Delete KEYP1 and VM1 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the Compute API ability of creating a keypair, listing +keypairs and creating a server with a provided keypair. +Specifically, the test verifies that: + +* Compute create keypair and list keypair APIs work correctly. +* While creating a server, keypair can be specified. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------- +Test Case 4 - List supported versions of the Compute API +-------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.test_versions.TestVersions.test_list_api_versions + +Test preconditions +------------------ + +* Compute versions extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Get a List of versioned endpoints in the SUT +* **Test assertion 1:** Verify endpoints versions start at 'v2.0' + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of listing all available APIs to API consumers. +Specifically, the test verifies that: + +* Compute list API versions API works correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------------- +Test Case 5 - Quotas management in Compute API +---------------------------------------------- + +Test case specification +----------------------- + +tempest.api.compute.test_quotas.QuotasTestJSON.test_get_default_quotas +tempest.api.compute.test_quotas.QuotasTestJSON.test_get_quotas + +Test preconditions +------------------ + +* Compute quotas extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' +* Test action 1: Get the default quota set using the tenant ID +* **Test assertion 1:** Verify the default quota set ID matches tenant ID and + the default quota set is complete +* Test action 2: Get the quota set using the tenant ID +* **Test assertion 2:** Verify the quota set ID matches tenant ID and the quota + set is complete +* Test action 3: Get the quota set using the user ID +* **Test assertion 3:** Verify the quota set ID matches tenant ID and the quota + set is complete + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of getting quota set. +Specifically, the test verifies that: + +* User can get the default quota set for its tenant. +* User can get the quota set for its tenant. +* User can get the quota set using user ID. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------- +Test Case 6 - Basic server operations in the Compute API +-------------------------------------------------------- + +Test case specification +----------------------- + +This test case evaluates the Compute API ability of basic server operations, including: + +- Create a server with admin password +- Create a server with a name that already exists +- Create a server with a numeric name +- Create a server with a really long metadata +- Create a server with a name whose length exceeding 255 characters +- Create a server with an unknown flavor +- Create a server with an unknown image ID +- Create a server with an invalid network UUID +- Delete a server using a server ID that exceeds length limit +- Delete a server using a negative server ID +- Get a nonexistent server details +- Verify the instance host name is the same as the server name +- Create a server with an invalid access IPv6 address +- List all existent servers +- Filter the (detailed) list of servers by flavor, image, server name, server status or limit +- Lock a server and try server stop, unlock and retry +- Get and delete metadata from a server +- List and set metadata for a server +- Reboot, rebuild, stop and start a server +- Update a server's access addresses and server name + +The reference is, + +tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_server_with_admin_password +tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_with_existing_server_name +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_numeric_server_name +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_server_metadata_exceeds_length_limit +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_server_name_length_exceeds_256 +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_flavor +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_image +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_network_uuid +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_delete_server_pass_id_exceeding_length_limit +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_delete_server_pass_negative_id +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_get_non_existent_server tempest.api.compute.servers.test_create_server.ServersTestJSON.test_host_name_is_same_as_server_name +tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_host_name_is_same_as_server_name +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_invalid_ip_v6_address tempest.api.compute.servers.test_create_server.ServersTestJSON.test_list_servers tempest.api.compute.servers.test_create_server.ServersTestJSON.test_list_servers_with_detail -tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_created_server_vcpus -tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_server_details -tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_host_name_is_same_as_server_name tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_list_servers tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_list_servers_with_detail -tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_created_server_vcpus -tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_server_details -tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_get_instance_action -tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_list_instance_actions tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_flavor tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_image tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_server_name tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_server_status tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_limit_results -tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_active_status tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_flavor tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_image tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_limit @@ -72,41 +400,302 @@ tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJS tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_status_non_existing tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_with_a_deleted_server tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_lock_unlock_server -tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_reboot_server_hard -tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_rebuild_server -tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_stop_start_server tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_delete_server_metadata_item tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_get_server_metadata_item tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_list_server_metadata tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_set_server_metadata tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_set_server_metadata_item tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_update_server_metadata -tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_server_with_admin_password -tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_specify_keypair -tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_with_existing_server_name -tempest.api.compute.servers.test_servers.ServersTestJSON.test_update_access_server_address -tempest.api.compute.servers.test_servers.ServersTestJSON.test_update_server_name -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_numeric_server_name -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_server_metadata_exceeds_length_limit -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_server_name_length_exceeds_256 -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_flavor -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_image -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_network_uuid -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_delete_server_pass_id_exceeding_length_limit -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_delete_server_pass_negative_id -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_get_non_existent_server -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_invalid_ip_v6_address +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_server_name_blank +tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_reboot_server_hard tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_non_existent_server +tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_rebuild_server tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_deleted_server tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_non_existent_server -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_reboot_deleted_server -tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_server_name_blank +tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_stop_start_server tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_stop_non_existent_server +tempest.api.compute.servers.test_servers.ServersTestJSON.test_update_access_server_address +tempest.api.compute.servers.test_servers.ServersTestJSON.test_update_server_name tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_update_name_of_non_existent_server tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_update_server_name_length_exceeds_256 tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_update_server_set_empty_name -tempest.api.compute.test_quotas.QuotasTestJSON.test_get_default_quotas -tempest.api.compute.test_quotas.QuotasTestJSON.test_get_quotas -tempest.api.compute.test_versions.TestVersions.test_list_api_versions +tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_created_server_vcpus +tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_server_details +tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_created_server_vcpus +tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_server_details + +tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_active_status +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_reboot_deleted_server + +Note: the last 2 test cases are the alias of another 2 test cases respectively, which are + +tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_server_status +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_deleted_server + +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of OpenStack. + +Test preconditions +------------------ + +* Compute quotas extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a server VM1 with a admin password 'testpassword' +* **Test assertion 1:** Verify the password returned in the response equals to 'testpassword' +* Test action 2: Generate a VM name VM_NAME +* Test action 3: Create 2 servers VM2 and VM3 both with name VM_NAME +* **Test assertion 2:** Verify VM2's ID is not equal to VM3's ID, and VM2's name equal to VM3's name +* Test action 4: Create a server VM4 with a numeric name '12345' +* **Test assertion 3:** Verify creating VM4 failed +* Test action 5: Create a server VM5 with a long metadata '{'a': 'b' * 260}' +* **Test assertion 4:** Verify creating VM5 failed +* Test action 6: Create a server VM6 with name length exceeding 255 characters +* **Test assertion 5:** Verify creating VM6 failed +* Test action 7: Create a server VM7 with an unknown flavor '-1' +* **Test assertion 6:** Verify creating VM7 failed +* Test action 8: Create a server VM8 with an unknown image ID '-1' +* **Test assertion 7:** Verify creating VM8 failed +* Test action 9: Create a server VM9 with an invalid network UUID 'a-b-c-d-e-f-g-h-i-j' +* **Test assertion 8:** Verify creating VM9 failed +* Test action 10: Delete a server using a server ID that exceeds system's max integer limit +* **Test assertion 9:** Verify deleting server failed +* Test action 11: Delete a server using a server ID '-1' +* **Test assertion 10:** Verify deleting server failed +* Test action 12: Get a nonexistent server by using a random generated server ID +* **Test assertion 11:** Verify get server failed +* Test action 13: SSH into a provided server and get server's hostname +* **Test assertion 12:** Verify server's host name is the same as the server name +* Test action 14: SSH into a provided server and get server's hostname (manual disk configuration) +* **Test assertion 13:** Verify server's host name is the same as the server name (manual disk configuration) +* Test action 15: Create a server with an invalid access IPv6 address +* **Test assertion 14:** Verify creating server failed, a bad request error is returned in response +* Test action 16: List all existent servers +* **Test assertion 15:** Verify a provided server is in the server list +* Test action 17: List all existent servers in detail +* **Test assertion 16:** Verify a provided server is in the detailed server list +* Test action 18: List all existent servers (manual disk configuration) +* **Test assertion 17:** Verify a provided server is in the server list (manual disk configuration) +* Test action 19: List all existent servers in detail (manual disk configuration) +* **Test assertion 18:** Verify a provided server is in the detailed server list (manual disk configuration) +* Test action 20: List all existent servers in detail and filter the server list by flavor +* **Test assertion 19:** Verify the filtered server list is correct +* Test action 21: List all existent servers in detail and filter the server list by image +* **Test assertion 20:** Verify the filtered server list is correct +* Test action 22: List all existent servers in detail and filter the server list by server name +* **Test assertion 21:** Verify the filtered server list is correct +* Test action 23: List all existent servers in detail and filter the server list by server status +* **Test assertion 22:** Verify the filtered server list is correct +* Test action 24: List all existent servers in detail and filter the server list by display limit '1' +* **Test assertion 23:** Verify the length of filtered server list is 1 +* Test action 25: List all existent servers and filter the server list by flavor +* **Test assertion 24:** Verify the filtered server list is correct +* Test action 26: List all existent servers and filter the server list by image +* **Test assertion 25:** Verify the filtered server list is correct +* Test action 27: List all existent servers and filter the server list by display limit '1' +* **Test assertion 26:** Verify the length of filtered server list is 1 +* Test action 28: List all existent servers and filter the server list by server name +* **Test assertion 27:** Verify the filtered server list is correct +* Test action 29: List all existent servers and filter the server list by server status +* **Test assertion 28:** Verify the filtered server list is correct +* Test action 30: List all existent servers and filter the server list by server name wildcard +* **Test assertion 29:** Verify the filtered server list is correct +* Test action 31: List all existent servers and filter the server list by part of server name +* **Test assertion 30:** Verify the filtered server list is correct +* Test action 32: List all existent servers and filter the server list by a future change-since date +* **Test assertion 31:** Verify the filtered server list is empty +* Test action 33: List all existent servers and filter the server list by a invalid change-since date format +* **Test assertion 32:** Verify a bad request error is returned in the response +* Test action 34: List all existent servers and filter the server list by display limit '1' +* **Test assertion 33:** Verify the length of filtered server list is 1 +* Test action 35: List all existent servers and filter the server list by a + display limit value greater than the length of the server list +* **Test assertion 34:** Verify the length of filtered server list equals to the length of server list +* Test action 36: List all existent servers and filter the server list by display limit '-1' +* **Test assertion 35:** Verify a bad request error is returned in the response +* Test action 37: List all existent servers and filter the server list by a string type limit value 'testing' +* **Test assertion 36:** Verify a bad request error is returned in the response +* Test action 38: List all existent servers and filter the server list by a nonexistent flavor +* **Test assertion 37:** Verify the filtered server list is empty +* Test action 39: List all existent servers and filter the server list by a nonexistent image +* **Test assertion 38:** Verify the filtered server list is empty +* Test action 40: List all existent servers and filter the server list by a nonexistent server name +* **Test assertion 39:** Verify the filtered server list is empty +* Test action 41: List all existent servers in detail and search the server list for a deleted server +* **Test assertion 40:** Verify the deleted server is not in the server list +* Test action 42: List all existent servers and filter the server list by a nonexistent server status +* **Test assertion 41:** Verify the filtered server list is empty +* Test action 43: List all existent servers in detail +* **Test assertion 42:** Verify a provided deleted server's id is not in the server list +* Test action 44: Lock a provided server VM10 and retrieve the server's status +* **Test assertion 43:** Verify VM10 is in 'ACTIVE' status +* Test action 45: Stop VM10 +* **Test assertion 44:** Verify stop VM10 failed +* Test action 46: Unlock VM10 and stop VM10 again +* **Test assertion 45:** Verify VM10 is stopped and in 'SHUTOFF' status +* Test action 47: Start VM10 +* **Test assertion 46:** Verify VM10 is in 'ACTIVE' status +* Test action 48: Delete metadata item 'key1' from a provided server +* **Test assertion 47:** Verify the metadata item is removed +* Test action 49: Get metadata item 'key2' from a provided server +* **Test assertion 48:** Verify the metadata item is correct +* Test action 50: List all metadata key/value pair for a provided server +* **Test assertion 49:** Verify all metadata are retrieved correctly +* Test action 51: Set metadata {'meta2': 'data2', 'meta3': 'data3'} for a provided server +* **Test assertion 50:** Verify server's metadata are replaced correctly +* Test action 52: Set metadata item nova's value to 'alt' for a provided server +* **Test assertion 51:** Verify server's metadata are set correctly +* Test action 53: Update metadata {'key1': 'alt1', 'key3': 'value3'} for a provided server +* **Test assertion 52:** Verify server's metadata are updated correctly +* Test action 54: Create a server with empty name parameter +* **Test assertion 53:** Verify create server failed +* Test action 55: Hard reboot a provided server +* **Test assertion 54:** Verify server is rebooted successfully +* Test action 56: Soft reboot a nonexistent server +* **Test assertion 55:** Verify reboot failed, an error is returned in the response +* Test action 57: Rebuild a provided server with new image, new server name and metadata +* **Test assertion 56:** Verify server is rebuilt successfully, server image, name and metadata are correct +* Test action 58: Create a server VM11 +* Test action 59: Delete VM11 and wait for VM11 to reach termination +* Test action 60: Rebuild VM11 with another image +* **Test assertion 57:** Verify rebuild server failed, an error is returned in the response +* Test action 61: Rebuild a nonexistent server +* **Test assertion 58:** Verify rebuild server failed, an error is returned in the response +* Test action 62: Stop a provided server +* **Test assertion 59:** Verify server reaches 'SHUTOFF' status +* Test action 63: Start the stopped server +* **Test assertion 60:** Verify server reaches 'ACTIVE' status +* Test action 64: Stop a provided server +* **Test assertion 61:** Verify stop server failed, an error is returned in the response +* Test action 65: Create a server VM12 and wait it to reach 'ACTIVE' status +* Test action 66: Update VM12's IPv4 and IPv6 access addresses +* **Test assertion 62:** Verify VM12's access addresses have been updated correctly +* Test action 67: Create a server VM13 and wait it to reach 'ACTIVE' status +* Test action 68: Update VM13's server name with non-ASCII characters '\u00CD\u00F1st\u00E1\u00F1c\u00E9' +* **Test assertion 63:** Verify VM13's server name has been updated correctly +* Test action 69: Update the server name of a nonexistent server +* **Test assertion 64:** Verify update server name failed, an 'object not found' error is returned in the response +* Test action 70: Update a provided server's name with a 256-character long name +* **Test assertion 65:** Verify update server name failed, a bad request is returned in the response +* Test action 71: Update a provided server's server name with an empty string +* **Test assertion 66:** Verify update server name failed, a bad request error is returned in the response +* Test action 72: Get the number of vcpus of a provided server +* Test action 73: Get the number of vcpus stated by the server's flavor +* **Test assertion 67:** Verify that the number of vcpus reported by the server + matches the amount stated by the server's flavor +* Test action 74: Create a server VM14 +* **Test assertion 68:** Verify VM14's server attributes are set correctly +* Test action 75: Get the number of vcpus of a provided server (manual disk configuration) +* Test action 76: Get the number of vcpus stated by the server's flavor (manual disk configuration) +* **Test assertion 69:** Verify that the number of vcpus reported by the server + matches the amount stated by the server's flavor (manual disk configuration) +* Test action 77: Create a server VM15 (manual disk configuration) +* **Test assertion 70:** Verify VM15's server attributes are set correctly (manual disk configuration) +* Test action 78: Delete all VMs created + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of basic server operations. +Specifically, the test verifies that: + +* If an admin password is provided on server creation, the server's root password should be set to that password +* Create a server with a name that already exists is allowed +* Create a server with a numeric name or a name that exceeds the length limit is not allowed +* Create a server with a metadata that exceeds the length limit is not allowed +* Create a server with an invalid flavor, an invalid image or an invalid network UUID is not allowed +* Delete a server with a server ID that exceeds the length limit or a nonexistent server ID is not allowed +* A provided server's host name is the same as the server name +* Create a server with an invalid IPv6 access address is not allowed +* A created server is in the (detailed) list of servers +* Filter the (detailed) list of servers by flavor, image, server name, server status, + and display limit, respectively. +* Filter the list of servers by a future date +* Filter the list of servers by an invalid date format, a negative display limit or a string type + display limit value is not allowed +* Filter the list of servers by a nonexistent flavor, image, server name or server status is not allowed +* Deleted servers are not in the list of servers +* Deleted servers do not show by default in list of servers +* Locked server is not allowed to be stopped by non-admin user +* Can get and delete metadata from servers +* Can list, set and update server metadata +* Create a server with name parameter empty is not allowed +* Hard reboot a server and the server should be power cycled +* Reboot, rebuild and stop a nonexistent server is not allowed +* Rebuild a server using the provided image and metadata +* Stop and restart a server +* A server's name and access addresses can be updated +* Update the name of a nonexistent server is not allowed +* Update name of a server to a name that exceeds the name length limit is not allowed +* Update name of a server to an empty string is not allowed +* The number of vcpus reported by the server matches the amount stated by the server's flavor +* The specified server attributes are set correctly + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +----------------------------------------------------------------- +Test Case 7 - Retrieve volume information through the Compute API +----------------------------------------------------------------- + +Test case specification +----------------------- + +This test case evaluates the Compute API ability of attaching volume to a +specific server and retrieve volume information, the reference is, + tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_attach_detach_volume tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_list_get_volume_attachments + +Test preconditions +------------------ + +* Compute volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create a server VM1 and a volume VOL1 +* Test action 2: Attach VOL1 to VM1 +* **Test assertion 1:** Stop VM1 successfully and wait VM1 to reach 'SHUTOFF' status +* **Test assertion 2:** Start VM1 successfully and wait VM1 to reach 'ACTIVE' status +* **Test assertion 3:** SSH into VM1 and verify VOL1 is in VM1's root disk devices +* Test action 3: Detach VOL1 from VM1 +* **Test assertion 4:** Stop VM1 successfully and wait VM1 to reach 'SHUTOFF' status +* **Test assertion 5:** Start VM1 successfully and wait VM1 to reach 'ACTIVE' status +* **Test assertion 6:** SSH into VM1 and verify VOL1 is not in VM1's root disk devices +* Test action 4: Create a server VM2 and a volume VOL2 +* Test action 5: Attach VOL2 to VM2 +* Test action 6: List VM2's volume attachments +* **Test assertion 7:** Verify the length of the list is 1 and VOL2 attachment is in the list +* Test action 7: Retrieve VM2's volume information +* **Test assertion 8:** Verify volume information is correct +* Test action 8: Delete VM1, VM2, VOL1 and VOL2 + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the functionality of retrieving volume information. +Specifically, the test verifies that: + +* Stop and start a server with an attached volume work correctly. +* Retrieve a server's volume information correctly. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A diff --git a/docs/testing/user/testspecification/vimoperationsimage/index.rst b/docs/testing/user/testspecification/vimoperationsimage/index.rst index 932c7382..8e5f08ce 100644 --- a/docs/testing/user/testspecification/vimoperationsimage/index.rst +++ b/docs/testing/user/testspecification/vimoperationsimage/index.rst @@ -1,6 +1,6 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. (c) Ericsson AB +.. (c) Ericsson AB, Huawei Technologies Co.,Ltd ======================================= VIM image operations test specification @@ -9,17 +9,191 @@ VIM image operations test specification .. toctree:: :maxdepth: 2 -Each test case requires documentation according to: -* Use case specification -* Test preconditions -* Basic test flow execution descriptor -* Post conditions and pass fail criteria +Scope +===== + +The VIM image test area evaluates the ability of the system under test to support +VIM image operations. The test cases documented here are the Image API test cases +in the Openstack Interop guideline 2016.8 as implemented by the Refstack client. +These test cases will evaluate basic Openstack (as a VIM) image operations including +image creation, image list, image update and image deletion capabilities using Glance v2 API. + +References +========== + +- Defcore test cases + + - https://github.com/openstack/interop/blob/master/2016.08.json + +- Refstack client + + - https://github.com/openstack/refstack-client + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test area + +- API - Application Programming Interface +- CRUD - Create, Read, Update, and Delete +- NFVi - Network Functions Virtualization infrastructure +- VIM - Virtual Infrastructure Manager + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + +Test Area Structure +==================== + +The test area is structured based on VIM image operations. Each test case is able +to run independently, i.e. irrelevant of the state created by a previous test. + +For brevity, the test cases in this test area are summarized together based on +the operations they are testing. + +Test Descriptions +================= + +API Used and Reference +---------------------- + +Images: https://developer.openstack.org/api-ref/image/v2/ + +- create image +- delete image +- show image details +- show images +- show image schema +- show images schema +- upload binary image data +- add image tag +- delete image tag + +--------------------------------------- +Image get tests using the Glance v2 API +--------------------------------------- + +Test case specification +----------------------- -tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_delete_image -tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_update_image tempest.api.image.v2.test_images.ListImagesTest.test_get_image_schema tempest.api.image.v2.test_images.ListImagesTest.test_get_images_schema +tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_delete_deleted_image +tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_image_null_id +tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_non_existent_image + +tempest.api.image.v2.test_images.ListUserImagesTest.test_get_image_schema +tempest.api.image.v2.test_images.ListUserImagesTest.test_get_images_schema + +Note: the latter two test cases are the alias of the former first two, respectively. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +Glance is available. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create 6 images and store their ids in a created images list. +* Test action 2: Use image v2 API to show image schema and check the body of the response. +* **Test assertion 1:** In the body of the response, the value of the key 'name' is 'image'. +* Test action 3: Use image v2 API to show images schema and check the body of the response. +* **Test assertion 2:** In the body of the response, the value of the key 'name' is 'images'. +* Test action 4: Create an image with name 'test', container_formats 'bare' and + disk_formats 'raw'. Delete this image with its id and then try to show it with + its id. Delete this deleted image again with its id and check the API's response code. +* **Test assertion 3:** The operations of showing and deleting a deleted image with its id + both get 404 response code. +* Test action 5: Use a null image id to show a image and check the API's response code. +* **Test assertion 4:** The API's response code is 404. +* Test action 6: Generate a random uuid and use it as the image id to show the image. +* **Test assertion 5:** The API's response code is 404. +* Test action 7: Delete the 6 images with the stored ids. Show all images and check + whether the 6 images' ids are not in the show list. +* **Test assertion 6:** The 6 images' ids are not found in the show list. + +Pass / fail criteria +'''''''''''''''''''' + +The first two test cases evaluate the ability to use Glance v2 API to show image +and images schema. The latter three test cases evaluate the ability to use Glance +v2 API to show images with a deleted image id, a null image id and a non-existing +image id. Specifically it verifies that: + +* Glance image get API can show the image and images schema. +* Glance image get API can't show an image with a deleted image id. +* Glance image get API can't show an image with a null image id. +* Glance image get API can't show an image with a non-existing image id. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +None + +-------------------------------------- +CRUD image operations in Images API v2 +-------------------------------------- + +Test case specification +----------------------- + +tempest.api.image.v2.test_images.ListImagesTest.test_list_no_params + tempest.api.image.v2.test_images.ListImagesTest.test_index_no_params +tempest.api.image.v2.test_images.ListUserImagesTest.test_list_no_params + +Note: the latter two test cases are the alias of the former one. Alias should +always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +Glance is available. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create 6 images and store their ids in a created images list. +* Test action 2: List all images and check whether the ids listed are in the created images list. +* **Test assertion 1:** The ids get from the list images API are in the created images list. + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the ability to use Glance v2 API to list images. +Specifically it verifies that: + +* Glance image API can show the images. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +None + +---------------------------------------- +Image list tests using the Glance v2 API +---------------------------------------- + +Test case specification +----------------------- + tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_container_format tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_disk_format tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_limit @@ -27,9 +201,7 @@ tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_min_max_s tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_size tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_status tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_visibility -tempest.api.image.v2.test_images.ListImagesTest.test_list_no_params -tempest.api.image.v2.test_images.ListUserImagesTest.test_get_image_schema -tempest.api.image.v2.test_images.ListUserImagesTest.test_get_images_schema + tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_container_format tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_disk_format tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_limit @@ -37,12 +209,198 @@ tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_min_m tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_size tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_status tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_visibility -tempest.api.image.v2.test_images.ListUserImagesTest.test_list_no_params + +Note: the latter 7 test cases are the alias of the former 7, respectively. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +Glance is available. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create 6 images with a random size ranging from 1024 to 4096 and + visibility 'private'; set their (container_format, disk_format) pair to be + (ami, ami), (ami, ari), (ami, aki), (ami, vhd), (ami, vmdk) and (ami, raw); + store their ids in a list and upload the binary images data. +* Test action 2: Use Glance v2 API to list all images whose container_format is 'ami' + and store the response details in a list. +* **Test assertion 1:** The list is not empty and all the values of container_format + in the list are 'ami'. +* Test action 3: Use Glance v2 API to list all images whose disk_format is 'raw' + and store the response details in a list. +* **Test assertion 2:** The list is not empty and all the values of disk_format + in the list are 'raw'. +* Test action 4: Use Glance v2 API to list one image by setting limit to be 1 and + store the response details in a list. +* **Test assertion 3:** The length of the list is one. +* Test action 5: Use Glance v2 API to list images by setting size_min and size_max, + and store the response images' sizes in a list. Choose the first image's size as + the median, size_min is median-500 and size_max is median+500. +* **Test assertion 4:** All sizes in the list are no less than size_min and no more + than size_max. +* Test action 6: Use Glance v2 API to show the first created image with its id and + get its size from the response. Use Glance v2 API to list images whose size is equal + to this size and store the response details in a list. +* **Test assertion 5:** All sizes of the images in the list are equal to the size + used to list the images. +* Test action 7: Use Glance v2 API to list the images whose status is active and + store the response details in a list. +* **Test assertion 6:** All status of images in the list are active. +* Test action 8: Use Glance v2 API to list the images whose visibility is private and + store the response details in a list. +* **Test assertion 7:** All images' values of visibility in the list are private. +* Test action 9: Delete the 6 images with the stored ids. Show images and check whether + the 6 ids are not in the show list. +* **Test assertion 8:** The stored 6 ids are not found in the show list. + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the ability to use Glance v2 API to list images with +different parameters. Specifically it verifies that: + +* Glance image API can show the images with the container_format. +* Glance image API can show the images with the disk_format. +* Glance image API can show the images by setting a limit number. +* Glance image API can show the images with the size_min and size_max. +* Glance image API can show the images with the size. +* Glance image API can show the images with the status. +* Glance image API can show the images with the visibility type. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +None + +------------------------------------------ +Image update tests using the Glance v2 API +------------------------------------------ + +Test case specification +----------------------- + +tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_update_image +tempest.api.image.v2.test_images_tags.ImagesTagsTest.test_update_delete_tags_for_image +tempest.api.image.v2.test_images_tags_negative.ImagesTagsNegativeTest.test_update_tags_for_non_existing_image + +Test preconditions +------------------ + +Glance is available. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create an image with container_formats 'ami', disk_formats 'ami' + and visibility 'private' and store its id returned in the response. Check whether + the status of the created image is 'queued'. +* **Test assertion 1:** The status of the created image is 'queued'. +* Test action 2: Use the stored image id to upload the binary image data and update + this image's name. Show this image with the stored id. Check if the stored id and + name used to update the image are equal to the id and name in the show list. +* **Test assertion 2:** The id and name returned in the show list are equal to + the stored id and name used to update the image. +* Test action 3: Create an image with container_formats 'bare', disk_formats 'raw' + and visibility 'private' and store its id returned in the response. +* Test action 4: Use the stored id to add a tag. Show the image with the stored id + and check if the tag used to add is in the image's tags returned in the show list. +* **Test assertion 3:** The tag used to add into the image is in the show list. +* Test action 5: Use the stored id to delete this tag. Show the image with the + stored id and check if the tag used to delete is not in the show list. +* **Test assertion 4:** The tag used to delete from the image is not in the show list. +* Test action 6: Generate a random uuid as the image id. Use the image id to add a tag + into the image's tags. +* **Test assertion 5:** The API's response code is 404. +* Test action 7: Delete the images created in test action 1 and 3. Show the images + and check whether the ids are not in the show list. +* **Test assertion 6:** The two ids are not found in the show list. + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the ability to use Glance v2 API to update images with +different parameters. Specifically it verifies that: + +* Glance image API can update image's name with the existing image id. +* Glance image API can update image's tags with the existing image id. +* Glance image API can't update image's tags with a non-existing image id. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +None + +-------------------------------------------- +Image deletion tests using the Glance v2 API +-------------------------------------------- + +Test case specification +----------------------- + +tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_delete_image tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_delete_image_null_id tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_delete_non_existing_image -tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_delete_deleted_image -tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_image_null_id -tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_non_existent_image -tempest.api.image.v2.test_images_tags.ImagesTagsTest.test_update_delete_tags_for_image tempest.api.image.v2.test_images_tags_negative.ImagesTagsNegativeTest.test_delete_non_existing_tag -tempest.api.image.v2.test_images_tags_negative.ImagesTagsNegativeTest.test_update_tags_for_non_existing_image + +Test preconditions +------------------ + +Glance is available. + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create an image with container_formats 'ami', disk_formats 'ami' + and visibility 'private'. Use the id of the created image to delete the image. + List all images and check whether this id is in the list. +* **Test assertion 1:** The id of the created image is not found in the list + of all images after the deletion operation. +* Test action 2: Delete images with a null id and check the API's response code. +* **Test assertion 2:** The API's response code is 404. +* Test action 3: Generate a random uuid and delete images with this uuid as image id. + Check the API's response code. +* **Test assertion 3:** The API's response code is 404. +* Test action 4: Create an image with container_formats 'bare', disk_formats 'raw' + and visibility 'private'. Delete this image's tag with the image id and a random tag + Check the API's response code. +* **Test assertion 4:** The API's response code is 404. +* Test action 5: Delete the images created in test action 1 and 4. List all images + and check whether the ids are in the list. +* **Test assertion 5:** The two ids are not found in the list. + +Pass / fail criteria +'''''''''''''''''''' + +The first three test cases evaluate the ability to use Glance v2 API to delete images +with an existing image id, a null image id and a non-existing image id. The last one +evaluates the ability to use the API to delete a non-existing image tag. +Specifically it verifies that: + +* Glance image deletion API can delete the image with an existing id. +* Glance image deletion API can't delete an image with a null image id. +* Glance image deletion API can't delete an image with a non-existing image id. +* Glance image deletion API can't delete an image tag with a non-existing image tag. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +None diff --git a/docs/testing/user/testspecification/vimoperationsvolume/index.rst b/docs/testing/user/testspecification/vimoperationsvolume/index.rst index ce039a5b..9a76c37c 100644 --- a/docs/testing/user/testspecification/vimoperationsvolume/index.rst +++ b/docs/testing/user/testspecification/vimoperationsvolume/index.rst @@ -1,36 +1,512 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. (c) Ericsson AB +.. (c) Ericsson AB, Huawei Technologies Co.,Ltd -======================================== +========================================= VIM volume operations test specification -======================================== +========================================= .. toctree:: :maxdepth: 2 -Each test case requires documentation according to: -* Use case specification -* Test preconditions -* Basic test flow execution descriptor -* Post conditions and pass fail criteria +Scope +===== + +The VIM volume operations test area evaluates the ability of the system under +test to support VIM volume operations. The test cases documented here are the +volume API test cases in the OpenStack Interop guideline 2016.8 as implemented +by the RefStack client. These test cases will evaluate basic OpenStack (as a VIM) +volume operations, including: + +- Volume attach and detach operations +- Volume service availability zone operations +- Volume cloning operations +- Image copy-to-volume operations +- Volume creation and deletion operations +- Volume service extension listing +- Volume metadata operations +- Volume snapshot operations + +References +================ + +- OpenStack Governance/Interop + + - https://wiki.openstack.org/wiki/Governance/InteropWG + +- Defcore test cases + + - https://github.com/openstack/interop/blob/master/2016.08.json + +- Refstack client + + - https://github.com/openstack/refstack-client + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test area + +- API - Application Programming Interface +- NFVi - Network Functions Virtualization infrastructure +- SUT - System Under Test +- VIM - Virtual Infrastructure Manager +- VM - Virtual Machine + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVI and VIM deployed with a Pharos compliant infrastructure. + +Test Area Structure +==================== + +The test area is structured based on VIM volume API operations. Each test case is +able to run independently, i.e. irrelevant of the state created by a previous test. +Specifically, every test performs clean-up operations which return the system to +the same state as before the test. + +For brevity, the test cases in this test area are summarized together based on +the operations they are testing. + +Test Descriptions +================= + +API Used and Reference +---------------------- + +Block storage: https://developer.openstack.org/api-ref/block-storage + +- create volume +- delete volume +- update volume +- attach volume to server +- detach volume from server +- create volume metadata +- update volume metadata +- delete volume metadata +- list volume + +- create snapshot +- update snapshot +- delete snapshot + +------------------------------------------------------------------------ +Test Case 1 - Volume attach and detach operations with the Cinder v2 API +------------------------------------------------------------------------ + +Test case specification +----------------------- -tempest.api.volume.test_availability_zone.AvailabilityZoneV2TestJSON.test_get_availability_zone_list -tempest.api.volume.test_extensions.ExtensionsV2TestJSON.test_list_extensions -tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_create_get_delete_snapshot_metadata -tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_crud_snapshot_metadata -tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_update_snapshot_metadata_item -tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_create_get_delete_volume_metadata -tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_crud_volume_metadata -tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_update_volume_metadata_item tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_attach_detach_volume_to_instance tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_get_volume_attachment -tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_reserve_unreserve_volume -tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_volume_bootable -tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_volume_readonly_update -tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_attach_volumes_with_nonexistent_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_detach_volumes_with_invalid_volume_id + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' +* Test action 1: Create a server VM1 +* Test action 2: Attach a provided VOL1 to VM1 +* **Test assertion 1:** Verify VOL1 is in 'in-use' status +* Test action 3: Detach VOL1 from VM1 +* **Test assertion 2:** Verify VOL1 is in 'available' status +* Test action 4: Create a server VM2 +* Test action 5: Attach a provided VOL2 to VM2 and wait for VOL2 to reach 'in-use' status +* Test action 6: Retrieve VOL2's attachment information ATTCH_INFO +* **Test assertion 3:** Verify ATTCH_INFO is correct +* Test action 7: Create a server VM3 and wait for VM3 to reach 'ACTIVE' status +* Test action 8: Attach a non-existent volume to VM3 +* **Test assertion 4:** Verify attach volume failed, a 'NOT FOUND' error is returned in the response +* Test action 9: Detach a volume from a server by using an invalid volume ID +* **Test assertion 5:** Verify detach volume failed, a 'NOT FOUND' error is returned in the response + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates the volume API ability of attaching a volume to a server +and detaching a volume from a server. Specifically, the test verifies that: + +* Volumes can be attached and detached from servers. +* Volume attachment information can be retrieved. +* Attach and detach a volume using an invalid volume ID is not allowed. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------------------------------- +Test Case 2 - Volume service availability zone operations with the Cinder v2 API +-------------------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_availability_zone.AvailabilityZoneV2TestJSON.test_get_availability_zone_list + +tempest.api.volume.test_availability_zone.AvailabilityZoneTestJSON.test_get_availability_zone_list + +Note: the second test case is the alias of the first one. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' +* Test action 1: List all existent availability zones +* **Test assertion 1:** Verify the availability zone list length is greater than 0 + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of listing availability zones. +Specifically, the test verifies that: + +* Availability zones can be listed. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------------- +Test Case 3 - Volume cloning operations with the Cinder v2 API +-------------------------------------------------------------- + +Test case specification +----------------------- + tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete_as_clone + +tempest.api.volume.test_volumes_get.VolumesGetTest.test_volume_create_get_update_delete_as_clone + +Note: the second test case is the alias of the first one. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API +* Cinder volume clones feature is enabled + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' +* Test action 1: Create a volume VOL1 +* Test action 2: Create a volume VOL2 from source volume VOL1 with a specific name and metadata +* Test action 2: Wait for VOL2 to reach 'available' status +* **Test assertion 1:** Verify the name of VOL2 is correct +* Test action 3: Retrieve VOL2's detail information +* **Test assertion 2:** Verify the retrieved volume name, ID and metadata are the same as VOL2 +* **Test assertion 3:** Verify VOL2's bootable flag is 'False' +* Test action 4: Update the name of VOL2 with the original value +* Test action 5: Update the name of VOL2 with a new value +* **Test assertion 4:** Verify the name of VOL2 is updated successfully +* Test action 6: Create a volume VOL3 with no name specified and a description contains characters '@#$%^*' +* **Test assertion 5:** Verify VOL3 is created successfully +* Test action 7: Update the name of VOL3 and description with the original value +* **Test assertion 6:** Verify VOL3's bootable flag is 'False' + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of creating a cloned volume from a source volume, +getting cloned volume detail information and updating cloned volume attributes. + +Specifically, the test verifies that: + +* Cloned volume can be created from a source volume. +* Cloned volume detail information can be retrieved. +* Cloned volume detail information can be updated. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------------------- +Test Case 4 - Image copy-to-volume operations with the Cinder v2 API +-------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_volume_bootable tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete_from_image + +tempest.api.volume.test_volumes_get.VolumesActionsTest.test_volume_bootable +tempest.api.volume.test_volumes_get.VolumesGetTest.test_volume_create_get_update_delete_from_image + +Note: the last 2 test cases are the alias of the former 2. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' +* Test action 1: Set a provided volume VOL1's bootable flag to 'True' +* Test action 2: Retrieve VOL1's bootable flag +* **Test assertion 1:** Verify VOL1's bootable flag is 'True' +* Test action 3: Set a provided volume VOL1's bootable flag to 'False' +* Test action 4: Retrieve VOL1's bootable flag +* **Test assertion 2:** Verify VOL1's bootable flag is 'False' +* Test action 5: Create a bootable volume VOL2 from one image with a specific name and metadata +* Test action 6: Wait for VOL2 to reach 'available' status +* **Test assertion 3:** Verify the name of VOL2 name is correct +* Test action 7: Retrieve VOL2's information +* **Test assertion 4:** Verify the retrieved volume name, ID and metadata are the same as VOL2 +* **Test assertion 5:** Verify VOL2's bootable flag is 'True' +* Test action 8: Update the name of VOL2 with the original value +* Test action 9: Update the name of VOL2 with a new value +* **Test assertion 6:** Verify the name of VOL2 is updated successfully +* Test action 10: Create a volume VOL3 with no name specified and a description contains characters '@#$%^*' +* **Test assertion 7:** Verify VOL3 is created successfully +* Test action 11: Update the name of VOL3 and description with the original value +* **Test assertion 8:** Verify VOL3's bootable flag is 'True' + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of updating volume's bootable flag and creating +a bootable volume from an image, getting bootable volume detail information and updating bootable volume. + +Specifically, the test verifies that: + +* Volume bootable flag can be set and retrieved. +* Bootable volume can be created from a source volume. +* Bootable volume detail information can be retrieved. +* Bootable volume detail information can be updated. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------------------------------------------- +Test Case 5 - Volume creation and deletion operations with the Cinder v2 API +---------------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_invalid_size +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_source_volid +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_volume_type +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_out_passing_size +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_size_negative +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_size_zero + +tempest.api.volume.test_volumes_get.VolumesGetTest.test_volume_create_get_update_delete +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_invalid_size +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_nonexistent_source_volid +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_nonexistent_volume_type + +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_without_passing_size +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_without_passing_size + +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_size_negative +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_size_zero + +Note: test cases 8 to 11 are the alias of the fist 4 test cases, test cases 12 and 13 are both alias of +test case 5, and test cases 14 and 15 are the alias of the cases 6 and 7, respectively. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of OpenStack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' +* Test action 1: Create a volume VOL1 with a specific name and metadata +* Test action 2: Wait for VOL1 to reach 'available' status +* **Test assertion 1:** Verify the name of VOL1 is correct +* Test action 3: Retrieve VOL1's information +* **Test assertion 2:** Verify the retrieved volume name, ID and metadata are the same as VOL1 +* **Test assertion 3:** Verify VOL1's bootable flag is 'False' +* Test action 4: Update the name of VOL1 with the original value +* Test action 5: Update the name of VOL1 with a new value +* **Test assertion 4:** Verify the name of VOL1 is updated successfully +* Test action 6: Create a volume VOL2 with no name specified and a description contains characters '@#$%^*' +* **Test assertion 5:** Verify VOL2 is created successfully +* Test action 7: Update the name of VOL2 and description with the original value +* **Test assertion 6:** Verify VOL2's bootable flag is 'False' +* Test action 8: Create a volume with an invalid size '#$%' +* **Test assertion 7:** Verify create volume failed, a bad request error is returned in the response +* Test action 9: Create a volume with a nonexistent source volume +* **Test assertion 8:** Verify create volume failed, a 'Not Found' error is returned in the response +* Test action 10: Create a volume with a nonexistent volume type +* **Test assertion 9:** Verify create volume failed, a 'Not Found' error is returned in the response +* Test action 11: Create a volume without passing a volume size +* **Test assertion 10:** Verify create volume failed, a bad request error is returned in the response +* Test action 12: Create a volume with a negative volume size +* **Test assertion 11:** Verify create volume failed, a bad request error is returned in the response +* Test action 13: Create a volume with volume size '0' +* **Test assertion 12:** Verify create volume failed, a bad request error is returned in the response + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of creating a volume, getting volume +detail information and updating volume, the reference is, +Specifically, the test verifies that: + +* Volume can be created from a source volume. +* Volume detail information can be retrieved/updated. +* Create a volume with an invalid size is not allowed. +* Create a volume with a nonexistent source volume or volume type is not allowed. +* Create a volume without passing a volume size is not allowed. +* Create a volume with a negative volume size is not allowed. +* Create a volume with volume size '0' is not allowed. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------------------------------- +Test Case 6 - Volume service extension listing operations with the Cinder v2 API +-------------------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_extensions.ExtensionsV2TestJSON.test_list_extensions + +tempest.api.volume.test_extensions.ExtensionsTestJSON.test_list_extensions + +Note: the second test case is the alias of the first one. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API +* At least one Cinder extension is configured + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: List all cinder service extensions +* **Test assertion 1:** Verify all extensions are list in the extension list + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of listing all existent volume service extensions. + +* Cinder service extensions can be listed. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------------------------- +Test Case 7 - Volume GET operations with the Cinder v2 API +---------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_get_invalid_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_get_volume_without_passing_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_volume_get_nonexistent_volume_id + +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_get_invalid_volume_id +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_get_volume_without_passing_volume_id" +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_volume_get_nonexistent_volume_id + +Note: the latter 3 test cases is the alias of the first 3 ones. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Retrieve a volume with an invalid volume ID +* **Test assertion 1:** Verify retrieve volume failed, a 'Not Found' error is returned in the response +* Test action 2: Retrieve a volume with an empty volume ID +* **Test assertion 2:** Verify retrieve volume failed, a 'Not Found' error is returned in the response +* Test action 3: Retrieve a volume with a nonexistent volume ID +* **Test assertion 3:** Verify retrieve volume failed, a 'Not Found' error is returned in the response + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of getting volumes. +Specifically, the test verifies that: + +* Get a volume with an invalid/an empty/a nonexistent volume ID is not allowed. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------------- +Test Case 8 - Volume listing operations with the Cinder v2 API +-------------------------------------------------------------- + +Test case specification +----------------------- + tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_by_name tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_by_name @@ -43,40 +519,475 @@ tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volumes_list_by_ tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volumes_list_by_status tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volumes_list_details_by_availability_zone tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volumes_list_details_by_status -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_attach_volumes_with_nonexistent_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_invalid_size -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_snapshot_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_source_volid -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_volume_type -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_out_passing_size -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_size_negative -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_size_zero -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_without_passing_size -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_delete_invalid_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_delete_volume_without_passing_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_detach_volumes_with_invalid_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_get_invalid_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_get_volume_without_passing_volume_id tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_detail_with_invalid_status tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_detail_with_nonexistent_name tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_with_invalid_status tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_with_nonexistent_name +tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_pagination +tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_with_multiple_params +tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_pagination + +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_by_name +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_details_by_name +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_param_display_name_and_status +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_with_detail_param_display_name_and_status +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_with_detail_param_metadata +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_with_details +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_with_param_metadata +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_by_availability_zone +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_by_status +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_details_by_availability_zone +tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_details_by_status +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_list_volumes_detail_with_invalid_status +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_list_volumes_detail_with_nonexistent_name +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_list_volumes_with_invalid_status +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_list_volumes_with_nonexistent_name +tempest.api.volume.v2.test_volumes_list.VolumesListTestJSON.test_volume_list_details_pagination +tempest.api.volume.v2.test_volumes_list.VolumesListTestJSON.test_volume_list_details_with_multiple_params +tempest.api.volume.v2.test_volumes_list.VolumesListTestJSON.test_volume_list_pagination + +Note: the latter 19 test cases is the alias of the first 19 ones. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API +* The backing file for the volume group that Nova uses has space for at least 3 1G volumes + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: List all existent volumes +* **Test assertion 1:** Verify the volume list is complete +* Test action 2: List existent volumes and filter the volume list by volume name +* **Test assertion 2:** Verify the length of filtered volume list is 1 and the retrieved volume is correct +* Test action 3: List existent volumes in detail and filter the volume list by volume name +* **Test assertion 3:** Verify the length of filtered volume list is 1 and the retrieved volume is correct +* Test action 4: List existent volumes and filter the volume list by volume name and status 'available' +* **Test assertion 4:** Verify the name and status parameters of the fetched volume are correct +* Test action 5: List existent volumes in detail and filter the volume list by volume name and status 'available' +* **Test assertion 5:** Verify the name and status parameters of the fetched volume are correct +* Test action 6: List all existent volumes in detail and filter the volume list by volume metadata +* **Test assertion 6:** Verify the metadata parameter of the fetched volume is correct +* Test action 7: List all existent volumes in detail +* **Test assertion 7:** Verify the volume list is complete +* Test action 8: List all existent volumes and filter the volume list by volume metadata +* **Test assertion 8:** Verify the metadata parameter of the fetched volume is correct +* Test action 9: List existent volumes and filter the volume list by availability zone +* **Test assertion 9:** Verify the availability zone parameter of the fetched volume is correct +* Test action 10: List all existent volumes and filter the volume list by volume status 'available' +* **Test assertion 10:** Verify the status parameter of the fetched volume is correct +* Test action 11: List existent volumes in detail and filter the volume list by availability zone +* **Test assertion 11:** Verify the availability zone parameter of the fetched volume is correct +* Test action 12: List all existent volumes in detail and filter the volume list by volume status 'available' +* **Test assertion 12:** Verify the status parameter of the fetched volume is correct +* Test action 13: List all existent volumes in detail and filter the volume list by an invalid volume status 'null' +* **Test assertion 13:** Verify the filtered volume list is empty +* Test action 14: List all existent volumes in detail and filter the volume list by a non-existent volume name +* **Test assertion 14:** Verify the filtered volume list is empty +* Test action 15: List all existent volumes and filter the volume list by an invalid volume status 'null' +* **Test assertion 15:** Verify the filtered volume list is empty +* Test action 16: List all existent volumes and filter the volume list by a non-existent volume name +* **Test assertion 16:** Verify the filtered volume list is empty +* Test action 17: List all existent volumes in detail and paginate the volume list by desired volume IDs +* **Test assertion 17:** Verify only the desired volumes are listed in the filtered volume list +* Test action 18: List all existent volumes in detail and filter the volume list by volume status 'available' and display limit '2' +* Test action 19: Sort the filtered volume list by IDs in ascending order +* **Test assertion 18:** Verify the length of filtered volume list is 2 +* **Test assertion 19:** Verify the status of retrieved volumes is correct +* **Test assertion 20:** Verify the filtered volume list is sorted correctly +* Test action 20: List all existent volumes in detail and filter the volume list by volume status 'available' and display limit '2' +* Test action 21: Sort the filtered volume list by IDs in descending order +* **Test assertion 21:** Verify the length of filtered volume list is 2 +* **Test assertion 22:** Verify the status of retrieved volumes is correct +* **Test assertion 23:** Verify the filtered volume list is sorted correctly +* Test action 22: List all existent volumes and paginate the volume list by desired volume IDs +* **Test assertion 24:** Verify only the desired volumes are listed in the filtered volume list + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of getting a list of volumes and filtering the volume list. +Specifically, the test verifies that: + +* Get a list of volumes (in detail) successful. +* Get a list of volumes (in detail) and filter volumes by name/status/metadata/availability zone successful. +* Volume list pagination functionality is working. +* Get a list of volumes in detail using combined condition successful. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------------------------------- +Test Case 9 - Volume metadata operations with the Cinder v2 API +--------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_create_get_delete_volume_metadata +tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_update_volume_metadata_item + +tempest.api.volume.test_volume_metadata.VolumesMetadataTest.test_crud_volume_metadata +tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_crud_volume_metadata + +tempest.api.volume.test_volume_metadata.VolumesMetadataTest.test_update_volume_metadata_item +tempest.api.volume.test_volume_metadata.VolumesMetadataTest.test_update_show_volume_metadata_item + +Note: Test case 3 and 4 are the alias of the first test case, and the last 2 test cases +are the alias of the second test case. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of OpenStack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create metadata for a provided volume VOL1 +* Test action 2: Get the metadata of VOL1 +* **Test assertion 1:** Verify the metadata of VOL1 is correct +* Test action 3: Update the metadata of VOL1 +* **Test assertion 2:** Verify the metadata of VOL1 is updated +* Test action 4: Delete one metadata item 'key1' of VOL1 +* **Test assertion 3:** Verify the metadata item 'key1' is deleted +* Test action 5: Create metadata for a provided volume VOL2 +* **Test assertion 4:** Verify the metadata of VOL2 is correct +* Test action 6: Update one metadata item 'key3' of VOL2 +* **Test assertion 5:** Verify the metadata of VOL2 is updated + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of creating metadata for a volume, getting the +metadata of a volume, updating volume metadata and deleting a metadata item of a volume. +Specifically, the test verifies that: + +* Create metadata for volume successfully. +* Get metadata of volume successfully. +* Update volume metadata and metadata item successfully. +* Delete metadata item of a volume successfully. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +--------------------------------------------------------------------------------- +Test Case 10 - Verification of read-only status on volumes with the Cinder v2 API +--------------------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_volume_readonly_update + +tempest.api.volume.test_volumes_actions.VolumesActionsTest.test_volume_readonly_update + +Note: the second test case is the alias of the first one. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Update a provided volume VOL1's read-only access mode to 'True' +* **Test assertion 1:** Verify VOL1 is in read-only access mode +* Test action 2: Update a provided volume VOL1's read-only access mode to 'False' +* **Test assertion 2:** Verify VOL1 is not in read-only access mode + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of setting and updating volume read-only access mode. +Specifically, the test verifies that: + +* Volume read-only access mode can be set and updated. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +------------------------------------------------------------------- +Test Case 11 - Volume reservation operations with the Cinder v2 API +------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_reserve_unreserve_volume tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_reserve_volume_with_negative_volume_status tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_reserve_volume_with_nonexistent_volume_id tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_unreserve_volume_with_nonexistent_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_empty_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_invalid_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_nonexistent_volume_id + +tempest.api.volume.test_volumes_actions.VolumesActionsTest.test_reserve_unreserve_volume +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_reserve_volume_with_negative_volume_status +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_reserve_volume_with_nonexistent_volume_id +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_unreserve_volume_with_nonexistent_volume_id + +Note: the last 4 test cases are the alias of the first 4 ones. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Update a provided volume VOL1 as reserved +* **Test assertion 1:** Verify VOL1 is in 'attaching' status +* Test action 2: Update VOL1 as un-reserved +* **Test assertion 2:** Verify VOL1 is in 'available' status +* Test action 3: Update a provided volume VOL2 as reserved +* Test action 4: Update VOL2 as reserved again +* **Test assertion 3:** Verify update VOL2 status failed, a bad request error is returned in the response +* Test action 5: Update VOL2 as un-reserved +* Test action 6: Update a non-existent volume as reserved by using an invalid volume ID +* **Test assertion 4:** Verify update non-existent volume as reserved failed, a 'Not Found' error is returned in the response +* Test action 7: Update a non-existent volume as un-reserved by using an invalid volume ID +* **Test assertion 5:** Verify update non-existent volume as un-reserved failed, a 'Not Found' error is returned in the response + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of reserving and un-reserving volumes. +Specifically, the test verifies that: + +* Volume can be reserved and un-reserved. +* Update a non-existent volume as reserved is not allowed. +* Update a non-existent volume as un-reserved is not allowed. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +---------------------------------------------------------------------------------- +Test Case 12 - Volume snapshot creation/deletion operations with the Cinder v2 API +---------------------------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_create_get_delete_snapshot_metadata +tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_update_snapshot_metadata_item +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_snapshot_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_delete_invalid_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_delete_volume_without_passing_volume_id tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_volume_delete_nonexistent_volume_id -tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_volume_get_nonexistent_volume_id tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshot_create_get_list_update_delete +tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_volume_from_snapshot tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshots_list_details_with_params tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshots_list_with_params -tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_volume_from_snapshot -tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_details_with_params -tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_with_params tempest.api.volume.test_volumes_snapshots_negative.VolumesV2SnapshotNegativeTestJSON.test_create_snapshot_with_nonexistent_volume_id tempest.api.volume.test_volumes_snapshots_negative.VolumesV2SnapshotNegativeTestJSON.test_create_snapshot_without_passing_volume_id -tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_pagination -tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_with_multiple_params -tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_pagination + +tempest.api.volume.test_snapshot_metadata.SnapshotMetadataTestJSON.test_crud_snapshot_metadata +tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_crud_snapshot_metadata + +tempest.api.volume.test_snapshot_metadata.SnapshotMetadataTestJSON.test_update_snapshot_metadata_item +tempest.api.volume.test_snapshot_metadata.SnapshotMetadataTestJSON.test_update_show_snapshot_metadata_item + +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_nonexistent_snapshot_id +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_delete_invalid_volume_id +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_delete_volume_without_passing_volume_id +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_volume_delete_nonexistent_volume_id +tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_snapshot_create_get_list_update_delete +tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_volume_from_snapshot + +tempest.api.volume.test_volumes_snapshots_list.VolumesSnapshotListTestJSON.test_snapshots_list_details_with_params +tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_details_with_params + +tempest.api.volume.test_volumes_snapshots_list.VolumesSnapshotListTestJSON.test_snapshots_list_with_params +tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_with_params + +tempest.api.volume.test_volumes_snapshots_negative.VolumesSnapshotNegativeTestJSON.test_create_snapshot_with_nonexistent_volume_id +tempest.api.volume.test_volumes_snapshots_negative.VolumesSnapshotNegativeTestJSON.test_create_snapshot_without_passing_volume_id + +Note: test case 13 and 14 are the alias of test case 1, test case 15 and 16 are the alias of test case 2, +test case 17 to 22 are the alias of test case 3 to 8 respectively, test case 23 and 24 are the alias of +test case 9, test case 25 and 26 are the alias of test case 10, and test case 27 and 28 are the alias of +test case 11 and 12 respectively. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of OpenStack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Create metadata for a provided snapshot SNAP1 +* Test action 2: Get the metadata of SNAP1 +* **Test assertion 1:** Verify the metadata of SNAP1 is correct +* Test action 3: Update the metadata of SNAP1 +* **Test assertion 2:** Verify the metadata of SNAP1 is updated +* Test action 4: Delete one metadata item 'key3' of SNAP1 +* **Test assertion 3:** Verify the metadata item 'key3' is deleted +* Test action 5: Create metadata for a provided snapshot SNAP2 +* **Test assertion 4:** Verify the metadata of SNAP2 is correct +* Test action 6: Update one metadata item 'key3' of SNAP2 +* **Test assertion 5:** Verify the metadata of SNAP2 is updated +* Test action 7: Create a volume with a nonexistent snapshot +* **Test assertion 6:** Verify create volume failed, a 'Not Found' error is returned in the response +* Test action 8: Delete a volume with an invalid volume ID +* **Test assertion 7:** Verify delete volume failed, a 'Not Found' error is returned in the response +* Test action 9: Delete a volume with an empty volume ID +* **Test assertion 8:** Verify delete volume failed, a 'Not Found' error is returned in the response +* Test action 10: Delete a volume with a nonexistent volume ID +* **Test assertion 9:** Verify delete volume failed, a 'Not Found' error is returned in the response +* Test action 11: Create a snapshot SNAP2 from a provided volume VOL1 +* Test action 12: Retrieve SNAP2's detail information +* **Test assertion 10:** Verify SNAP2 is created from VOL1 +* Test action 13: Update the name and description of SNAP2 +* **Test assertion 11:** Verify the name and description of SNAP2 are updated in the response body of update snapshot API +* Test action 14: Retrieve SNAP2's detail information +* **Test assertion 12:** Verify the name and description of SNAP2 are correct +* Test action 15: Delete SNAP2 +* Test action 16: Create a volume VOL2 with a volume size +* Test action 17: Create a snapshot SNAP3 from VOL2 +* Test action 18: Create a volume VOL3 from SNAP3 with a bigger volume size +* Test action 19: Retrieve VOL3's detail information +* **Test assertion 13:** Verify volume size and source snapshot of VOL3 are correct +* Test action 20: List all snapshots in detail and filter the snapshot list by name +* **Test assertion 14:** Verify the filtered snapshot list is correct +* Test action 21: List all snapshots in detail and filter the snapshot list by status +* **Test assertion 15:** Verify the filtered snapshot list is correct +* Test action 22: List all snapshots in detail and filter the snapshot list by name and status +* **Test assertion 16:** Verify the filtered snapshot list is correct +* Test action 23: List all snapshots and filter the snapshot list by name +* **Test assertion 17:** Verify the filtered snapshot list is correct +* Test action 24: List all snapshots and filter the snapshot list by status +* **Test assertion 18:** Verify the filtered snapshot list is correct +* Test action 25: List all snapshots and filter the snapshot list by name and status +* **Test assertion 19:** Verify the filtered snapshot list is correct +* Test action 26: Create a snapshot from a nonexistent volume by using an invalid volume ID +* **Test assertion 20:** Verify create snapshot failed, a 'Not Found' error is returned in the response +* Test action 27: Create a snapshot from a volume by using an empty volume ID +* **Test assertion 21:** Verify create snapshot failed, a 'Not Found' error is returned in the response + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of managing snapshot and snapshot metadata. +Specifically, the test verifies that: + +* Create metadata for snapshot successfully. +* Get metadata of snapshot successfully. +* Update snapshot metadata and metadata item successfully. +* Delete metadata item of a snapshot successfully. +* Create a volume from a nonexistent snapshot is not allowed. +* Delete a volume using an invalid volume ID is not allowed. +* Delete a volume without passing the volume ID is not allowed. +* Delete a non-existent volume is not allowed. +* Create snapshot successfully. +* Get snapshot's detail information successfully. +* Update snapshot attributes successfully. +* Delete snapshot successfully. +* Creates a volume and a snapshot passing a size different from the source successfully. +* List snapshot details by display_name and status filters successfully. +* Create a snapshot from a nonexistent volume is not allowed. +* Create a snapshot from a volume without passing the volume ID is not allowed. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A + +-------------------------------------------------------------- +Test Case 13 - Volume update operations with the Cinder v2 API +-------------------------------------------------------------- + +Test case specification +----------------------- + +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_empty_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_invalid_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_nonexistent_volume_id + +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_update_volume_with_empty_volume_id +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_update_volume_with_invalid_volume_id +tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_update_volume_with_nonexistent_volume_id + +Note: the last 3 test cases are the alias of the first 3 ones. +Alias should always be included so that the test run will be tempest version agnostic, +which can be used to test different version of Openstack. + +Test preconditions +------------------ + +* Volume extension API + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Test execution +'''''''''''''' + +* Test action 1: Update a volume by using an empty volume ID +* **Test assertion 1:** Verify update volume failed, a 'Not Found' error is returned in the response +* Test action 2: Update a volume by using an invalid volume ID +* **Test assertion 2:** Verify update volume failed, a 'Not Found' error is returned in the response +* Test action 3: Update a non-existent volume by using a random generated volume ID +* **Test assertion 3:** Verify update volume failed, a 'Not Found' error is returned in the response + +Pass / fail criteria +'''''''''''''''''''' + +This test case evaluates the volume API ability of updating volume attributes. +Specifically, the test verifies that: + +* Update a volume without passing the volume ID is not allowed. +* Update a volume using an invalid volume ID is not allowed. +* Update a non-existent volume is not allowed. + +In order to pass this test, all test assertions listed in the test execution above need to pass. + +Post conditions +--------------- + +N/A |