summaryrefslogtreecommitdiffstats
path: root/docs/testing
diff options
context:
space:
mode:
Diffstat (limited to 'docs/testing')
-rw-r--r--docs/testing/developer/testscope/index.rst93
-rw-r--r--docs/testing/user/certificationworkflow/ApplicationForm.rst10
-rw-r--r--docs/testing/user/certificationworkflow/index.rst122
-rw-r--r--docs/testing/user/cvpaddendum/index.rst388
-rw-r--r--docs/testing/user/testspecification/dynamicnetwork/index.rst413
-rw-r--r--docs/testing/user/testspecification/securitygroup/index.rst450
-rw-r--r--docs/testing/user/testspecification/vimoperationscompute/index.rst661
-rw-r--r--docs/testing/user/testspecification/vimoperationsvolume/index.rst999
-rw-r--r--docs/testing/user/testspecification/vping/index.rst279
-rw-r--r--docs/testing/user/userguide/testing_guide.rst58
10 files changed, 3381 insertions, 92 deletions
diff --git a/docs/testing/developer/testscope/index.rst b/docs/testing/developer/testscope/index.rst
index b2dceec4..35ff60af 100644
--- a/docs/testing/developer/testscope/index.rst
+++ b/docs/testing/developer/testscope/index.rst
@@ -118,6 +118,8 @@ Basic server operations in the Compute API
| tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_server_details
| tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_created_server_vcpus
| tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_server_details
+| tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_server_status
+| tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_deleted_server
Retrieve volume information through the Compute API
@@ -283,12 +285,14 @@ Volume service availability zone operations with the Cinder v2 API
------------------------------------------------------------------
| tempest.api.volume.test_availability_zone.AvailabilityZoneV2TestJSON.test_get_availability_zone_list
+| tempest.api.volume.test_availability_zone.AvailabilityZoneTestJSON.test_get_availability_zone_list
Volume cloning operations with the Cinder v2 API
------------------------------------------------
| tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete_as_clone
+| tempest.api.volume.test_volumes_get.VolumesGetTest.test_volume_create_get_update_delete_as_clone
Image copy-to-volume operations with the Cinder v2 API
@@ -296,6 +300,8 @@ Image copy-to-volume operations with the Cinder v2 API
| tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_volume_bootable
| tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete_from_image
+| tempest.api.volume.test_volumes_get.VolumesActionsTest.test_volume_bootable
+| tempest.api.volume.test_volumes_get.VolumesGetTest.test_volume_create_get_update_delete_from_image
Volume creation and deletion operations with the Cinder v2 API
@@ -308,12 +314,21 @@ Volume creation and deletion operations with the Cinder v2 API
| tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_out_passing_size
| tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_size_negative
| tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_size_zero
+| tempest.api.volume.test_volumes_get.VolumesGetTest.test_volume_create_get_update_delete
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_invalid_size
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_nonexistent_source_volid
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_nonexistent_volume_type
+| tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_without_passing_size
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_without_passing_size
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_size_negative
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_size_zero
Volume service extension listing operations with the Cinder v2 API
------------------------------------------------------------------
| tempest.api.volume.test_extensions.ExtensionsV2TestJSON.test_list_extensions
+| tempest.api.volume.test_extensions.ExtensionsTestJSON.test_list_extensions
Volume GET operations with the Cinder v2 API
@@ -322,6 +337,9 @@ Volume GET operations with the Cinder v2 API
| tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_get_invalid_volume_id
| tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_get_volume_without_passing_volume_id
| tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_volume_get_nonexistent_volume_id
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_get_invalid_volume_id
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_get_volume_without_passing_volume_id
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_volume_get_nonexistent_volume_id
Volume listing operations with the Cinder v2 API
@@ -346,6 +364,25 @@ Volume listing operations with the Cinder v2 API
| tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_pagination
| tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_with_multiple_params
| tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_pagination
+| tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list
+| tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_by_name
+| tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_details_by_name
+| tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_param_display_name_and_status
+| tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_with_detail_param_display_name_and_status
+| tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_with_detail_param_metadata
+| tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_with_details
+| tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_with_param_metadata
+| tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_by_availability_zone
+| tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_by_status
+| tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_details_by_availability_zone
+| tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_details_by_status
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_list_volumes_detail_with_invalid_status
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_list_volumes_detail_with_nonexistent_name
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_list_volumes_with_invalid_status
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_list_volumes_with_nonexistent_name
+| tempest.api.volume.v2.test_volumes_list.VolumesListTestJSON.test_volume_list_details_pagination
+| tempest.api.volume.v2.test_volumes_list.VolumesListTestJSON.test_volume_list_details_with_multiple_params
+| tempest.api.volume.v2.test_volumes_list.VolumesListTestJSON.test_volume_list_pagination
Volume metadata operations with the Cinder v2 API
@@ -353,12 +390,17 @@ Volume metadata operations with the Cinder v2 API
| tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_create_get_delete_volume_metadata
| tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_update_volume_metadata_item
+| tempest.api.volume.test_volume_metadata.VolumesMetadataTest.test_crud_volume_metadata
+| tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_crud_volume_metadata
+| tempest.api.volume.test_volume_metadata.VolumesMetadataTest.test_update_volume_metadata_item
+| tempest.api.volume.test_volume_metadata.VolumesMetadataTest.test_update_show_volume_metadata_item
Verification of read-only status on volumes with the Cinder v2 API
------------------------------------------------------------------
| tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_volume_readonly_update
+| tempest.api.volume.test_volumes_actions.VolumesActionsTest.test_volume_readonly_update
Volume reservation operations with the Cinder v2 API
@@ -368,6 +410,10 @@ Volume reservation operations with the Cinder v2 API
| tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_reserve_volume_with_negative_volume_status
| tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_reserve_volume_with_nonexistent_volume_id
| tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_unreserve_volume_with_nonexistent_volume_id
+| tempest.api.volume.test_volumes_actions.VolumesActionsTest.test_reserve_unreserve_volume
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_reserve_volume_with_negative_volume_status
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_reserve_volume_with_nonexistent_volume_id
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_unreserve_volume_with_nonexistent_volume_id
Volume snapshot creation/deletion operations with the Cinder v2 API
@@ -385,6 +431,22 @@ Volume snapshot creation/deletion operations with the Cinder v2 API
| tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshots_list_with_params
| tempest.api.volume.test_volumes_snapshots_negative.VolumesV2SnapshotNegativeTestJSON.test_create_snapshot_with_nonexistent_volume_id
| tempest.api.volume.test_volumes_snapshots_negative.VolumesV2SnapshotNegativeTestJSON.test_create_snapshot_without_passing_volume_id
+| tempest.api.volume.test_snapshot_metadata.SnapshotMetadataTestJSON.test_crud_snapshot_metadata
+| tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_crud_snapshot_metadata
+| tempest.api.volume.test_snapshot_metadata.SnapshotMetadataTestJSON.test_update_snapshot_metadata_item
+| tempest.api.volume.test_snapshot_metadata.SnapshotMetadataTestJSON.test_update_show_snapshot_metadata_item
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_nonexistent_snapshot_id
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_delete_invalid_volume_id
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_delete_volume_without_passing_volume_id
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_volume_delete_nonexistent_volume_id
+| tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_snapshot_create_get_list_update_delete
+| tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_volume_from_snapshot
+| tempest.api.volume.test_volumes_snapshots_list.VolumesSnapshotListTestJSON.test_snapshots_list_details_with_params
+| tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_details_with_params
+| tempest.api.volume.test_volumes_snapshots_list.VolumesSnapshotListTestJSON.test_snapshots_list_with_params
+| tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_with_params
+| tempest.api.volume.test_volumes_snapshots_negative.VolumesSnapshotNegativeTestJSON.test_create_snapshot_with_nonexistent_volume_id
+| tempest.api.volume.test_volumes_snapshots_negative.VolumesSnapshotNegativeTestJSON.test_create_snapshot_without_passing_volume_id
Volume update operations with the Cinder v2 API
@@ -393,6 +455,9 @@ Volume update operations with the Cinder v2 API
| tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_empty_volume_id
| tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_invalid_volume_id
| tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_nonexistent_volume_id
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_update_volume_with_empty_volume_id
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_update_volume_with_invalid_volume_id
+| tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_update_volume_with_nonexistent_volume_id
---------------------------
@@ -411,6 +476,14 @@ Verify high availability of OpenStack controller services
| opnfv.ha.tc010.disk_I/O_block
| opnfv.ha.tc011.load_balance_service_down
+----------------------------------------
+Test Area vPing - Basic VNF Connectivity
+----------------------------------------
+
+| opnfv.vping.userdata
+| opnfv.vping.ssh
+
+
Optional CVP Test Areas
========================
@@ -561,3 +634,23 @@ Test Case 25: IPv6 Address Assignment - Dual Net, Dual Stack, Multiple Prefixes,
| tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_slaac
+------------------------------------------------------------------------
+Filtering Packets Based on Security Rules and Port Security in Data Path
+------------------------------------------------------------------------
+
+| tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_port_security_macspoofing_port
+| tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic
+| tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_in_tenant_traffic
+| tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_multiple_security_groups
+| tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_port_security_disable_security_group
+| tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_port_update_new_security_group
+
+------------------------------------------------------------
+Dynamic Network Runtime Operations Through the Life of a VNF
+------------------------------------------------------------
+
+| tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops
+| tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic
+| tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details
+| tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_instance_port_admin_state
+| tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state
diff --git a/docs/testing/user/certificationworkflow/ApplicationForm.rst b/docs/testing/user/certificationworkflow/ApplicationForm.rst
index e69de29b..9199d71e 100644
--- a/docs/testing/user/certificationworkflow/ApplicationForm.rst
+++ b/docs/testing/user/certificationworkflow/ApplicationForm.rst
@@ -0,0 +1,10 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Intel Corporation and others.
+
+.._cvp-application-form:
+
+======================================================
+OPNFV COMPLIANCE VERIFICATION PROGRAM APPLICATION FORM
+======================================================
+
diff --git a/docs/testing/user/certificationworkflow/index.rst b/docs/testing/user/certificationworkflow/index.rst
index e69de29b..d26fe5d8 100644
--- a/docs/testing/user/certificationworkflow/index.rst
+++ b/docs/testing/user/certificationworkflow/index.rst
@@ -0,0 +1,122 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) Intel Corporation, Ericsson AB, Huawei, and others
+
+================================================================
+OPNFV Compliance Verification Program certification workflow
+================================================================
+
+.. toctree::
+ :maxdepth: 2
+
+Introduction
+============
+
+This document provides guidance for testers on how to obtain OPNFV compliance
+certification. The OPNFV Compliance Verification Program (CVP) is administered by
+the OPNFV Compliance and Certification (C&C) committee.
+
+For further information about the workflow and general inquiries about the
+program, please check out the CVP web portal https://cvp.opnfv.org, or contact
+the C&C committee by email address cvp@opnfv.org. This email address should be used
+for all communication with the CVP.
+
+Step 1: Applying
+================
+
+A tester should start the process by completing an application.
+The application form can found on the CVP portal and the following
+information should be provided:
+
+ - Organization name
+ - Organization website (if public)
+ - Product name and/or identifier
+ - Product specifications
+ - Product public documentation
+ - Product categories, choose one: (i) software and hardware (ii) software
+ and third party hardware (please specify)
+ - Primary contact name, business email, postal address and phone number
+ Only the primary email address should be used for
+ official communication with OPNFV CVP.
+ - User ID for CVP web portal
+ The CVP web portal supports (i) Linux Foundation user ID, and (ii) Openstack user
+ ID, in the current release. Specify either one as user ID for the CVP web portal.
+ If a new user ID is needed, visit https://identity.linuxfoundation.org, or
+ https://www.openstack.org/join/register.
+ - Location where the verification testing is to be conducted. Choose one:
+ (internal vendor lab, third-party lab)
+ - If the test is to be conducted by a third-party lab, please specify
+ name and contact information of the third-party lab, including email, address and
+ phone number.
+
+Please email the completed application using the primary contact email
+account in order to establish identity.
+
+Once the application information is received and in order, an email response will be
+sent to the primary contact with confirmation and information to proceed.
+
+[Editor's note:
+No fee has been established at this time for CVP applications. Recommend
+we skip fee for the initial release of CVP.]
+
+Step 2: Testing
+===============
+
+The following documents guide testers to prepare the test environment and run tests:
+
+ - System Requirements and Preparation Guide for OPNFV CVP [2]
+ - OPNFV CVP Test Case Specifications [3]
+ - Dovetail Test Tool User Guide [4]
+
+[Editor's note: The above three documents are still work in progress. Names and
+references will be corrected when they are merged.]
+
+A unique Test ID is generated by the Dovetail tool for each test run. Please take a note
+of this ID for future reference.
+
+Step 3: Submitting Test Results
+===============================
+
+Testers can upload the test results to the CVP web portal https://cvp.opnfv.org.
+By default, the results are visible only to the tester who uploaded the data.
+
+Testers can self-review the test results through the portal until they are ready to ask
+for CVP review. They may also update with or add new test results as needed.
+
+Once the tester is satisfied with the test result, the tester grants access to the test result
+for CVP review via the portal. The test result is identified by the unique Test ID.
+
+When a test result is made visible to the reviewers, the web portal will notify
+cvp@opnfv.org and Cc the primary contact email that a review request has been made and reference
+the Test ID. This will alert the C&C Committee to start the CVP review process.
+
+Step 4: CVP Review
+===================
+
+Upon receiving the email notification and the Test ID, the C&C Committee conducts a
+peer based review of the test result. Persons employed by the same organization that submitted
+the test results or by affiliated organizations will not be part of the reviewers.
+
+The primary contact may be asked via email for any missing information or
+clarification of the application. The reviewers will make a determination and recommend
+compliance or non-compliance to
+the C&C Committee. Normally, the outcome of the review should be communicated
+to the tester within 10 business days after all required information is in order.
+
+If an application is denied, an appeal can be made to the C&C Committee or ultimately to the
+Board of Directors of OPNFV.
+
+Step 5: Grant of Use of Logo
+============================
+
+If an application is approved, further information will be communicated to the tester
+on the guidelines of using OPNFV CVP logos and the status of compliance for promotional purposes.
+
+References
+==========
+
+[1] The OPNFV CVP Guidelines v.16 [Editor's note: link to be provided.]
+[2] System Requirements and Preparation Guide for OPNFV CVP [Editor's note: link to be provided.]
+[3] OPNFV CVP Test Case Specifications v1.0 [Editor's note: link to be provided.]
+[4] Dovetail Test Tool User Guide v1.0 [Editor's note: link to be provided.]
+
diff --git a/docs/testing/user/cvpaddendum/index.rst b/docs/testing/user/cvpaddendum/index.rst
new file mode 100644
index 00000000..ee356ff9
--- /dev/null
+++ b/docs/testing/user/cvpaddendum/index.rst
@@ -0,0 +1,388 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) Intel and others
+
+====================================================================
+Compliance Verification Program - Guidelines Addendum for Danube
+====================================================================
+
+.. toctree::
+ :maxdepth: 2
+
+
+Introduction
+============
+
+This addendum provides a high-level description of the testing scope and
+pass/fail criteria used in the Compliance Verification Program (CVP) for the
+OPNFV Danube release. This information is intended as an overview for CVP
+testers and for the Dovetail Project to help guide test-tool and test-case
+development for the OPNFV Danube release. The Dovetail project is responsible for documenting
+test-case specifications as well as implementing the CVP tool-chain through collaboration
+with the OPNFV testing community. CVP testing focuses on establishing the
+ability of the System Under Test (SUT) to perform NFVI and VIM operations and support
+Service Provider oriented features that ensure manageable, resilient and secure
+networks.
+
+
+Meaning of Compliance
+=====================
+
+OPNFV Compliance indicates adherence to NFV platform behavior defined as
+various platform capabilities or features to prepare, instantiate, operate and remove
+VNFs running on the NFVI. Danube compliance evaluates the ability of a platform
+to support Service Provider network capabilities and workloads that are
+supported in the OPNFV platform as of this release. Compliance test cases shall
+be designated as compulsory or optional based on the maturity of OPNFV
+capabilities as well as industry expectations. Compulsory test cases may for
+example include NFVI management capabilities whereas tests for certain
+high-availability features may be deemed as optional.
+
+Test coverage and pass/fail criteria are
+designed to ensure an acceptable level of compliance but not be so restrictive
+as to disqualify variations in platform implementations, capabilities and features.
+
+
+SUT Assumptions
+===============
+
+Assumptions about the System Under Test (SUT) include ...
+
+ - The minimal specification of physical infrastructure, including controller
+ nodes, compute nodes and networks, is defined by the Pharos specification
+ [2].
+ - The SUT is fully deployed and operational, i.e. SUT deployment tools are
+ out of scope of testing.
+
+
+Scope of Testing
+================
+
+The OPNFV CVP Guidelines [1], as approved by the Board of Directors, outlines
+the key objectives of the CVP as follows:
+ - Help build the market for
+ - OPNFV based infrastructure
+ - applications designed to run on that infrastructure
+ - Reduce adoption risks for end-users
+ - Decrease testing costs by verifying hardware and software platform
+ interfaces and components
+ - Enhance interoperability
+
+The guidelines further directs the scope to be constrained to "features,
+capabilities, components, and interfaces included in an OPNFV release that are
+generally available in the industry (e.g., through adoption by an upstream
+community)", and that compliance verification is evaluated using "functional tests
+that focus on defined interfaces and/or behaviors without regard to the
+the implementation of the underlying system under test".
+
+OPNFV provides a broad range of capabilities, including the reference platform itself
+as well as tools-chains and methodologies for building infrastructures, and
+deploying and testing the platform.
+Not all these aspects are in scope for CVP and not all functions and
+components are tested in the initial version of CVP. For example, the deployment tools
+for the SUT and CI/CD toolchain are currently out of scope.
+Similarly, performance benchmarking related testing is also out of scope or
+for further study. Newer functional areas such as MANO (outside of APIs in the NFVI and
+VIM) are still developing and are for future considerations.
+
+General Approach
+----------------
+
+In order to meet the above objectives for CVP, we aim to follow a general approach
+by first identifying the overall requirements for all stake-holders,
+then analyzing what OPNFV and the upstream communities can effectively test and verify
+presently to derive an initial working scope for CVP, and to recommend what the
+community should strive to achieve in future releases.
+
+The overall requirements for CVP can be cateorized by the basic cloud
+capabilities representing common operations needed by basic VNFs, and additional
+requirements for VNFs that go beyond the common cloud capabilities including
+functional extensions, operational capabilities and additional carrier grade
+requirements.
+
+For the basic NFV requirements, we will analyze the required test cases,
+leverage or improve upon existing test cases in OPNFV projects
+and upstream projects whenever we can, and bridge the gaps when we must, to meet
+these basic requirements.
+
+We are not yet ready to support requirements
+such as hardware portability, carrier grade performance, fault management and other
+operational features, security, MANO and VNF verification.
+These areas are being studied for future considerations.
+
+In some areas, we will start with a limited level of verification
+initially, constrained by what community resources are able to support at this
+time, but still serve a basic need that is not being fulfilled elsewhere.
+In these areas, we bring significant value to the community we
+serve by starting a new area of verification, breaking new ground and
+expanding it in the future.
+
+In other areas, the functions being verified have yet to reach
+wide adoption but are seen as important requirements in NFV,
+or features are only needed for specific NFV use cases but
+an industry consensus about the APIs and behaviors is still deemed beneficial. In such
+cases, we plan to incorporate the test areas as optional. An optional test
+area will not have to be run or passed in order to achieve compliance.
+Optional tests provide an opportunity for vendors to demonstrate compliance with specific OPNFV
+features beyond the mandatory test scope.
+
+Analysis of Scope
+-----------------
+
+Following on these high level objectives and the identified general approach,
+we seek to define the initial verification scope by the analysis summarized
+in the following categories:
+
+1. Basic Cloud Capabilities
+
+The intent of these tests is to verify that the SUT has the required
+capabilities that a basic VNF needs, and these capabilities are implemented
+in a way that enables this basic VNF to run on any OPNFV compliant
+deployment.
+
+A basic VNF can be thought of as a single virtual machine that is networked
+and can perform the simplest network functions, for example, a simple forwarding
+gateway, or a set of such virtual machines connected only by simple virtual network
+services. Running such basic VNF leads to a set of common requirements, including:
+ - image management (Refstack testing Glance API)
+ - identity management (Refstack testing Keystone Identity API)
+ - virtual compute (Refstack testing Nova Compute API)
+ - virtual storage (Refstack testing Cinder API)
+ - virtual networks (Refstack testing Neutron Network API)
+ - forwarding packets through virtual networks in data path
+ - filtering packets based on security rules and port security in data path
+ - dynamic network runtime operations through the life of a VNF (e.g. attach/detach,
+ enable/disable, read stats)
+ - correct behavior after common virtual machine life cycles events (e.g.
+ suspend/resume, reboot, migrate)
+ - simple virtual machine resource scheduling on multiple nodes
+
+OPNFV mainly supports Openstack as the VIM up to the Danube release. The
+VNFs used in the CVP program, and features in scope for the program which are
+considered to be basic to all VNFs, require commercial Openstack distributions
+to support a common basic level of cloud capabilities, and to be compliant
+to a common specification for these capabilities. This requirement significantly
+overlaps with Openstack community's Interop working group's goals, but they are not
+identical. The CVP runs the Openstack Refstack-Compute test cases to verify
+compliance to the basic common API requirements of cloud
+management functions and VNF (as a VM) management for OPNFV.
+Additional NFV specific requirements are added in network data path validation,
+packet filtering by security group rules and port security, life cycle runtime events of
+virtual networks, multiple networks in a topology, validation
+of VNF's functional state after common lifecylce events including reboot, pause,
+suspense, stop/start and cold migration. In addition, the
+basic requirement also verifies that the SUT can allocate VNF resources based
+on simple anti-affinity rules.
+
+The combined test cases help to ensure that these basic operations are always
+supported by a compliant platform and they adhere to
+a common standard to enable portability across OPNFV compliant platforms.
+
+2. NFV specific functional requirements
+
+NFV has functional requirements beyond the basic common cloud
+capabilities, esp. in the networking area. Examples like SDNVPN, IPv6, SFC may
+be considered additional NFV requirements beyond general purpose cloud
+computing. These feature requirements expand beyond common Openstack (or other
+VIM) requirements. OPNFV CVP will incorporate test cases to verify
+compliance in these areas as they become mature. Because these extensions
+may impose new API demands, maturity and industry adoption is a prerequisite for
+making them a mandatory requirement for OPNFV compliance. At the time of Danube,
+we have not identified a new functional area that is mandatory for CVP.
+In the meantime, CVP
+intends to offer tests in some of these areas as an optional extension of the test
+report to be submitted for review, noting that passing these tests will not be
+required to pass OPNFV compliance verification.
+
+SDNVPN is relevant due to the wide adoption of MPLS/BGP based VPNs in wide area
+networks, which makes it necessary for data centers hosting VNFs to be able to
+seamlessly interconnect with such networks. IPv6 is a high priority service provider
+requirement to ease IP addressing and operational issues. SFC is also an important
+NFV requirement, however its implementation has not yet been accepted or adopted
+in the upstream at the time of Danube.
+
+3. High availability
+
+High availability is a common carrier grade requirement. Availability of a
+platform involves many aspects of the SUT, for example hardware or lower layer
+system failures or system overloads, and is also highly dependent on
+configurations. The current OPNFV high availability verification focuses on
+Openstack control service failures and resource overloads, and verifies service
+continuity when the system encounters such failures or resource overloads, and
+also verifies the system heals after a failure episode within a reasonable time
+window. These service HA capabilities are commonly adopted in the industry
+and should be a mandatory requirement.
+
+The current test cases in HA cover the basic area of failure and resource
+overload conditions for a cloud platform's service availability, including all
+of the basic cloud capability services, and basic compute and storage loads,
+so it is a meaningful first step for CVP. We expect additional high availability
+scenarios be extended in future releases.
+
+4. Resiliency
+
+Resiliency testing involves stressing the SUT and verifying its ability
+to absorb stress conditions and still provide an acceptable level of service.
+Resiliency is an important requirement for end-users.
+
+The OPNFV testing projects have started testing
+OPNFV system resiliency in
+the Danube release that can be used to provide limited coverage in this area.
+However, this is a relatively new test methodology in OPNFV, additional study
+and testing experiences are still needed. We defer the resilency testing to
+future CVP releases.
+
+5. Security
+
+Security is among the top priorities as a carrier grade requirement by the
+end-users. Some of the basic common functions, including virtual network isolation,
+security groups, port security and role based access control are already covered as
+part of the basic cloud capabilities that are verified in CVP. These test cases
+however do not yet cover the basic required security capabilities expected of an end-user
+deployment. It is an area that we should address in the near future, to define
+a common set of requirements and develop test cases for verifying those
+requirements.
+
+Another common requirement is security vulnerability scanning.
+While the OPNFV security project integrated tools for security vulnerability
+scanning, this has not been fully analyzed or exercised in Danube release.
+This area needs further work to identify the required level of security for the
+purpose of OPNFV in order to be integrated into the CVP. End-user inputs on
+specific requirements in security is needed.
+
+6. Service assurance
+
+Service assurance (SA) is a broad area of concern for reliability of the NFVI/VIM
+and VNFs, and depends upon multiple subsystems of an NFV platform for essential
+information and control mechanisms. These subsystems include telemetry, fault management
+(e.g. alarms), performance management, audits, and control mechanisms such as security
+and configuration policies.
+
+The current Danube release implements some enabling capabilities in NFVI/VIM
+such as telemetry, policy, and fault management. However, the specification of expected
+system components, behavior and the test cases to verify them have not yet
+been adequately developed. We will therefore not be testing this area at this time
+but defer to future study.
+
+7. Use case testing
+
+More complex interactions among multiple VNFs and between VNFs and the cloud
+platform can be better exercised through selected more realistic use cases.
+
+End-users see use case level testing as a significant tool in
+verifying OPNFV compliance because it validates design patterns and support
+for the types of NFVI features that users
+care about. There are a lot of projects in OPNFV developing use cases
+and sample VNFs, however most are still in early phase and require further
+enhancements to become useful additions to the CVP.
+
+Many of these use case test cases do not add new API requirements to the SUT per se, but
+exercise aspects of the SUT's functional capabilities in more complex ways.
+Other use cases, such as SDNVPN, will require additional API level extension,
+and to clearly separate the two, we will categorize the latter as
+NFV specific functional requirements and not simply as use cases.
+
+Examples such as vIMS, or those which are not yet available
+in Danube release e.g. vCPE,
+will be valuable additions to the CVP. These use cases need to
+be widely accepted, and since they are more complex, using these VNFs for CVP demands
+higher level of community resources to implement, analyze and document these VNFs.
+
+Use case testing is not ready for CVP at the time of Danube, but can be incorporated
+in Euphrates or as a future roadmap area.
+
+Finally, we take a preliminary look at future roadmap ideas that may be helpful
+for the community to plan and pull resources around.
+
+8. Future CVP scope enhancements
+
+Some possible areas of enhancement for the CVP scope in subsequent releases include:
+
+ - service assurance, as discussed above
+ - use case testing, as discussed above
+ - platform in-place upgrade
+ - API backward compatibility / micro-versioning
+ - workload migration
+ - multisite federation
+ - platform operational insights, e.g. telemetry, logging
+ - efficiency, e.g. hardware and energy footprint of the platform
+ - interoperability with workload automation platforms e.g. ONAP
+
+And enhancements may also be made to the existing test areas as well,
+particularly those with limited coverage in this release.
+
+Summary of Test Scope
+---------------------
+
+The above analysis concludes with the following scope summarized below.
+These tested areas represent significant advancement in the
+direction to meet the CVP's objectives and end-user expectations, and is a
+good basis for the initial phase of CVP.
+
+- Test Area: common cloud capabilities
+ - Openstack Refstack-compute test cases
+ Image, Identity, Compute, Network, Storage
+ - OPNFV-Functest/vPing, including both user data and ssh
+ - Port security and security groups
+ - VM lifecycle events
+ - VM networking
+ - VM resource scheduling
+ - Mandatory
+
+- Test Area: SDNVPN
+ - OPNFV-SDNVPN
+ - Optional
+
+- Test Area: IPv6
+ - OPNFV-IPv6
+ - Limited to overlay tests, v6Ping
+ - Optional
+
+- Test Area: High Availability
+ - OPNFV-HA
+ - OPNFV-Yardstick
+ - Limited to service continuity verification on control services
+ - Mandatory
+
+- Test Area: Resilency with Load
+ - OPNFV-Bottlenecks
+ - OPNFV-Yardstick
+ - Limited to compute resource load testing
+ - Optional
+
+Note 1: While the current scope of compliance includes functional verification
+of certain performance-enhancing NFVI features, no performance measurements or
+assessment of performance capabilities are included as of this release.
+
+Note 2: The SUT is limited to NFVI and VIM functions. While testing MANO
+component capabilities is out of scope, certain APIs exposed towards MANO are
+used by the current OPNFV compliance testing suite. MANO and other operational
+elements may be part of the test infrastructure; for example used for workload
+deployment and provisioning.
+
+Criteria for Awarding Compliance
+================================
+
+This section provides guidance on compliance criteia for each test area. The
+criteria described here are high-level, detailed pass/fail metrics are
+documented in Dovetail test specifications.
+
+1. All mandatory test cases must pass.
+
+Exceptions to this rule may be legitimate, e.g. due to imperfect test tools or
+reasonable circumstances that we can not foresee. These exceptions must be
+documented and accepted by the reviewers.
+
+2. Optional test cases are optional to run. Its test results, pass or fail,
+ do not impact compliance.
+
+Applicants who choose to run the optional test cases can include the results
+of the optional test cases to highlight the additional compliance.
+
+References
+==========
+
+[1] The OPNFV CVP Guidelines v.16 [Editor's note: link to be provided.]
+[2] Pharos specification xxx [Editor's note: link to be provided.]
+
diff --git a/docs/testing/user/testspecification/dynamicnetwork/index.rst b/docs/testing/user/testspecification/dynamicnetwork/index.rst
new file mode 100644
index 00000000..8fa084c7
--- /dev/null
+++ b/docs/testing/user/testspecification/dynamicnetwork/index.rst
@@ -0,0 +1,413 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) Huawei Technologies Co.,Ltd
+
+=====================================================
+Dynamic Network Runtime Operations test specification
+=====================================================
+
+.. toctree::
+ :maxdepth: 2
+
+Scope
+=====
+
+The dynamic network runtime operations test area evaluates the ability of the
+system under test to support dynamic network runtime operations through the
+life of a VNF (e.g. attach/detach, enable/disable, read stats).
+The tests in this test area will evaluate IPv4 network runtime operations
+functionality. These runtime operations includes hotpluging network interface,
+detaching floating-ip from VM, attaching floating-ip to VM, updating subnet's
+DNS, updating VM instance port admin state and updating router admin state.
+
+References
+==========
+
+- DNS
+
+ - https://docs.openstack.org/newton/networking-guide/config-dns-res.html
+
+Definitions and abbreviations
+=============================
+
+The following terms and abbreviations are used in conjunction with this test
+area
+
+- API - Application Programming Interface
+- DNS - Domain Name System
+- ICMP - Internet Control Message Protocol
+- MAC - Media Access Control
+- NIC - Network Interface Controller
+- NFVi - Network Functions Virtualization infrastructure
+- SSH - Secure Shell
+- TCP - Transmission Control Protocol
+- VIM - Virtual Infrastructure Manager
+- VM - Virtual Machine
+
+System Under Test (SUT)
+=======================
+
+The system under test is assumed to be the NFVi and VIM in operation on a
+Pharos compliant infrastructure.
+
+Test Area Structure
+===================
+
+The test area is structured based on dynamic network runtime operations. Each
+test case is able to run independently, i.e. irrelevant of the state created by
+a previous test. Specifically, every test performs clean-up operations which
+return the system to the same state as before the test.
+
+Test Descriptions
+=================
+
+API Used and Reference
+----------------------
+
+Security Groups: https://developer.openstack.org/api-ref/network/v2/index.html#security-groups-security-groups
+
+- create security group
+- delete security group
+
+Networks: https://developer.openstack.org/api-ref/networking/v2/index.html#networks
+
+- create network
+- delete network
+
+Routers and interface: https://developer.openstack.org/api-ref/networking/v2/index.html#routers-routers
+
+- create router
+- update router
+- delete router
+- add interface to router
+
+Subnets: https://developer.openstack.org/api-ref/networking/v2/index.html#subnets
+
+- create subnet
+- update subnet
+- delete subnet
+
+Servers: https://developer.openstack.org/api-ref/compute/
+
+- create keypair
+- create server
+- delete server
+- add/assign floating IP
+- disassociate floating IP
+
+Ports: https://developer.openstack.org/api-ref/networking/v2/index.html#ports
+
+- create port
+- update port
+- delete port
+
+Floating IPs: https://developer.openstack.org/api-ref/networking/v2/index.html#floating-ips-floatingips
+
+- create floating IP
+- delete floating IP
+
+--------------------------------------
+Test Case 1 - Basic network operations
+--------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops
+
+Test preconditions
+------------------
+
+* Nova has been configured to boot VMs with Neutron-managed networking
+* Openstack nova, neutron services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a security group SG1, which has rules for allowing
+ incoming SSH and ICMP traffic
+* Test action 2: Create a neutron network NET1
+* Test action 3: Create a tenant router R1 which routes traffic to public network
+* Test action 4: Create a subnet SUBNET1 and add it as router interface
+* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating
+ IP FIP1 (via R1) to VM1
+* **Test assertion 1:** Ping FIP1 and SSH to VM1 via FIP1 successfully
+* **Test assertion 2:** Ping the internal gateway from VM1 successfully
+* **Test assertion 3:** Ping the default gateway from VM1 using its floating IP
+ FIP1 successfully
+* Test action 6: Detach FIP1 from VM1
+* **Test assertion 4:** VM1 becomes unreachable after FIP1 disassociated
+* Test action 7: Create a new server VM2 with NET1, and associate floating IP FIP1 to VM2
+* **Test assertion 5:** Ping FIP1 and SSH to VM2 via FIP1 successfully
+* Test action 8: Delete SG1, NET1, SUBNET1, R1, VM1, VM2 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the functionality of basic network operations.
+Specifically, the test verifies that:
+
+* The Tempest host can ping VM's IP address. This implies, but does not
+ guarantee (see the ssh check that follows), that the VM has been assigned the
+ correct IP address and has connectivity to the Tempest host.
+
+* The Tempest host can perform key-based authentication to an ssh server hosted
+ at VM's IP address. This check guarantees that the IP address is associated
+ with the target VM.
+
+* The Tempest host can ssh into the VM via the IP address and successfully ping
+ the internal gateway address, implying connectivity to another VM on the same network.
+
+* The Tempest host can ssh into the VM via the IP address and successfully ping
+ the default gateway, implying external connectivity.
+
+* Detach the floating-ip from the VM and VM becomes unreachable.
+
+* Associate attached floating ip to a new VM and the new VM is reachable.
+
+* Floating IP status is updated correctly after each change.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+---------------------------------------
+Test Case 2 - Hotplug network interface
+---------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic
+
+Test preconditions
+------------------
+
+* Nova has been configured to boot VMs with Neutron-managed networking
+* Compute interface_attach feature is enabled
+* VM vnic_type is not set to 'direct' or 'macvtap'
+* Openstack nova, neutron services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a security group SG1, which has rules for allowing
+ incoming SSH and ICMP traffic
+* Test action 2: Create a neutron network NET1
+* Test action 3: Create a tenant router R1 which routes traffic to public network
+* Test action 4: Create a subnet SUBNET1 and add it as router interface
+* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating
+ IP FIP1 (via R1) to VM1
+* **Test assertion 1:** Ping FIP1 and SSH to VM1 with FIP1 successfully
+* Test action 6: Create a second neutron network NET2 and subnet SUBNET2, and
+ attach VM1 to NET2
+* Test action 7: Get VM1's ethernet interface NIC2 for NET2
+* **Test assertion 2:** Ping NET2's internal gateway successfully
+* Test action 8: Delete SG1, NET1, NET2, SUBNET1, SUBNET2, R1, NIC2, VM1 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the functionality of adding network to an active VM.
+Specifically, the test verifies that:
+
+* New network interface can be added to an existing VM successfully.
+
+* The Tempest host can ssh into the VM via the IP address and successfully ping
+ the new network's internal gateway address, implying connectivity to the new network.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+-------------------------------------------
+Test Case 3 - Update subnet's configuration
+-------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details
+
+Test preconditions
+------------------
+
+* Nova has been configured to boot VMs with Neutron-managed networking
+* DHCP client is available
+* Tenant networks should be non-shared and isolated
+* Openstack nova, neutron services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a security group SG1, which has rules for allowing
+ incoming SSH and ICMP traffic
+* Test action 2: Create a neutron network NET1
+* Test action 3: Create a tenant router R1 which routes traffic to public network
+* Test action 4: Create a subnet SUBNET1 and add it as router interface,
+ configure SUBNET1 with dns nameserver '1.2.3.4'
+* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating
+ IP FIP1 (via R1) to VM1
+* **Test assertion 1:** Ping FIP1 and SSH to VM1 with FIP1 successfully
+* **Test assertion 2:** Retrieve the VM1's configured dns and verify it matches
+ the one configured for SUBNET1
+* Test action 6: Update SUBNET1's dns to '9.8.7.6'
+* **Test assertion 3:** After triggering the DHCP renew from the VM manually,
+ retrieve the VM1's configured dns and verify it has been successfully updated
+* Test action 7: Delete SG1, NET1, SUBNET1, R1, VM1 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the functionality of updating subnet's configurations.
+Specifically, the test verifies that:
+
+* Updating subnet's DNS server configurations are affecting the VMs.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+----------------------------------------
+Test Case 4 - Update VM port admin state
+----------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_instance_port_admin_state
+
+Test preconditions
+------------------
+
+* Nova has been configured to boot VMs with Neutron-managed networking
+* Network port_admin_state_change feature is enabled
+* Openstack nova, neutron services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a security group SG1, which has rules for allowing
+ incoming SSH and ICMP traffic
+* Test action 2: Create a neutron network NET1
+* Test action 3: Create a tenant router R1 which routes traffic to public network
+* Test action 4: Create a subnet SUBNET1 and add it as router interface
+* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating
+ IP FIP1 (via R1) to VM1
+* Test action 6: Create a server VM2 with SG1 and NET1, and assign a floating
+ IP FIP2 to VM2
+* Test action 7: Get a SSH client SSHCLNT1 to VM2
+* **Test assertion 1:** Ping FIP1 and SSH to VM1 with FIP1 successfully
+* **Test assertion 2:** Ping FIP1 via SSHCLNT1 successfully
+* Test action 8: Update admin_state_up attribute of VM1 port to False
+* **Test assertion 3:** Ping FIP1 and SSH to VM1 with FIP1 failed
+* **Test assertion 4:** Ping FIP1 via SSHCLNT1 failed
+* Test action 9: Update admin_state_up attribute of VM1 port to True
+* **Test assertion 5:** Ping FIP1 and SSH to VM1 with FIP1 successfully
+* **Test assertion 6:** Ping FIP1 via SSHCLNT1 successfully
+* Test action 10: Delete SG1, NET1, SUBNET1, R1, SSHCLNT1, VM1, VM2 and FIP1, FIP2
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the VM public and project connectivity status by changing VM port
+admin_state_up to True & False. Specifically, the test verifies that:
+
+* Public and project connectivity is reachable before updating admin_state_up
+ attribute of VM port to False.
+
+* Public and project connectivity is unreachable after updating admin_state_up
+ attribute of VM port to False.
+
+* Public and project connectivity is reachable after updating admin_state_up
+ attribute of VM port from False to True.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+---------------------------------------
+Test Case 5 - Update router admin state
+---------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state
+
+Test preconditions
+------------------
+
+* Nova has been configured to boot VMs with Neutron-managed networking
+* Multi-tenant networks capabilities
+* Openstack nova, neutron services are available
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a security group SG1, which has rules for allowing
+ incoming SSH and ICMP traffic
+* Test action 2: Create a neutron network NET1
+* Test action 3: Create a tenant router R1 which routes traffic to public network
+* Test action 4: Create a subnet SUBNET1 and add it as router interface
+* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating
+ IP FIP1 (via R1) to VM1
+* **Test assertion 1:** Ping FIP1 and SSH to VM1 with FIP1 successfully
+* Test action 6: Update admin_state_up attribute of R1 to False
+* **Test assertion 2:** Ping FIP1 and SSH to VM1 with FIP1 failed
+* Test action 7: Update admin_state_up attribute of R1 to True
+* **Test assertion 3:** Ping FIP1 and SSH to VM1 with FIP1 successfully
+* Test action 8: Delete SG1, NET1, SUBNET1, R1, VM1 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the router public connectivity status by changing
+router admin_state_up to True & False.
+Specifically, the test verifies that:
+
+* Public connectivity is reachable before updating admin_state_up attribute of router to False.
+
+* Public connectivity is unreachable after updating admin_state_up attribute of router to False.
+
+* Public connectivity is reachable after updating admin_state_up attribute of router.
+ from False to True
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
diff --git a/docs/testing/user/testspecification/securitygroup/index.rst b/docs/testing/user/testspecification/securitygroup/index.rst
new file mode 100644
index 00000000..0621b84d
--- /dev/null
+++ b/docs/testing/user/testspecification/securitygroup/index.rst
@@ -0,0 +1,450 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) Huawei Technologies Co.,Ltd
+
+===================================================
+Security Group and Port Security test specification
+===================================================
+
+.. toctree::
+ :maxdepth: 2
+
+Scope
+=====
+
+The security group and port security test area evaluates the ability of the
+system under test to support packet filtering by security group and port security.
+The tests in this test area will evaluate preventing MAC spoofing by port security,
+basic security group operations including testing cross/in tenant traffic, testing
+multiple security groups, using port security to disable security groups and
+updating security groups.
+
+References
+==========
+
+N/A
+
+Definitions and abbreviations
+=============================
+
+The following terms and abbreviations are used in conjunction with this test
+area
+
+- API - Application Programming Interface
+- ICMP - Internet Control Message Protocol
+- MAC - Media Access Control
+- NFVi - Network Functions Virtualization infrastructure
+- SSH - Secure Shell
+- TCP - Transmission Control Protocol
+- VIM - Virtual Infrastructure Manager
+- VM - Virtual Machine
+
+System Under Test (SUT)
+=======================
+
+The system under test is assumed to be the NFVi and VIM in operation on a
+Pharos compliant infrastructure.
+
+Test Area Structure
+===================
+
+The test area is structured based on the basic operations of security group and
+port security. Each test case is able to run independently, i.e. irrelevant of
+the state created by a previous test. Specifically, every test performs clean-up
+operations which return the system to the same state as before the test.
+
+Test Descriptions
+=================
+
+API Used and Reference
+----------------------
+
+Security Groups: https://developer.openstack.org/api-ref/network/v2/index.html#security-groups-security-groups
+
+- create security group
+- delete security group
+
+Networks: https://developer.openstack.org/api-ref/networking/v2/index.html#networks
+
+- create network
+- delete network
+- list networks
+- create floating ip
+- delete floating ip
+
+Routers and interface: https://developer.openstack.org/api-ref/networking/v2/index.html#routers-routers
+
+- create router
+- delete router
+- list routers
+- add interface to router
+
+Subnets: https://developer.openstack.org/api-ref/networking/v2/index.html#subnets
+
+- create subnet
+- list subnets
+- delete subnet
+
+Servers: https://developer.openstack.org/api-ref/compute/
+
+- create keypair
+- create server
+- delete server
+- add/assign floating ip
+
+Ports: https://developer.openstack.org/api-ref/networking/v2/index.html#ports
+
+- update port
+- list ports
+- show port details
+
+--------------------------------------------
+Test Case 1 - Port Security and MAC Spoofing
+--------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_port_security_macspoofing_port
+
+Test preconditions
+------------------
+
+* Neutron port-security extension API
+* Neutron security-group extension API
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a security group SG1, which has rules for allowing incoming
+ SSH and ICMP traffic
+* Test action 2: Create a neutron network NET1
+* Test action 3: Create a tenant router R1 which routes traffic to public network
+* Test action 4: Create a subnet SUBNET1 and add it as router interface
+* Test action 5: Create a server VM1 with SG1 and NET1, and assign a floating ip
+ FIP1 (via R1) to VM1
+* Test action 6: Verify can ping FIP1 successfully and can SSH to VM1 with FIP1
+* Test action 7: Create a second neutron network NET2 and subnet SUBNET2, and attach VM1 to NET2
+* Test action 8: Get VM1's ethernet interface NIC2 for NET2
+* Test action 9: Create second server VM2 on NET2
+* Test action 10: Verify VM1 is able to communicate with VM2 via NIC2
+* Test action 11: Login to VM1 and spoof the MAC address of NIC2 to "00:00:00:00:00:01"
+* Test action 12: Verify VM1 fails to communicate with VM2 via NIC2
+* **Test assertion 1:** The ping operation is failed
+* Test action 13: Update 'security_groups' to be none for VM1's NIC2 port
+* Test action 14: Update 'port_security_enable' to be False for VM1's NIC2 port
+* Test action 15: Verify now VM1 is able to communicate with VM2 via NIC2
+* **Test assertion 2:** The ping operation is successful
+* Test action 16: Delete SG1, NET1, NET2, SUBNET1, SUBNET2, R1, VM1, VM2 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability to prevent MAC spoofing by using port security.
+Specifically, the test verifies that:
+
+* With port security, the ICMP packets from a spoof server cannot pass the port.
+
+* Without port security, the ICMP packets from a spoof server can pass the port.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+------------------------------------------------------
+Test Case 2 - Test Security Group Cross Tenant Traffic
+------------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic
+
+Test preconditions
+------------------
+
+* Neutron security-group extension API
+* Two tenants
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a neutron network NET1 for primary tenant
+* Test action 2: Create a primary tenant router R1 which routes traffic to public network
+* Test action 3: Create a subnet SUBNET1 and add it as router interface
+* Test action 4: Create 2 empty security groups SG1 and SG2 for primary tenant
+* Test action 5: Add a tcp rule to SG1
+* Test action 6: Create a server VM1 with SG1, SG2 and NET1, and assign a floating ip
+ FIP1 (via R1) to VM1
+* Test action 7: Repeat test action 1 to 6 and create NET2, R2, SUBNET2, SG3, SG4,
+ FIP2 and VM2 for an alt_tenant
+* Test action 8: Verify VM1 fails to communicate with VM2 through FIP2
+* **Test assertion 1:** The ping operation is failed
+* Test action 9: Add ICMP rule to SG4
+* Test action 10: Verify VM1 is able to communicate with VM2 through FIP2
+* **Test assertion 2:** The ping operation is successful
+* Test action 11: Verify VM2 fails to communicate with VM1 through FIP1
+* **Test assertion 3:** The ping operation is failed
+* Test action 12: Add ICMP rule to SG2
+* Test action 13: Verify VM2 is able to communicate with VM1 through FIP1
+* **Test assertion 4:** The ping operation is successful
+* Test action 14: Delete SG1, SG2, SG3, SG4, NET1, NET2, SUBNET1, SUBNET2, R1, R2,
+ VM1, VM2, FIP1 and FIP2
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability of the security group to filter packets cross tenant.
+Specifically, the test verifies that:
+
+* Without ICMP security group rule, the ICMP packets cannot be received by the server
+ in another tenant which differs from the source server.
+
+* With ingress ICMP security group rule enabled only at tenant1, the server in tenant2
+ can ping server in tenant1 but not the reverse direction.
+
+* With ingress ICMP security group rule enabled at tenant2 also, the ping works from both directions.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+---------------------------------------------------
+Test Case 3 - Test Security Group in Tenant Traffic
+---------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_in_tenant_traffic
+
+Test preconditions
+------------------
+
+* Neutron security-group extension API
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a neutron network NET1
+* Test action 2: Create a tenant router R1 which routes traffic to public network
+* Test action 3: Create a subnet SUBNET1 and add it as router interface
+* Test action 4: Create 2 empty security groups SG1 and SG2
+* Test action 5: Add a tcp rule to SG1
+* Test action 6: Create a server VM1 with SG1, SG2 and NET1, and assign a floating ip
+ FIP1 (via R1) to VM1
+* Test action 7: Create second server VM2 with default security group and NET1
+* Test action 8: Verify VM1 fails to communicate with VM2 through VM2's fixed ip
+* **Test assertion 1:** The ping operation is failed
+* Test action 9: Add ICMP security group rule to default security group
+* Test action 10: Verify VM1 is able to communicate with VM2 through VM2's fixed ip
+* **Test assertion 2:** The ping operation is successful
+* Test action 11: Delete SG1, SG2, NET1, SUBNET1, R1, VM1, VM2 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability of the security group to filter packets in one tenant.
+Specifically, the test verifies that:
+
+* Without ICMP security group rule, the ICMP packets cannot be received by the server
+ in the same tenant.
+
+* With ICMP security group rule, the ICMP packets can be received by the server
+ in the same tenant.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+-------------------------------------------
+Test Case 4 - Test Multiple Security Groups
+-------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_multiple_security_groups
+
+Test preconditions
+------------------
+
+* Neutron security-group extension API
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a neutron network NET1
+* Test action 2: Create a tenant router R1 which routes traffic to public network
+* Test action 3: Create a subnet SUBNET1 and add it as router interface
+* Test action 4: Create 2 empty security groups SG1 and SG2
+* Test action 5: Add a tcp rule to SG1
+* Test action 6: Create a server VM1 with SG1, SG2 and NET1, and assign a floating ip
+ FIP1 (via R1) to VM1
+* Test action 7: Verify failed to ping FIP1
+* **Test assertion 1:** The ping operation is failed
+* Test action 8: Add ICMP security group rule to SG2
+* Test action 9: Verify can ping FIP1 successfully
+* **Test assertion 2:** The ping operation is successful
+* Test action 10: Verify can SSH to VM1 with FIP1
+* **Test assertion 3:** Can SSH to VM1 successfully
+* Test action 11: Delete SG1, SG2, NET1, SUBNET1, R1, VM1 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability of multiple security groups to filter packets.
+Specifically, the test verifies that:
+
+* A server with 2 security groups, one with TCP rule and without ICMP rule,
+ cannot receive the ICMP packets sending from the tempest host machine.
+
+* A server with 2 security groups, one with TCP rule and the other with ICMP rule,
+ can receive the ICMP packets sending from the tempest host machine and be connected
+ via the SSH client.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+-------------------------------------------------------
+Test Case 5 - Test Port Security Disable Security Group
+-------------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_port_security_disable_security_group
+
+Test preconditions
+------------------
+
+* Neutron security-group extension API
+* Neutron port-security extension API
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a neutron network NET1
+* Test action 2: Create a tenant router R1 which routes traffic to public network
+* Test action 3: Create a subnet SUBNET1 and add it as router interface
+* Test action 4: Create 2 empty security groups SG1 and SG2
+* Test action 5: Add a tcp rule to SG1
+* Test action 6: Create a server VM1 with SG1, SG2 and NET1, and assign a floating ip
+ FIP1 (via R1) to VM1
+* Test action 7: Create second server VM2 with default security group and NET1
+* Test action 8: Update 'security_groups' to be none and 'port_security_enabled' to be
+ True for VM2's port
+* Test action 9: Verify VM1 fails to communicate with VM2 through VM2's fixed ip
+* **Test assertion 1:** The ping operation is failed
+* Test action 10: Update 'security_groups' to be none and 'port_security_enabled' to be
+ False for VM2's port
+* Test action 11: Verify VM1 is able to communicate with VM2 through VM2's fixed ip
+* **Test assertion 2:** The ping operation is successful
+* Test action 12: Delete SG1, SG2, NET1, SUBNET1, R1, VM1, VM2 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability of port security to disable security group.
+Specifically, the test verifies that:
+
+* The ICMP packets cannot pass the port whose 'port_security_enabled' is True
+ and security_groups is none.
+
+* The ICMP packets can pass the port whose 'port_security_enabled' is False
+ and security_groups is none.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+---------------------------------------------
+Test Case 6 - Test Update Port Security Group
+---------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_port_update_new_security_group
+
+Test preconditions
+------------------
+
+* Neutron security-group extension API
+* One public network
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a neutron network NET1
+* Test action 2: Create a tenant router R1 which routes traffic to public network
+* Test action 3: Create a subnet SUBNET1 and add it as router interface
+* Test action 4: Create 2 empty security groups SG1 and SG2
+* Test action 5: Add a tcp rule to SG1
+* Test action 6: Create a server VM1 with SG1, SG2 and NET1, and assign a floating ip
+ FIP1 (via R1) to VM1
+* Test action 7: Create third empty security group SG3
+* Test action 8: Add ICMP rule to SG3
+* Test action 9: Create second server VM2 with default security group and NET1
+* Test action 10: Verify VM1 fails to communicate with VM2 through VM2's fixed ip
+* **Test assertion 1:** The ping operation is failed
+* Test action 11: Update 'security_groups' to be SG3 for VM2's port
+* Test action 12: Verify VM1 is able to communicate with VM2 through VM2's fixed ip
+* **Test assertion 2:** The ping operation is successful
+* Test action 13: Delete SG1, SG2, SG3, NET1, SUBNET1, R1, VM1, VM2 and FIP1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the ability to update port with a new security group.
+Specifically, the test verifies that:
+
+* Without ICMP security group rule, the VM cannot receive ICMP packets.
+
+* Update the port's security group which has ICMP rule, the VM can receive ICMP packets.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
diff --git a/docs/testing/user/testspecification/vimoperationscompute/index.rst b/docs/testing/user/testspecification/vimoperationscompute/index.rst
index 4ed37809..42a8e70e 100644
--- a/docs/testing/user/testspecification/vimoperationscompute/index.rst
+++ b/docs/testing/user/testspecification/vimoperationscompute/index.rst
@@ -1,6 +1,6 @@
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
-.. (c) Ericsson AB
+.. (c) Ericsson AB, Huawei Technologies Co.,Ltd
=========================================
VIM compute operations test specification
@@ -12,47 +12,375 @@ VIM compute operations test specification
Scope
=====
+The VIM compute operations test area evaluates the ability of the system under
+test to support VIM compute operations. The test cases documented here are the
+compute API test cases in the OpenStack Interop guideline 2016.8 as implemented
+by the RefStack client. These test cases will evaluate basic OpenStack (as a VIM)
+compute operations, including:
+
+- Image management operations
+- Basic support operations
+- API version support operations
+- Quotas management operations
+- Basic server operations
+- Volume management operations
+
References
================
+- OpenStack Governance/Interop
+
+ - https://wiki.openstack.org/wiki/Governance/InteropWG
+
+- Defcore test cases
+
+ - https://github.com/openstack/interop/blob/master/2016.08.json
+
+- Refstack client
+
+ - https://github.com/openstack/refstack-client
Definitions and abbreviations
=============================
+The following terms and abbreviations are used in conjunction with this test area
-Use case description
-====================
-
+- API - Application Programming Interface
+- NFVi - Network Functions Virtualization infrastructure
+- SUT - System Under Test
+- UUID - Universally Unique Identifier
+- VIM - Virtual Infrastructure Manager
+- VM - Virtual Machine
System Under Test (SUT)
=======================
+The system under test is assumed to be the NFVi and VIM deployed with a Pharos compliant infrastructure.
-Test Suite Structure
+Test Area Structure
====================
+The test area is structured based on VIM compute API operations. Each test case is
+able to run independently, i.e. irrelevant of the state created by a previous test.
+Specifically, every test performs clean-up operations which return the system to
+the same state as before the test.
+
+For brevity, the test cases in this test area are summarized together based on
+the operations they are testing.
Test Descriptions
=================
+API Used and Reference
+----------------------
+
+Servers: https://developer.openstack.org/api-ref/compute/
+
+- create server
+- delete server
+- list servers
+- start server
+- stop server
+- update server
+- get server action
+- set server metadata
+- update server metadata
+- rebuild server
+
+- create image
+- delete image
+
+- create keypair
+- delete keypair
+
+Block storage: https://developer.openstack.org/api-ref/block-storage
+
+- create volume
+- delete volume
+- attach volume to server
+- detach volume from server
+
+-----------------------------------------------------
+Test Case 1 - Image operations within the Compute API
+-----------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image
+tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_image_specify_multibyte_character_image_name
+
+Test preconditions
+------------------
+
+* Compute server extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a server VM1 with an image IMG1 and wait for VM1 to reach 'ACTIVE' status
+* Test action 2: Create a new server image IMG2 from VM1, specifying image name
+ and image metadata. Wait for IMG2 to reach 'ACTIVE' status, and then delete IMG2
+* **Test assertion 1:** Verify IMG2 is created with correct image name and image
+ metadata; verify IMG1's 'minRam' equals to IMG2's 'minRam' and IMG2's 'minDisk' equals
+ to IMG1's 'minDisk' or VM1's flavor disk size
+* **Test assertion 2:** Verify IMG2 is deleted correctly
+* Test action 3: Create another server IMG3 from VM1, specifying image name
+ with a 3 byte utf-8 character
+* **Test assertion 3:** Verify IMG3 is created correctly
+* Test action 4: Delete VM1, IMG1 and IMG3
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the Compute API ability of creating image from server,
+deleting image, creating server image with multi-byte character name.
+Specifically, the test verifies that:
+
+* Compute server create image and delete image APIs work correctly.
+* Compute server image can be created with multi-byte character name.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+-----------------------------------------------------
+Test Case 2 - Action operation within the Compute API
+-----------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_get_instance_action
+tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_list_instance_actions
+
+Test preconditions
+------------------
+
+* Compute server extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a server VM1 and wait for VM1 to reach 'ACTIVE' status
+* Test action 2: Get the action details ACT_DTL of VM1
+* **Test assertion 1:** Verify ACT_DTL's 'instance_uuid' matches VM1's ID and
+ ACT_DTL's 'action' matched 'create'
+* Test action 3: Create a server VM2 and wait for VM2 to reach 'ACTIVE' status
+* Test action 4: Delete server VM2 and wait for VM2 to reach termination
+* Test action 5: Get the action list ACT_LST of VM2
+* **Test assertion 2:** Verify ACT_LST's length is 2 and two actions are 'create' and 'delete'
+* Test action 6: Delete VM1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the Compute API ability of getting the action details
+of a provided server and getting the action list of a deleted server.
+Specifically, the test verifies that:
+
+* Get the details of the action in a specified server.
+* List the actions that were performed on the specified server.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+--------------------------------------------------------------------------
+Test Case 3 - Generate, import and delete SSH keys within Compute services
+--------------------------------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_specify_keypair
+
+Test preconditions
+------------------
+
+* Compute server extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a keypair KEYP1 and list all existing keypairs
+* Test action 2: Create a server VM1 with KEYP1 and wait for VM1 to reach 'ACTIVE' status
+* Test action 3: Show details of VM1
+* **Test assertion 1:** Verify value of 'key_name' in the details equals to the name of KEYP1
+* Test action 4: Delete KEYP1 and VM1
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the Compute API ability of creating a keypair, listing
+keypairs and creating a server with a provided keypair.
+Specifically, the test verifies that:
+
+* Compute create keypair and list keypair APIs work correctly.
+* While creating a server, keypair can be specified.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+--------------------------------------------------------
+Test Case 4 - List supported versions of the Compute API
+--------------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.compute.test_versions.TestVersions.test_list_api_versions
+
+Test preconditions
+------------------
+
+* Compute versions extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Get a List of versioned endpoints in the SUT
+* **Test assertion 1:** Verify endpoints versions start at 'v2.0'
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the functionality of listing all available APIs to API consumers.
+Specifically, the test verifies that:
+
+* Compute list API versions API works correctly.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+----------------------------------------------
+Test Case 5 - Quotas management in Compute API
+----------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.compute.test_quotas.QuotasTestJSON.test_get_default_quotas
+tempest.api.compute.test_quotas.QuotasTestJSON.test_get_quotas
+
+Test preconditions
+------------------
+
+* Compute quotas extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+* Test action 1: Get the default quota set using the tenant ID
+* **Test assertion 1:** Verify the default quota set ID matches tenant ID and
+ the default quota set is complete
+* Test action 2: Get the quota set using the tenant ID
+* **Test assertion 2:** Verify the quota set ID matches tenant ID and the quota
+ set is complete
+* Test action 3: Get the quota set using the user ID
+* **Test assertion 3:** Verify the quota set ID matches tenant ID and the quota
+ set is complete
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the functionality of getting quota set.
+Specifically, the test verifies that:
+
+* User can get the default quota set for its tenant.
+* User can get the quota set for its tenant.
+* User can get the quota set using user ID.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+--------------------------------------------------------
+Test Case 6 - Basic server operations in the Compute API
+--------------------------------------------------------
+
+Test case specification
+-----------------------
+
+This test case evaluates the Compute API ability of basic server operations, including:
+
+- Create a server with admin password
+- Create a server with a name that already exists
+- Create a server with a numeric name
+- Create a server with a really long metadata
+- Create a server with a name whose length exceeding 255 characters
+- Create a server with an unknown flavor
+- Create a server with an unknown image ID
+- Create a server with an invalid network UUID
+- Delete a server using a server ID that exceeds length limit
+- Delete a server using a negative server ID
+- Get a nonexistent server details
+- Verify the instance host name is the same as the server name
+- Create a server with an invalid access IPv6 address
+- List all existent servers
+- Filter the (detailed) list of servers by flavor, image, server name, server status or limit
+- Lock a server and try server stop, unlock and retry
+- Get and delete metadata from a server
+- List and set metadata for a server
+- Reboot, rebuild, stop and start a server
+- Update a server's access addresses and server name
+
+The reference is,
+
+tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_server_with_admin_password
+tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_with_existing_server_name
+tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_numeric_server_name
+tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_server_metadata_exceeds_length_limit
+tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_server_name_length_exceeds_256
+tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_flavor
+tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_image
+tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_network_uuid
+tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_delete_server_pass_id_exceeding_length_limit
+tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_delete_server_pass_negative_id
+tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_get_non_existent_server
tempest.api.compute.servers.test_create_server.ServersTestJSON.test_host_name_is_same_as_server_name
+tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_host_name_is_same_as_server_name
+tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_invalid_ip_v6_address
tempest.api.compute.servers.test_create_server.ServersTestJSON.test_list_servers
tempest.api.compute.servers.test_create_server.ServersTestJSON.test_list_servers_with_detail
-tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_created_server_vcpus
-tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_server_details
-tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_host_name_is_same_as_server_name
tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_list_servers
tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_list_servers_with_detail
-tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_created_server_vcpus
-tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_server_details
-tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_get_instance_action
-tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_list_instance_actions
tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_flavor
tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_image
tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_server_name
tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_server_status
tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_limit_results
-tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_active_status
tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_flavor
tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_image
tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_limit
@@ -72,41 +400,302 @@ tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJS
tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_status_non_existing
tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_with_a_deleted_server
tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_lock_unlock_server
-tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_reboot_server_hard
-tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_rebuild_server
-tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_stop_start_server
tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_delete_server_metadata_item
tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_get_server_metadata_item
tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_list_server_metadata
tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_set_server_metadata
tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_set_server_metadata_item
tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_update_server_metadata
-tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_server_with_admin_password
-tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_specify_keypair
-tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_with_existing_server_name
-tempest.api.compute.servers.test_servers.ServersTestJSON.test_update_access_server_address
-tempest.api.compute.servers.test_servers.ServersTestJSON.test_update_server_name
-tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_numeric_server_name
-tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_server_metadata_exceeds_length_limit
-tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_server_name_length_exceeds_256
-tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_flavor
-tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_image
-tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_network_uuid
-tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_delete_server_pass_id_exceeding_length_limit
-tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_delete_server_pass_negative_id
-tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_get_non_existent_server
-tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_invalid_ip_v6_address
+tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_server_name_blank
+tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_reboot_server_hard
tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_non_existent_server
+tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_rebuild_server
tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_deleted_server
tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_non_existent_server
-tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_reboot_deleted_server
-tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_server_name_blank
+tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_stop_start_server
tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_stop_non_existent_server
+tempest.api.compute.servers.test_servers.ServersTestJSON.test_update_access_server_address
+tempest.api.compute.servers.test_servers.ServersTestJSON.test_update_server_name
tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_update_name_of_non_existent_server
tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_update_server_name_length_exceeds_256
tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_update_server_set_empty_name
-tempest.api.compute.test_quotas.QuotasTestJSON.test_get_default_quotas
-tempest.api.compute.test_quotas.QuotasTestJSON.test_get_quotas
-tempest.api.compute.test_versions.TestVersions.test_list_api_versions
+tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_created_server_vcpus
+tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_server_details
+tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_created_server_vcpus
+tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_server_details
+
+tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_active_status
+tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_reboot_deleted_server
+
+Note: the last 2 test cases are the alias of another 2 test cases respectively, which are
+
+tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_server_status
+tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_deleted_server
+
+Alias should always be included so that the test run will be tempest version agnostic,
+which can be used to test different version of OpenStack.
+
+Test preconditions
+------------------
+
+* Compute quotas extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a server VM1 with a admin password 'testpassword'
+* **Test assertion 1:** Verify the password returned in the response equals to 'testpassword'
+* Test action 2: Generate a VM name VM_NAME
+* Test action 3: Create 2 servers VM2 and VM3 both with name VM_NAME
+* **Test assertion 2:** Verify VM2's ID is not equal to VM3's ID, and VM2's name equal to VM3's name
+* Test action 4: Create a server VM4 with a numeric name '12345'
+* **Test assertion 3:** Verify creating VM4 failed
+* Test action 5: Create a server VM5 with a long metadata '{'a': 'b' * 260}'
+* **Test assertion 4:** Verify creating VM5 failed
+* Test action 6: Create a server VM6 with name length exceeding 255 characters
+* **Test assertion 5:** Verify creating VM6 failed
+* Test action 7: Create a server VM7 with an unknown flavor '-1'
+* **Test assertion 6:** Verify creating VM7 failed
+* Test action 8: Create a server VM8 with an unknown image ID '-1'
+* **Test assertion 7:** Verify creating VM8 failed
+* Test action 9: Create a server VM9 with an invalid network UUID 'a-b-c-d-e-f-g-h-i-j'
+* **Test assertion 8:** Verify creating VM9 failed
+* Test action 10: Delete a server using a server ID that exceeds system's max integer limit
+* **Test assertion 9:** Verify deleting server failed
+* Test action 11: Delete a server using a server ID '-1'
+* **Test assertion 10:** Verify deleting server failed
+* Test action 12: Get a nonexistent server by using a random generated server ID
+* **Test assertion 11:** Verify get server failed
+* Test action 13: SSH into a provided server and get server's hostname
+* **Test assertion 12:** Verify server's host name is the same as the server name
+* Test action 14: SSH into a provided server and get server's hostname (manual disk configuration)
+* **Test assertion 13:** Verify server's host name is the same as the server name (manual disk configuration)
+* Test action 15: Create a server with an invalid access IPv6 address
+* **Test assertion 14:** Verify creating server failed, a bad request error is returned in response
+* Test action 16: List all existent servers
+* **Test assertion 15:** Verify a provided server is in the server list
+* Test action 17: List all existent servers in detail
+* **Test assertion 16:** Verify a provided server is in the detailed server list
+* Test action 18: List all existent servers (manual disk configuration)
+* **Test assertion 17:** Verify a provided server is in the server list (manual disk configuration)
+* Test action 19: List all existent servers in detail (manual disk configuration)
+* **Test assertion 18:** Verify a provided server is in the detailed server list (manual disk configuration)
+* Test action 20: List all existent servers in detail and filter the server list by flavor
+* **Test assertion 19:** Verify the filtered server list is correct
+* Test action 21: List all existent servers in detail and filter the server list by image
+* **Test assertion 20:** Verify the filtered server list is correct
+* Test action 22: List all existent servers in detail and filter the server list by server name
+* **Test assertion 21:** Verify the filtered server list is correct
+* Test action 23: List all existent servers in detail and filter the server list by server status
+* **Test assertion 22:** Verify the filtered server list is correct
+* Test action 24: List all existent servers in detail and filter the server list by display limit '1'
+* **Test assertion 23:** Verify the length of filtered server list is 1
+* Test action 25: List all existent servers and filter the server list by flavor
+* **Test assertion 24:** Verify the filtered server list is correct
+* Test action 26: List all existent servers and filter the server list by image
+* **Test assertion 25:** Verify the filtered server list is correct
+* Test action 27: List all existent servers and filter the server list by display limit '1'
+* **Test assertion 26:** Verify the length of filtered server list is 1
+* Test action 28: List all existent servers and filter the server list by server name
+* **Test assertion 27:** Verify the filtered server list is correct
+* Test action 29: List all existent servers and filter the server list by server status
+* **Test assertion 28:** Verify the filtered server list is correct
+* Test action 30: List all existent servers and filter the server list by server name wildcard
+* **Test assertion 29:** Verify the filtered server list is correct
+* Test action 31: List all existent servers and filter the server list by part of server name
+* **Test assertion 30:** Verify the filtered server list is correct
+* Test action 32: List all existent servers and filter the server list by a future change-since date
+* **Test assertion 31:** Verify the filtered server list is empty
+* Test action 33: List all existent servers and filter the server list by a invalid change-since date format
+* **Test assertion 32:** Verify a bad request error is returned in the response
+* Test action 34: List all existent servers and filter the server list by display limit '1'
+* **Test assertion 33:** Verify the length of filtered server list is 1
+* Test action 35: List all existent servers and filter the server list by a
+ display limit value greater than the length of the server list
+* **Test assertion 34:** Verify the length of filtered server list equals to the length of server list
+* Test action 36: List all existent servers and filter the server list by display limit '-1'
+* **Test assertion 35:** Verify a bad request error is returned in the response
+* Test action 37: List all existent servers and filter the server list by a string type limit value 'testing'
+* **Test assertion 36:** Verify a bad request error is returned in the response
+* Test action 38: List all existent servers and filter the server list by a nonexistent flavor
+* **Test assertion 37:** Verify the filtered server list is empty
+* Test action 39: List all existent servers and filter the server list by a nonexistent image
+* **Test assertion 38:** Verify the filtered server list is empty
+* Test action 40: List all existent servers and filter the server list by a nonexistent server name
+* **Test assertion 39:** Verify the filtered server list is empty
+* Test action 41: List all existent servers in detail and search the server list for a deleted server
+* **Test assertion 40:** Verify the deleted server is not in the server list
+* Test action 42: List all existent servers and filter the server list by a nonexistent server status
+* **Test assertion 41:** Verify the filtered server list is empty
+* Test action 43: List all existent servers in detail
+* **Test assertion 42:** Verify a provided deleted server's id is not in the server list
+* Test action 44: Lock a provided server VM10 and retrieve the server's status
+* **Test assertion 43:** Verify VM10 is in 'ACTIVE' status
+* Test action 45: Stop VM10
+* **Test assertion 44:** Verify stop VM10 failed
+* Test action 46: Unlock VM10 and stop VM10 again
+* **Test assertion 45:** Verify VM10 is stopped and in 'SHUTOFF' status
+* Test action 47: Start VM10
+* **Test assertion 46:** Verify VM10 is in 'ACTIVE' status
+* Test action 48: Delete metadata item 'key1' from a provided server
+* **Test assertion 47:** Verify the metadata item is removed
+* Test action 49: Get metadata item 'key2' from a provided server
+* **Test assertion 48:** Verify the metadata item is correct
+* Test action 50: List all metadata key/value pair for a provided server
+* **Test assertion 49:** Verify all metadata are retrieved correctly
+* Test action 51: Set metadata {'meta2': 'data2', 'meta3': 'data3'} for a provided server
+* **Test assertion 50:** Verify server's metadata are replaced correctly
+* Test action 52: Set metadata item nova's value to 'alt' for a provided server
+* **Test assertion 51:** Verify server's metadata are set correctly
+* Test action 53: Update metadata {'key1': 'alt1', 'key3': 'value3'} for a provided server
+* **Test assertion 52:** Verify server's metadata are updated correctly
+* Test action 54: Create a server with empty name parameter
+* **Test assertion 53:** Verify create server failed
+* Test action 55: Hard reboot a provided server
+* **Test assertion 54:** Verify server is rebooted successfully
+* Test action 56: Soft reboot a nonexistent server
+* **Test assertion 55:** Verify reboot failed, an error is returned in the response
+* Test action 57: Rebuild a provided server with new image, new server name and metadata
+* **Test assertion 56:** Verify server is rebuilt successfully, server image, name and metadata are correct
+* Test action 58: Create a server VM11
+* Test action 59: Delete VM11 and wait for VM11 to reach termination
+* Test action 60: Rebuild VM11 with another image
+* **Test assertion 57:** Verify rebuild server failed, an error is returned in the response
+* Test action 61: Rebuild a nonexistent server
+* **Test assertion 58:** Verify rebuild server failed, an error is returned in the response
+* Test action 62: Stop a provided server
+* **Test assertion 59:** Verify server reaches 'SHUTOFF' status
+* Test action 63: Start the stopped server
+* **Test assertion 60:** Verify server reaches 'ACTIVE' status
+* Test action 64: Stop a provided server
+* **Test assertion 61:** Verify stop server failed, an error is returned in the response
+* Test action 65: Create a server VM12 and wait it to reach 'ACTIVE' status
+* Test action 66: Update VM12's IPv4 and IPv6 access addresses
+* **Test assertion 62:** Verify VM12's access addresses have been updated correctly
+* Test action 67: Create a server VM13 and wait it to reach 'ACTIVE' status
+* Test action 68: Update VM13's server name with non-ASCII characters '\u00CD\u00F1st\u00E1\u00F1c\u00E9'
+* **Test assertion 63:** Verify VM13's server name has been updated correctly
+* Test action 69: Update the server name of a nonexistent server
+* **Test assertion 64:** Verify update server name failed, an 'object not found' error is returned in the response
+* Test action 70: Update a provided server's name with a 256-character long name
+* **Test assertion 65:** Verify update server name failed, a bad request is returned in the response
+* Test action 71: Update a provided server's server name with an empty string
+* **Test assertion 66:** Verify update server name failed, a bad request error is returned in the response
+* Test action 72: Get the number of vcpus of a provided server
+* Test action 73: Get the number of vcpus stated by the server's flavor
+* **Test assertion 67:** Verify that the number of vcpus reported by the server
+ matches the amount stated by the server's flavor
+* Test action 74: Create a server VM14
+* **Test assertion 68:** Verify VM14's server attributes are set correctly
+* Test action 75: Get the number of vcpus of a provided server (manual disk configuration)
+* Test action 76: Get the number of vcpus stated by the server's flavor (manual disk configuration)
+* **Test assertion 69:** Verify that the number of vcpus reported by the server
+ matches the amount stated by the server's flavor (manual disk configuration)
+* Test action 77: Create a server VM15 (manual disk configuration)
+* **Test assertion 70:** Verify VM15's server attributes are set correctly (manual disk configuration)
+* Test action 78: Delete all VMs created
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the functionality of basic server operations.
+Specifically, the test verifies that:
+
+* If an admin password is provided on server creation, the server's root password should be set to that password
+* Create a server with a name that already exists is allowed
+* Create a server with a numeric name or a name that exceeds the length limit is not allowed
+* Create a server with a metadata that exceeds the length limit is not allowed
+* Create a server with an invalid flavor, an invalid image or an invalid network UUID is not allowed
+* Delete a server with a server ID that exceeds the length limit or a nonexistent server ID is not allowed
+* A provided server's host name is the same as the server name
+* Create a server with an invalid IPv6 access address is not allowed
+* A created server is in the (detailed) list of servers
+* Filter the (detailed) list of servers by flavor, image, server name, server status,
+ and display limit, respectively.
+* Filter the list of servers by a future date
+* Filter the list of servers by an invalid date format, a negative display limit or a string type
+ display limit value is not allowed
+* Filter the list of servers by a nonexistent flavor, image, server name or server status is not allowed
+* Deleted servers are not in the list of servers
+* Deleted servers do not show by default in list of servers
+* Locked server is not allowed to be stopped by non-admin user
+* Can get and delete metadata from servers
+* Can list, set and update server metadata
+* Create a server with name parameter empty is not allowed
+* Hard reboot a server and the server should be power cycled
+* Reboot, rebuild and stop a nonexistent server is not allowed
+* Rebuild a server using the provided image and metadata
+* Stop and restart a server
+* A server's name and access addresses can be updated
+* Update the name of a nonexistent server is not allowed
+* Update name of a server to a name that exceeds the name length limit is not allowed
+* Update name of a server to an empty string is not allowed
+* The number of vcpus reported by the server matches the amount stated by the server's flavor
+* The specified server attributes are set correctly
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+-----------------------------------------------------------------
+Test Case 7 - Retrieve volume information through the Compute API
+-----------------------------------------------------------------
+
+Test case specification
+-----------------------
+
+This test case evaluates the Compute API ability of attaching volume to a
+specific server and retrieve volume information, the reference is,
+
tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_attach_detach_volume
tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_list_get_volume_attachments
+
+Test preconditions
+------------------
+
+* Compute volume extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create a server VM1 and a volume VOL1
+* Test action 2: Attach VOL1 to VM1
+* **Test assertion 1:** Stop VM1 successfully and wait VM1 to reach 'SHUTOFF' status
+* **Test assertion 2:** Start VM1 successfully and wait VM1 to reach 'ACTIVE' status
+* **Test assertion 3:** SSH into VM1 and verify VOL1 is in VM1's root disk devices
+* Test action 3: Detach VOL1 from VM1
+* **Test assertion 4:** Stop VM1 successfully and wait VM1 to reach 'SHUTOFF' status
+* **Test assertion 5:** Start VM1 successfully and wait VM1 to reach 'ACTIVE' status
+* **Test assertion 6:** SSH into VM1 and verify VOL1 is not in VM1's root disk devices
+* Test action 4: Create a server VM2 and a volume VOL2
+* Test action 5: Attach VOL2 to VM2
+* Test action 6: List VM2's volume attachments
+* **Test assertion 7:** Verify the length of the list is 1 and VOL2 attachment is in the list
+* Test action 7: Retrieve VM2's volume information
+* **Test assertion 8:** Verify volume information is correct
+* Test action 8: Delete VM1, VM2, VOL1 and VOL2
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the functionality of retrieving volume information.
+Specifically, the test verifies that:
+
+* Stop and start a server with an attached volume work correctly.
+* Retrieve a server's volume information correctly.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
diff --git a/docs/testing/user/testspecification/vimoperationsvolume/index.rst b/docs/testing/user/testspecification/vimoperationsvolume/index.rst
index ce039a5b..9a76c37c 100644
--- a/docs/testing/user/testspecification/vimoperationsvolume/index.rst
+++ b/docs/testing/user/testspecification/vimoperationsvolume/index.rst
@@ -1,36 +1,512 @@
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
-.. (c) Ericsson AB
+.. (c) Ericsson AB, Huawei Technologies Co.,Ltd
-========================================
+=========================================
VIM volume operations test specification
-========================================
+=========================================
.. toctree::
:maxdepth: 2
-Each test case requires documentation according to:
-* Use case specification
-* Test preconditions
-* Basic test flow execution descriptor
-* Post conditions and pass fail criteria
+Scope
+=====
+
+The VIM volume operations test area evaluates the ability of the system under
+test to support VIM volume operations. The test cases documented here are the
+volume API test cases in the OpenStack Interop guideline 2016.8 as implemented
+by the RefStack client. These test cases will evaluate basic OpenStack (as a VIM)
+volume operations, including:
+
+- Volume attach and detach operations
+- Volume service availability zone operations
+- Volume cloning operations
+- Image copy-to-volume operations
+- Volume creation and deletion operations
+- Volume service extension listing
+- Volume metadata operations
+- Volume snapshot operations
+
+References
+================
+
+- OpenStack Governance/Interop
+
+ - https://wiki.openstack.org/wiki/Governance/InteropWG
+
+- Defcore test cases
+
+ - https://github.com/openstack/interop/blob/master/2016.08.json
+
+- Refstack client
+
+ - https://github.com/openstack/refstack-client
+
+Definitions and abbreviations
+=============================
+
+The following terms and abbreviations are used in conjunction with this test area
+
+- API - Application Programming Interface
+- NFVi - Network Functions Virtualization infrastructure
+- SUT - System Under Test
+- VIM - Virtual Infrastructure Manager
+- VM - Virtual Machine
+
+System Under Test (SUT)
+=======================
+
+The system under test is assumed to be the NFVI and VIM deployed with a Pharos compliant infrastructure.
+
+Test Area Structure
+====================
+
+The test area is structured based on VIM volume API operations. Each test case is
+able to run independently, i.e. irrelevant of the state created by a previous test.
+Specifically, every test performs clean-up operations which return the system to
+the same state as before the test.
+
+For brevity, the test cases in this test area are summarized together based on
+the operations they are testing.
+
+Test Descriptions
+=================
+
+API Used and Reference
+----------------------
+
+Block storage: https://developer.openstack.org/api-ref/block-storage
+
+- create volume
+- delete volume
+- update volume
+- attach volume to server
+- detach volume from server
+- create volume metadata
+- update volume metadata
+- delete volume metadata
+- list volume
+
+- create snapshot
+- update snapshot
+- delete snapshot
+
+------------------------------------------------------------------------
+Test Case 1 - Volume attach and detach operations with the Cinder v2 API
+------------------------------------------------------------------------
+
+Test case specification
+-----------------------
-tempest.api.volume.test_availability_zone.AvailabilityZoneV2TestJSON.test_get_availability_zone_list
-tempest.api.volume.test_extensions.ExtensionsV2TestJSON.test_list_extensions
-tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_create_get_delete_snapshot_metadata
-tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_crud_snapshot_metadata
-tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_update_snapshot_metadata_item
-tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_create_get_delete_volume_metadata
-tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_crud_volume_metadata
-tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_update_volume_metadata_item
tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_attach_detach_volume_to_instance
tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_get_volume_attachment
-tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_reserve_unreserve_volume
-tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_volume_bootable
-tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_volume_readonly_update
-tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_attach_volumes_with_nonexistent_volume_id
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_detach_volumes_with_invalid_volume_id
+
+Test preconditions
+------------------
+
+* Volume extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+* Test action 1: Create a server VM1
+* Test action 2: Attach a provided VOL1 to VM1
+* **Test assertion 1:** Verify VOL1 is in 'in-use' status
+* Test action 3: Detach VOL1 from VM1
+* **Test assertion 2:** Verify VOL1 is in 'available' status
+* Test action 4: Create a server VM2
+* Test action 5: Attach a provided VOL2 to VM2 and wait for VOL2 to reach 'in-use' status
+* Test action 6: Retrieve VOL2's attachment information ATTCH_INFO
+* **Test assertion 3:** Verify ATTCH_INFO is correct
+* Test action 7: Create a server VM3 and wait for VM3 to reach 'ACTIVE' status
+* Test action 8: Attach a non-existent volume to VM3
+* **Test assertion 4:** Verify attach volume failed, a 'NOT FOUND' error is returned in the response
+* Test action 9: Detach a volume from a server by using an invalid volume ID
+* **Test assertion 5:** Verify detach volume failed, a 'NOT FOUND' error is returned in the response
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates the volume API ability of attaching a volume to a server
+and detaching a volume from a server. Specifically, the test verifies that:
+
+* Volumes can be attached and detached from servers.
+* Volume attachment information can be retrieved.
+* Attach and detach a volume using an invalid volume ID is not allowed.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+--------------------------------------------------------------------------------
+Test Case 2 - Volume service availability zone operations with the Cinder v2 API
+--------------------------------------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.volume.test_availability_zone.AvailabilityZoneV2TestJSON.test_get_availability_zone_list
+
+tempest.api.volume.test_availability_zone.AvailabilityZoneTestJSON.test_get_availability_zone_list
+
+Note: the second test case is the alias of the first one.
+Alias should always be included so that the test run will be tempest version agnostic,
+which can be used to test different version of Openstack.
+
+Test preconditions
+------------------
+
+* Volume extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+* Test action 1: List all existent availability zones
+* **Test assertion 1:** Verify the availability zone list length is greater than 0
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test case evaluates the volume API ability of listing availability zones.
+Specifically, the test verifies that:
+
+* Availability zones can be listed.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+--------------------------------------------------------------
+Test Case 3 - Volume cloning operations with the Cinder v2 API
+--------------------------------------------------------------
+
+Test case specification
+-----------------------
+
tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete_as_clone
+
+tempest.api.volume.test_volumes_get.VolumesGetTest.test_volume_create_get_update_delete_as_clone
+
+Note: the second test case is the alias of the first one.
+Alias should always be included so that the test run will be tempest version agnostic,
+which can be used to test different version of Openstack.
+
+Test preconditions
+------------------
+
+* Volume extension API
+* Cinder volume clones feature is enabled
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+* Test action 1: Create a volume VOL1
+* Test action 2: Create a volume VOL2 from source volume VOL1 with a specific name and metadata
+* Test action 2: Wait for VOL2 to reach 'available' status
+* **Test assertion 1:** Verify the name of VOL2 is correct
+* Test action 3: Retrieve VOL2's detail information
+* **Test assertion 2:** Verify the retrieved volume name, ID and metadata are the same as VOL2
+* **Test assertion 3:** Verify VOL2's bootable flag is 'False'
+* Test action 4: Update the name of VOL2 with the original value
+* Test action 5: Update the name of VOL2 with a new value
+* **Test assertion 4:** Verify the name of VOL2 is updated successfully
+* Test action 6: Create a volume VOL3 with no name specified and a description contains characters '@#$%^*'
+* **Test assertion 5:** Verify VOL3 is created successfully
+* Test action 7: Update the name of VOL3 and description with the original value
+* **Test assertion 6:** Verify VOL3's bootable flag is 'False'
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test case evaluates the volume API ability of creating a cloned volume from a source volume,
+getting cloned volume detail information and updating cloned volume attributes.
+
+Specifically, the test verifies that:
+
+* Cloned volume can be created from a source volume.
+* Cloned volume detail information can be retrieved.
+* Cloned volume detail information can be updated.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+--------------------------------------------------------------------
+Test Case 4 - Image copy-to-volume operations with the Cinder v2 API
+--------------------------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_volume_bootable
tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete_from_image
+
+tempest.api.volume.test_volumes_get.VolumesActionsTest.test_volume_bootable
+tempest.api.volume.test_volumes_get.VolumesGetTest.test_volume_create_get_update_delete_from_image
+
+Note: the last 2 test cases are the alias of the former 2.
+Alias should always be included so that the test run will be tempest version agnostic,
+which can be used to test different version of Openstack.
+
+Test preconditions
+------------------
+
+* Volume extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+* Test action 1: Set a provided volume VOL1's bootable flag to 'True'
+* Test action 2: Retrieve VOL1's bootable flag
+* **Test assertion 1:** Verify VOL1's bootable flag is 'True'
+* Test action 3: Set a provided volume VOL1's bootable flag to 'False'
+* Test action 4: Retrieve VOL1's bootable flag
+* **Test assertion 2:** Verify VOL1's bootable flag is 'False'
+* Test action 5: Create a bootable volume VOL2 from one image with a specific name and metadata
+* Test action 6: Wait for VOL2 to reach 'available' status
+* **Test assertion 3:** Verify the name of VOL2 name is correct
+* Test action 7: Retrieve VOL2's information
+* **Test assertion 4:** Verify the retrieved volume name, ID and metadata are the same as VOL2
+* **Test assertion 5:** Verify VOL2's bootable flag is 'True'
+* Test action 8: Update the name of VOL2 with the original value
+* Test action 9: Update the name of VOL2 with a new value
+* **Test assertion 6:** Verify the name of VOL2 is updated successfully
+* Test action 10: Create a volume VOL3 with no name specified and a description contains characters '@#$%^*'
+* **Test assertion 7:** Verify VOL3 is created successfully
+* Test action 11: Update the name of VOL3 and description with the original value
+* **Test assertion 8:** Verify VOL3's bootable flag is 'True'
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test case evaluates the volume API ability of updating volume's bootable flag and creating
+a bootable volume from an image, getting bootable volume detail information and updating bootable volume.
+
+Specifically, the test verifies that:
+
+* Volume bootable flag can be set and retrieved.
+* Bootable volume can be created from a source volume.
+* Bootable volume detail information can be retrieved.
+* Bootable volume detail information can be updated.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+----------------------------------------------------------------------------
+Test Case 5 - Volume creation and deletion operations with the Cinder v2 API
+----------------------------------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_invalid_size
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_source_volid
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_volume_type
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_out_passing_size
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_size_negative
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_size_zero
+
+tempest.api.volume.test_volumes_get.VolumesGetTest.test_volume_create_get_update_delete
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_invalid_size
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_nonexistent_source_volid
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_nonexistent_volume_type
+
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_without_passing_size
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_without_passing_size
+
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_size_negative
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_size_zero
+
+Note: test cases 8 to 11 are the alias of the fist 4 test cases, test cases 12 and 13 are both alias of
+test case 5, and test cases 14 and 15 are the alias of the cases 6 and 7, respectively.
+Alias should always be included so that the test run will be tempest version agnostic,
+which can be used to test different version of OpenStack.
+
+Test preconditions
+------------------
+
+* Volume extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+* Test action 1: Create a volume VOL1 with a specific name and metadata
+* Test action 2: Wait for VOL1 to reach 'available' status
+* **Test assertion 1:** Verify the name of VOL1 is correct
+* Test action 3: Retrieve VOL1's information
+* **Test assertion 2:** Verify the retrieved volume name, ID and metadata are the same as VOL1
+* **Test assertion 3:** Verify VOL1's bootable flag is 'False'
+* Test action 4: Update the name of VOL1 with the original value
+* Test action 5: Update the name of VOL1 with a new value
+* **Test assertion 4:** Verify the name of VOL1 is updated successfully
+* Test action 6: Create a volume VOL2 with no name specified and a description contains characters '@#$%^*'
+* **Test assertion 5:** Verify VOL2 is created successfully
+* Test action 7: Update the name of VOL2 and description with the original value
+* **Test assertion 6:** Verify VOL2's bootable flag is 'False'
+* Test action 8: Create a volume with an invalid size '#$%'
+* **Test assertion 7:** Verify create volume failed, a bad request error is returned in the response
+* Test action 9: Create a volume with a nonexistent source volume
+* **Test assertion 8:** Verify create volume failed, a 'Not Found' error is returned in the response
+* Test action 10: Create a volume with a nonexistent volume type
+* **Test assertion 9:** Verify create volume failed, a 'Not Found' error is returned in the response
+* Test action 11: Create a volume without passing a volume size
+* **Test assertion 10:** Verify create volume failed, a bad request error is returned in the response
+* Test action 12: Create a volume with a negative volume size
+* **Test assertion 11:** Verify create volume failed, a bad request error is returned in the response
+* Test action 13: Create a volume with volume size '0'
+* **Test assertion 12:** Verify create volume failed, a bad request error is returned in the response
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test case evaluates the volume API ability of creating a volume, getting volume
+detail information and updating volume, the reference is,
+Specifically, the test verifies that:
+
+* Volume can be created from a source volume.
+* Volume detail information can be retrieved/updated.
+* Create a volume with an invalid size is not allowed.
+* Create a volume with a nonexistent source volume or volume type is not allowed.
+* Create a volume without passing a volume size is not allowed.
+* Create a volume with a negative volume size is not allowed.
+* Create a volume with volume size '0' is not allowed.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+--------------------------------------------------------------------------------
+Test Case 6 - Volume service extension listing operations with the Cinder v2 API
+--------------------------------------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.volume.test_extensions.ExtensionsV2TestJSON.test_list_extensions
+
+tempest.api.volume.test_extensions.ExtensionsTestJSON.test_list_extensions
+
+Note: the second test case is the alias of the first one.
+Alias should always be included so that the test run will be tempest version agnostic,
+which can be used to test different version of Openstack.
+
+Test preconditions
+------------------
+
+* Volume extension API
+* At least one Cinder extension is configured
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: List all cinder service extensions
+* **Test assertion 1:** Verify all extensions are list in the extension list
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test case evaluates the volume API ability of listing all existent volume service extensions.
+
+* Cinder service extensions can be listed.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+----------------------------------------------------------
+Test Case 7 - Volume GET operations with the Cinder v2 API
+----------------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_get_invalid_volume_id
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_get_volume_without_passing_volume_id
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_volume_get_nonexistent_volume_id
+
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_get_invalid_volume_id
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_get_volume_without_passing_volume_id"
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_volume_get_nonexistent_volume_id
+
+Note: the latter 3 test cases is the alias of the first 3 ones.
+Alias should always be included so that the test run will be tempest version agnostic,
+which can be used to test different version of Openstack.
+
+Test preconditions
+------------------
+
+* Volume extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Retrieve a volume with an invalid volume ID
+* **Test assertion 1:** Verify retrieve volume failed, a 'Not Found' error is returned in the response
+* Test action 2: Retrieve a volume with an empty volume ID
+* **Test assertion 2:** Verify retrieve volume failed, a 'Not Found' error is returned in the response
+* Test action 3: Retrieve a volume with a nonexistent volume ID
+* **Test assertion 3:** Verify retrieve volume failed, a 'Not Found' error is returned in the response
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test case evaluates the volume API ability of getting volumes.
+Specifically, the test verifies that:
+
+* Get a volume with an invalid/an empty/a nonexistent volume ID is not allowed.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+--------------------------------------------------------------
+Test Case 8 - Volume listing operations with the Cinder v2 API
+--------------------------------------------------------------
+
+Test case specification
+-----------------------
+
tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list
tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_by_name
tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_by_name
@@ -43,40 +519,475 @@ tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volumes_list_by_
tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volumes_list_by_status
tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volumes_list_details_by_availability_zone
tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volumes_list_details_by_status
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_attach_volumes_with_nonexistent_volume_id
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_invalid_size
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_snapshot_id
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_source_volid
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_volume_type
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_out_passing_size
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_size_negative
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_size_zero
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_without_passing_size
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_delete_invalid_volume_id
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_delete_volume_without_passing_volume_id
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_detach_volumes_with_invalid_volume_id
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_get_invalid_volume_id
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_get_volume_without_passing_volume_id
tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_detail_with_invalid_status
tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_detail_with_nonexistent_name
tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_with_invalid_status
tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_with_nonexistent_name
+tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_pagination
+tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_with_multiple_params
+tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_pagination
+
+tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list
+tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_by_name
+tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_details_by_name
+tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_param_display_name_and_status
+tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_with_detail_param_display_name_and_status
+tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_with_detail_param_metadata
+tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_with_details
+tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_with_param_metadata
+tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_by_availability_zone
+tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_by_status
+tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_details_by_availability_zone
+tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volume_list_details_by_status
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_list_volumes_detail_with_invalid_status
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_list_volumes_detail_with_nonexistent_name
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_list_volumes_with_invalid_status
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_list_volumes_with_nonexistent_name
+tempest.api.volume.v2.test_volumes_list.VolumesListTestJSON.test_volume_list_details_pagination
+tempest.api.volume.v2.test_volumes_list.VolumesListTestJSON.test_volume_list_details_with_multiple_params
+tempest.api.volume.v2.test_volumes_list.VolumesListTestJSON.test_volume_list_pagination
+
+Note: the latter 19 test cases is the alias of the first 19 ones.
+Alias should always be included so that the test run will be tempest version agnostic,
+which can be used to test different version of Openstack.
+
+Test preconditions
+------------------
+
+* Volume extension API
+* The backing file for the volume group that Nova uses has space for at least 3 1G volumes
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: List all existent volumes
+* **Test assertion 1:** Verify the volume list is complete
+* Test action 2: List existent volumes and filter the volume list by volume name
+* **Test assertion 2:** Verify the length of filtered volume list is 1 and the retrieved volume is correct
+* Test action 3: List existent volumes in detail and filter the volume list by volume name
+* **Test assertion 3:** Verify the length of filtered volume list is 1 and the retrieved volume is correct
+* Test action 4: List existent volumes and filter the volume list by volume name and status 'available'
+* **Test assertion 4:** Verify the name and status parameters of the fetched volume are correct
+* Test action 5: List existent volumes in detail and filter the volume list by volume name and status 'available'
+* **Test assertion 5:** Verify the name and status parameters of the fetched volume are correct
+* Test action 6: List all existent volumes in detail and filter the volume list by volume metadata
+* **Test assertion 6:** Verify the metadata parameter of the fetched volume is correct
+* Test action 7: List all existent volumes in detail
+* **Test assertion 7:** Verify the volume list is complete
+* Test action 8: List all existent volumes and filter the volume list by volume metadata
+* **Test assertion 8:** Verify the metadata parameter of the fetched volume is correct
+* Test action 9: List existent volumes and filter the volume list by availability zone
+* **Test assertion 9:** Verify the availability zone parameter of the fetched volume is correct
+* Test action 10: List all existent volumes and filter the volume list by volume status 'available'
+* **Test assertion 10:** Verify the status parameter of the fetched volume is correct
+* Test action 11: List existent volumes in detail and filter the volume list by availability zone
+* **Test assertion 11:** Verify the availability zone parameter of the fetched volume is correct
+* Test action 12: List all existent volumes in detail and filter the volume list by volume status 'available'
+* **Test assertion 12:** Verify the status parameter of the fetched volume is correct
+* Test action 13: List all existent volumes in detail and filter the volume list by an invalid volume status 'null'
+* **Test assertion 13:** Verify the filtered volume list is empty
+* Test action 14: List all existent volumes in detail and filter the volume list by a non-existent volume name
+* **Test assertion 14:** Verify the filtered volume list is empty
+* Test action 15: List all existent volumes and filter the volume list by an invalid volume status 'null'
+* **Test assertion 15:** Verify the filtered volume list is empty
+* Test action 16: List all existent volumes and filter the volume list by a non-existent volume name
+* **Test assertion 16:** Verify the filtered volume list is empty
+* Test action 17: List all existent volumes in detail and paginate the volume list by desired volume IDs
+* **Test assertion 17:** Verify only the desired volumes are listed in the filtered volume list
+* Test action 18: List all existent volumes in detail and filter the volume list by volume status 'available' and display limit '2'
+* Test action 19: Sort the filtered volume list by IDs in ascending order
+* **Test assertion 18:** Verify the length of filtered volume list is 2
+* **Test assertion 19:** Verify the status of retrieved volumes is correct
+* **Test assertion 20:** Verify the filtered volume list is sorted correctly
+* Test action 20: List all existent volumes in detail and filter the volume list by volume status 'available' and display limit '2'
+* Test action 21: Sort the filtered volume list by IDs in descending order
+* **Test assertion 21:** Verify the length of filtered volume list is 2
+* **Test assertion 22:** Verify the status of retrieved volumes is correct
+* **Test assertion 23:** Verify the filtered volume list is sorted correctly
+* Test action 22: List all existent volumes and paginate the volume list by desired volume IDs
+* **Test assertion 24:** Verify only the desired volumes are listed in the filtered volume list
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test case evaluates the volume API ability of getting a list of volumes and filtering the volume list.
+Specifically, the test verifies that:
+
+* Get a list of volumes (in detail) successful.
+* Get a list of volumes (in detail) and filter volumes by name/status/metadata/availability zone successful.
+* Volume list pagination functionality is working.
+* Get a list of volumes in detail using combined condition successful.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+---------------------------------------------------------------
+Test Case 9 - Volume metadata operations with the Cinder v2 API
+---------------------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_create_get_delete_volume_metadata
+tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_update_volume_metadata_item
+
+tempest.api.volume.test_volume_metadata.VolumesMetadataTest.test_crud_volume_metadata
+tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_crud_volume_metadata
+
+tempest.api.volume.test_volume_metadata.VolumesMetadataTest.test_update_volume_metadata_item
+tempest.api.volume.test_volume_metadata.VolumesMetadataTest.test_update_show_volume_metadata_item
+
+Note: Test case 3 and 4 are the alias of the first test case, and the last 2 test cases
+are the alias of the second test case.
+Alias should always be included so that the test run will be tempest version agnostic,
+which can be used to test different version of OpenStack.
+
+Test preconditions
+------------------
+
+* Volume extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create metadata for a provided volume VOL1
+* Test action 2: Get the metadata of VOL1
+* **Test assertion 1:** Verify the metadata of VOL1 is correct
+* Test action 3: Update the metadata of VOL1
+* **Test assertion 2:** Verify the metadata of VOL1 is updated
+* Test action 4: Delete one metadata item 'key1' of VOL1
+* **Test assertion 3:** Verify the metadata item 'key1' is deleted
+* Test action 5: Create metadata for a provided volume VOL2
+* **Test assertion 4:** Verify the metadata of VOL2 is correct
+* Test action 6: Update one metadata item 'key3' of VOL2
+* **Test assertion 5:** Verify the metadata of VOL2 is updated
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test case evaluates the volume API ability of creating metadata for a volume, getting the
+metadata of a volume, updating volume metadata and deleting a metadata item of a volume.
+Specifically, the test verifies that:
+
+* Create metadata for volume successfully.
+* Get metadata of volume successfully.
+* Update volume metadata and metadata item successfully.
+* Delete metadata item of a volume successfully.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+---------------------------------------------------------------------------------
+Test Case 10 - Verification of read-only status on volumes with the Cinder v2 API
+---------------------------------------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_volume_readonly_update
+
+tempest.api.volume.test_volumes_actions.VolumesActionsTest.test_volume_readonly_update
+
+Note: the second test case is the alias of the first one.
+Alias should always be included so that the test run will be tempest version agnostic,
+which can be used to test different version of Openstack.
+
+Test preconditions
+------------------
+
+* Volume extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Update a provided volume VOL1's read-only access mode to 'True'
+* **Test assertion 1:** Verify VOL1 is in read-only access mode
+* Test action 2: Update a provided volume VOL1's read-only access mode to 'False'
+* **Test assertion 2:** Verify VOL1 is not in read-only access mode
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test case evaluates the volume API ability of setting and updating volume read-only access mode.
+Specifically, the test verifies that:
+
+* Volume read-only access mode can be set and updated.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+-------------------------------------------------------------------
+Test Case 11 - Volume reservation operations with the Cinder v2 API
+-------------------------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_reserve_unreserve_volume
tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_reserve_volume_with_negative_volume_status
tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_reserve_volume_with_nonexistent_volume_id
tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_unreserve_volume_with_nonexistent_volume_id
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_empty_volume_id
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_invalid_volume_id
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_nonexistent_volume_id
+
+tempest.api.volume.test_volumes_actions.VolumesActionsTest.test_reserve_unreserve_volume
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_reserve_volume_with_negative_volume_status
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_reserve_volume_with_nonexistent_volume_id
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_unreserve_volume_with_nonexistent_volume_id
+
+Note: the last 4 test cases are the alias of the first 4 ones.
+Alias should always be included so that the test run will be tempest version agnostic,
+which can be used to test different version of Openstack.
+
+Test preconditions
+------------------
+
+* Volume extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Update a provided volume VOL1 as reserved
+* **Test assertion 1:** Verify VOL1 is in 'attaching' status
+* Test action 2: Update VOL1 as un-reserved
+* **Test assertion 2:** Verify VOL1 is in 'available' status
+* Test action 3: Update a provided volume VOL2 as reserved
+* Test action 4: Update VOL2 as reserved again
+* **Test assertion 3:** Verify update VOL2 status failed, a bad request error is returned in the response
+* Test action 5: Update VOL2 as un-reserved
+* Test action 6: Update a non-existent volume as reserved by using an invalid volume ID
+* **Test assertion 4:** Verify update non-existent volume as reserved failed, a 'Not Found' error is returned in the response
+* Test action 7: Update a non-existent volume as un-reserved by using an invalid volume ID
+* **Test assertion 5:** Verify update non-existent volume as un-reserved failed, a 'Not Found' error is returned in the response
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test case evaluates the volume API ability of reserving and un-reserving volumes.
+Specifically, the test verifies that:
+
+* Volume can be reserved and un-reserved.
+* Update a non-existent volume as reserved is not allowed.
+* Update a non-existent volume as un-reserved is not allowed.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+----------------------------------------------------------------------------------
+Test Case 12 - Volume snapshot creation/deletion operations with the Cinder v2 API
+----------------------------------------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_create_get_delete_snapshot_metadata
+tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_update_snapshot_metadata_item
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_snapshot_id
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_delete_invalid_volume_id
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_delete_volume_without_passing_volume_id
tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_volume_delete_nonexistent_volume_id
-tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_volume_get_nonexistent_volume_id
tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshot_create_get_list_update_delete
+tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_volume_from_snapshot
tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshots_list_details_with_params
tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshots_list_with_params
-tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_volume_from_snapshot
-tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_details_with_params
-tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_with_params
tempest.api.volume.test_volumes_snapshots_negative.VolumesV2SnapshotNegativeTestJSON.test_create_snapshot_with_nonexistent_volume_id
tempest.api.volume.test_volumes_snapshots_negative.VolumesV2SnapshotNegativeTestJSON.test_create_snapshot_without_passing_volume_id
-tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_pagination
-tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_with_multiple_params
-tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_pagination
+
+tempest.api.volume.test_snapshot_metadata.SnapshotMetadataTestJSON.test_crud_snapshot_metadata
+tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_crud_snapshot_metadata
+
+tempest.api.volume.test_snapshot_metadata.SnapshotMetadataTestJSON.test_update_snapshot_metadata_item
+tempest.api.volume.test_snapshot_metadata.SnapshotMetadataTestJSON.test_update_show_snapshot_metadata_item
+
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_create_volume_with_nonexistent_snapshot_id
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_delete_invalid_volume_id
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_delete_volume_without_passing_volume_id
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_volume_delete_nonexistent_volume_id
+tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_snapshot_create_get_list_update_delete
+tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_volume_from_snapshot
+
+tempest.api.volume.test_volumes_snapshots_list.VolumesSnapshotListTestJSON.test_snapshots_list_details_with_params
+tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_details_with_params
+
+tempest.api.volume.test_volumes_snapshots_list.VolumesSnapshotListTestJSON.test_snapshots_list_with_params
+tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_with_params
+
+tempest.api.volume.test_volumes_snapshots_negative.VolumesSnapshotNegativeTestJSON.test_create_snapshot_with_nonexistent_volume_id
+tempest.api.volume.test_volumes_snapshots_negative.VolumesSnapshotNegativeTestJSON.test_create_snapshot_without_passing_volume_id
+
+Note: test case 13 and 14 are the alias of test case 1, test case 15 and 16 are the alias of test case 2,
+test case 17 to 22 are the alias of test case 3 to 8 respectively, test case 23 and 24 are the alias of
+test case 9, test case 25 and 26 are the alias of test case 10, and test case 27 and 28 are the alias of
+test case 11 and 12 respectively.
+Alias should always be included so that the test run will be tempest version agnostic,
+which can be used to test different version of OpenStack.
+
+Test preconditions
+------------------
+
+* Volume extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create metadata for a provided snapshot SNAP1
+* Test action 2: Get the metadata of SNAP1
+* **Test assertion 1:** Verify the metadata of SNAP1 is correct
+* Test action 3: Update the metadata of SNAP1
+* **Test assertion 2:** Verify the metadata of SNAP1 is updated
+* Test action 4: Delete one metadata item 'key3' of SNAP1
+* **Test assertion 3:** Verify the metadata item 'key3' is deleted
+* Test action 5: Create metadata for a provided snapshot SNAP2
+* **Test assertion 4:** Verify the metadata of SNAP2 is correct
+* Test action 6: Update one metadata item 'key3' of SNAP2
+* **Test assertion 5:** Verify the metadata of SNAP2 is updated
+* Test action 7: Create a volume with a nonexistent snapshot
+* **Test assertion 6:** Verify create volume failed, a 'Not Found' error is returned in the response
+* Test action 8: Delete a volume with an invalid volume ID
+* **Test assertion 7:** Verify delete volume failed, a 'Not Found' error is returned in the response
+* Test action 9: Delete a volume with an empty volume ID
+* **Test assertion 8:** Verify delete volume failed, a 'Not Found' error is returned in the response
+* Test action 10: Delete a volume with a nonexistent volume ID
+* **Test assertion 9:** Verify delete volume failed, a 'Not Found' error is returned in the response
+* Test action 11: Create a snapshot SNAP2 from a provided volume VOL1
+* Test action 12: Retrieve SNAP2's detail information
+* **Test assertion 10:** Verify SNAP2 is created from VOL1
+* Test action 13: Update the name and description of SNAP2
+* **Test assertion 11:** Verify the name and description of SNAP2 are updated in the response body of update snapshot API
+* Test action 14: Retrieve SNAP2's detail information
+* **Test assertion 12:** Verify the name and description of SNAP2 are correct
+* Test action 15: Delete SNAP2
+* Test action 16: Create a volume VOL2 with a volume size
+* Test action 17: Create a snapshot SNAP3 from VOL2
+* Test action 18: Create a volume VOL3 from SNAP3 with a bigger volume size
+* Test action 19: Retrieve VOL3's detail information
+* **Test assertion 13:** Verify volume size and source snapshot of VOL3 are correct
+* Test action 20: List all snapshots in detail and filter the snapshot list by name
+* **Test assertion 14:** Verify the filtered snapshot list is correct
+* Test action 21: List all snapshots in detail and filter the snapshot list by status
+* **Test assertion 15:** Verify the filtered snapshot list is correct
+* Test action 22: List all snapshots in detail and filter the snapshot list by name and status
+* **Test assertion 16:** Verify the filtered snapshot list is correct
+* Test action 23: List all snapshots and filter the snapshot list by name
+* **Test assertion 17:** Verify the filtered snapshot list is correct
+* Test action 24: List all snapshots and filter the snapshot list by status
+* **Test assertion 18:** Verify the filtered snapshot list is correct
+* Test action 25: List all snapshots and filter the snapshot list by name and status
+* **Test assertion 19:** Verify the filtered snapshot list is correct
+* Test action 26: Create a snapshot from a nonexistent volume by using an invalid volume ID
+* **Test assertion 20:** Verify create snapshot failed, a 'Not Found' error is returned in the response
+* Test action 27: Create a snapshot from a volume by using an empty volume ID
+* **Test assertion 21:** Verify create snapshot failed, a 'Not Found' error is returned in the response
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test case evaluates the volume API ability of managing snapshot and snapshot metadata.
+Specifically, the test verifies that:
+
+* Create metadata for snapshot successfully.
+* Get metadata of snapshot successfully.
+* Update snapshot metadata and metadata item successfully.
+* Delete metadata item of a snapshot successfully.
+* Create a volume from a nonexistent snapshot is not allowed.
+* Delete a volume using an invalid volume ID is not allowed.
+* Delete a volume without passing the volume ID is not allowed.
+* Delete a non-existent volume is not allowed.
+* Create snapshot successfully.
+* Get snapshot's detail information successfully.
+* Update snapshot attributes successfully.
+* Delete snapshot successfully.
+* Creates a volume and a snapshot passing a size different from the source successfully.
+* List snapshot details by display_name and status filters successfully.
+* Create a snapshot from a nonexistent volume is not allowed.
+* Create a snapshot from a volume without passing the volume ID is not allowed.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
+
+--------------------------------------------------------------
+Test Case 13 - Volume update operations with the Cinder v2 API
+--------------------------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_empty_volume_id
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_invalid_volume_id
+tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_nonexistent_volume_id
+
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_update_volume_with_empty_volume_id
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_update_volume_with_invalid_volume_id
+tempest.api.volume.test_volumes_negative.VolumesNegativeTest.test_update_volume_with_nonexistent_volume_id
+
+Note: the last 3 test cases are the alias of the first 3 ones.
+Alias should always be included so that the test run will be tempest version agnostic,
+which can be used to test different version of Openstack.
+
+Test preconditions
+------------------
+
+* Volume extension API
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Update a volume by using an empty volume ID
+* **Test assertion 1:** Verify update volume failed, a 'Not Found' error is returned in the response
+* Test action 2: Update a volume by using an invalid volume ID
+* **Test assertion 2:** Verify update volume failed, a 'Not Found' error is returned in the response
+* Test action 3: Update a non-existent volume by using a random generated volume ID
+* **Test assertion 3:** Verify update volume failed, a 'Not Found' error is returned in the response
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test case evaluates the volume API ability of updating volume attributes.
+Specifically, the test verifies that:
+
+* Update a volume without passing the volume ID is not allowed.
+* Update a volume using an invalid volume ID is not allowed.
+* Update a non-existent volume is not allowed.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+N/A
diff --git a/docs/testing/user/testspecification/vping/index.rst b/docs/testing/user/testspecification/vping/index.rst
new file mode 100644
index 00000000..d7a207c0
--- /dev/null
+++ b/docs/testing/user/testspecification/vping/index.rst
@@ -0,0 +1,279 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) Ericsson AB
+
+========================
+Vping test specification
+========================
+
+.. toctree::
+ :maxdepth: 2
+
+Scope
+=====
+
+The vping test area evaluates basic NFVi capabilities of the system under test.
+These capabilities include creating a small number of virtual machines,
+establishing basic L3 connectivity between them and verifying connectivity by
+means of ICMP packets.
+
+
+References
+==========
+
+- Neutron Client
+
+ - https://docs.openstack.org/developer/python-neutronclient/usage/library.html
+
+- Nova Client
+
+ - https://docs.openstack.org/developer/python-novaclient/ref/v2/servers.html
+
+- SSHClient
+
+ - http://docs.paramiko.org/en/2.2/
+
+- SCPClient
+
+ - https://pypi.python.org/pypi/scp
+
+
+Definitions and abbreviations
+=============================
+
+The following terms and abbreviations are used in conjunction with this test
+area
+
+- ICMP - Internet Control Message Protocol
+- L3 - Layer 3
+- NFVi - Network functions virtualization infrastructure
+- SCP - Secure Copy
+- SSH - Secure Shell
+- VM - Virtual machine
+
+
+System Under Test (SUT)
+=======================
+
+The system under test is assumed to be the NFVi and VIM in operation on a
+Pharos compliant infrastructure.
+
+
+Test Area Structure
+===================
+
+The test area is structured in two separate tests which are executed
+sequentially. The order of the tests is arbitrary as there are no dependencies
+across the tests.
+
+
+Test Descriptions
+=================
+
+--------------------------------------------------------------------
+Test Case 1 - vPing using userdata provided by nova metadata service
+--------------------------------------------------------------------
+
+Short name
+----------
+
+opnfv.vping.userdata
+
+
+Use case specification
+----------------------
+
+This test evaluates the use case where an NFVi tenant boots up two VMs and
+requires L3 connectivity between those VMs. The target IP is passed to the VM
+that will initiate pings by using a custom userdata script provided by nova metadata service.
+
+
+Test preconditions
+------------------
+
+At least one compute node is available. No further pre-configuration needed.
+
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Methodology for verifying connectivity
+''''''''''''''''''''''''''''''''''''''
+
+Connectivity between VMs is tested by sending ICMP ping packets between
+selected VMs. The target IP is passed to the VM sending pings by using a
+custom userdata script by means of the config driver mechanism provided by
+Nova metadata service. Whether or not a ping was successful is determined by
+checking the console output of the source VMs.
+
+
+Test execution
+''''''''''''''
+
+* Test action 1:
+ * Create a private tenant network by using neutron client
+ * Create one subnet and one router in the network by neutron client
+ * Add one interface between the subnet and router
+ * Add one gateway route to the router by neutron client
+ * Store the network id in the response
+* **Test assertion 1:** The network id, subnet id and router id can be found in the response
+* Test action 2:
+ * Create an security group by using neutron client
+ * Store the security group id parameter in the response
+* **Test assertion 2:** The security group id can be found in the response
+* Test action 3: boot VM1 by using nova client with configured name, image, flavor, private tenant
+ network created in test action 1, security group created in test action 2
+* **Test assertion 3:** The VM1 object can be found in the response
+* Test action 4: Generate ping script with the IP of VM1 to be passed as userdata provided by
+ the **nova metadata service**.
+* Test action 5: Boot VM2 by using nova client with configured name, image, flavor, private tenant
+ network created in test action 1, security group created in test action 2, userdata created
+ in test action 4
+* **Test assertion 4:** The VM2 object can be found in the response
+* Test action 6: Inside VM2, the ping script is executed automatically when booted and it contains a
+ loop doing the ping until the return code is 0 or timeout reached. For each ping, when the return
+ code is 0, "vPing OK" is printed in the VM2 console-log, otherwise, "vPing KO" is printed.
+ Monitoring the console-log of VM2 to see the response generated by the script.
+* **Test assertion 5:** "vPing OK" is detected, when monitoring the console-log in VM2
+* Test action 7: delete VM1, VM2
+* **Test assertion 6:** VM1 and VM2 are not present in the VM list
+* Test action 8: delete security group, gateway, interface, router, subnet and network
+* **Test assertion 7:** The security group, gateway, interface, router, subnet and network are
+ no longer present in the lists after deleting
+
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates basic NFVi capabilities of the system under test.
+Specifically, the test verifies that:
+
+* Neutron client network, subnet, router, interface create commands return valid "id" parameters
+ which are shown in the create response message
+* Neutron client interface add command to add between subnet and router returns success code
+* Neutron client gateway add command to add to router returns success code
+* Neutron client security group create command returns valid "id" parameter which is shown in
+ the response message
+* Nova client VM create command returns valid VM attributes response message
+* Nova metadata server can transfer userdata configuration at nova client VM booting time
+* Ping command from one VM to the other in same private tenant network returns valid code
+* All items created using neutron client or nova client create commands are able to be removed by
+ using the returned identifiers
+
+In order to pass this test, all test assertions listed in the test execution
+above need to pass.
+
+
+Post conditions
+---------------
+
+None
+
+
+----------------------------------------------
+Test Case 2 - vPing using SSH to a floating IP
+----------------------------------------------
+
+Short name
+----------
+
+opnfv.vping.ssh
+
+
+Use case specification
+----------------------
+
+This test evaluates the use case where an NFVi tenant boots up two VMs and requires
+L3 connectivity between those VMs. An SSH connection is establised from the host to
+a floating IP associated with VM2 and ``ping`` is executed on VM2 with the IP of VM1 as target.
+
+
+Test preconditions
+------------------
+
+At least one compute node is available. There should exist an OpenStack external network
+and can assign floating IP.
+
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Methodology for verifying connectivity
+''''''''''''''''''''''''''''''''''''''
+
+Connectivity between VMs is tested by sending ICMP ping packets between
+selected VMs. To this end, the test establishes an SSH connection from the host
+running the test suite to a floating IP associated with VM2 and executes ``ping``
+on VM2 with the IP of VM1 as target.
+
+
+Test execution
+''''''''''''''
+
+
+* Test action 1:
+ * Create a private tenant network by neutron client
+ * Create one subnet and one router are created in the network by using neutron client
+ * Create one interface between the subnet and router
+ * Add one gateway route to the router by neutron client
+ * Store the network id in the response
+* **Test assertion 1:** The network id, subnet id and router id can be found in the response
+* Test action 2:
+ * Create an security group by using neutron client
+ * Store the security group id parameter in the response
+* **Test assertion 2:** The security group id can be found in the response
+* Test action 3: Boot VM1 by using nova client with configured name, image, flavor, private tenant
+ network created in test action 1, security group created in test action 2
+* **Test assertion 3:** The VM1 object can be found in the response
+* Test action 4: Boot VM2 by using nova client with configured name, image, flavor, private tenant
+ network created in test action 1, security group created in test action 2
+* **Test assertion 4:** The VM2 object can be found in the response
+* Test action 5: create one floating IP by using neutron client, storing the floating IP address
+ returned in the response
+* **Test assertion 5:** Floating IP address can be found in the response
+* Test action 6: Assign the floating IP address created in test action 5 to VM2 by using nova client
+* **Test assertion 6:** The assigned floating IP can be found in the VM2 console log file
+* Test action 7: Establish SSH connection between the test host and VM2 through the floating IP
+* **Test assertion 7:** SSH connection between the test host and VM2 is established within
+ 300 seconds
+* Test action 8: Copy the Ping script from the test host to VM2 by using SCPClient
+* **Test assertion 8:** The Ping script can be found inside VM2
+* Test action 9: Inside VM2, to execute the Ping script to ping VM1, the Ping script contains a
+ loop doing the ping until the return code is 0 or timeout reached, for each ping, when the return
+ code is 0, "vPing OK" is printed in the VM2 console-log, otherwise, "vPing KO" is printed.
+ Monitoring the console-log of VM2 to see the response generated by the script.
+* **Test assertion 9:** "vPing OK" is detected, when monitoring the console-log in VM2
+* Test action 10: delete VM1, VM2
+* **Test assertion 10:** VM1 and VM2 are not present in the VM list
+* Test action 11: delete floating IP, security group, gateway, interface, router, subnet and network
+* **Test assertion 11:** The security group, gateway, interface, router, subnet and network are
+ no longer present in the lists after deleting
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test evaluates basic NFVi capabilities of the system under test.
+Specifically, the test verifies that:
+
+* Neutron client network, subnet, router, interface create commands return valid "id" parameters
+ which are shown in the create response message
+* Neutron client interface add command to add between subnet and router return success code
+* Neutron client gateway add command to add to router return success code
+* Neutron client security group create command returns valid "id" parameter which is shown in the
+ response message
+* Nova client VM create command returns valid VM attributes response message
+* Neutron client floating IP create command return valid floating IP address
+* Nova client add floating IP command returns valid response message
+* SSH connection can be established using a floating IP
+* Ping command from one VM to another in same private tenant network returns valid code
+* All items created using neutron client or nova client create commands are able to be removed by
+ using the returned identifiers
+
+In order to pass this test, all test assertions listed in the test execution
+above need to pass.
+
+
+Post conditions
+---------------
+
+None
diff --git a/docs/testing/user/userguide/testing_guide.rst b/docs/testing/user/userguide/testing_guide.rst
index 9e46b561..08fd8acf 100644
--- a/docs/testing/user/userguide/testing_guide.rst
+++ b/docs/testing/user/userguide/testing_guide.rst
@@ -261,8 +261,8 @@ If the Test Host is online, you can directly pull.
.. code-block:: bash
- $ sudo docker pull opnfv/dovetail:cvp.0.3.0
- cvp.0.3.0: Pulling from opnfv/dovetail
+ $ sudo docker pull opnfv/dovetail:cvp.0.5.0
+ cvp.0.5.0: Pulling from opnfv/dovetail
30d541b48fc0: Pull complete
8ecd7f80d390: Pull complete
46ec9927bb81: Pull complete
@@ -275,9 +275,9 @@ If the Test Host is online, you can directly pull.
bf7c644692de: Pull complete
cdc345e3f363: Pull complete
Digest: sha256:d571b1073b2fdada79562e8cc67f63018e8d89268ff7faabee3380202c05edee
- Status: Downloaded newer image for opnfv/dovetail:cvp.0.3.0
+ Status: Downloaded newer image for opnfv/dovetail:cvp.0.5.0
-An example of the <tag> is *cvp.0.3.0*.
+An example of the <tag> is *cvp.0.5.0*.
If the Test Host is offline, you will need to first pull the Dovetail Docker image, and all the
dependent images that Dovetail uses, to a host that is online. The reason that you need
@@ -287,9 +287,10 @@ offline, then all these dependencies will also need to be manually copied.
.. code-block:: bash
- $ sudo docker pull opnfv/dovetail:cvp.0.3.0
- $ sudo docker pull opnfv/functest:cvp.0.2.0
- $ sudo docker pull opnfv/yardstick:danube.3.1
+ $ sudo docker pull opnfv/dovetail:cvp.0.5.0
+ $ sudo docker pull opnfv/functest:cvp.0.5.0
+ $ sudo docker pull opnfv/yardstick:danube.3.2
+ $ sudo docker pull opnfv/bottlenecks:cvp.0.4.0
$ sudo wget -nc http://artifacts.opnfv.org/sdnvpn/ubuntu-16.04-server-cloudimg-amd64-disk1.img -P {ANY_DIR}
Once all these images are pulled, save the images, copy to the Test Host, and then load
@@ -299,7 +300,8 @@ At the online host, save images.
.. code-block:: bash
- $ sudo docker save -o dovetail.tar opnfv/dovetail:cvp.0.3.0 opnfv/functest:cvp.0.2.0 opnfv/yardstick:danube.3.1
+ $ sudo docker save -o dovetail.tar opnfv/dovetail:cvp.0.5.0 opnfv/functest:cvp.0.5.0 \
+ opnfv/yardstick:danube.3.2 opnfv/bottlenecks:cvp.0.4.0
Copy dovetail.tar file to the Test Host, and then load the images on the Test Host.
@@ -315,9 +317,10 @@ Now check to see that the Dovetail image has been pulled or loaded properly.
$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
- opnfv/functest cvp.0.2.0 9eaeaea5f203 8 days ago 1.53GB
- opnfv/dovetail cvp.0.3.0 5d25b289451c 8 days ago 516MB
- opnfv/yardstick danube.3.1 574596b6ea12 8 days ago 1.2GB
+ opnfv/functest cvp.0.5.0 9eaeaea5f203 8 days ago 1.53GB
+ opnfv/dovetail cvp.0.5.0 5d25b289451c 8 days ago 516MB
+ opnfv/yardstick danube.3.2 574596b6ea12 8 days ago 1.2GB
+ opnfv/bottlenecks cvp.0.4.0 00450688bcae 3 hours ago 622 MB
Regardless of whether you pulled down the Dovetail image directly online, or loaded from
a static image tar file, you are ready to run Dovetail.
@@ -463,8 +466,39 @@ When a tester is performing trial runs, Dovetail stores results in a local file
* For each test case result in Yardstick, the logs are stored in dovetail_ha_tcXXX.out, respectively.
#. OPNFV web interface
+ CVP will host a web site to collect test results. Users can upload their results to this web site,
+ so they can review these results in the future.
- Wait for the complement of LF, test community, etc.
+ * web site url
+
+ * Wait for the complement of LF, test community, etc.
+
+ * Sign in / Sign up
+
+ * You need to sign in you account, then you can upload results, and check your private results.
+ CVP is now using openstack id as account provider, but will soon support Linux Foundation ID
+ as well.
+
+ * If you already have a openstack id, you can sign in directly with your id.
+
+ * If you do not have a openstack id, you can sign up a new one on the sign up page.
+
+ * If you do not sign in, you can only check the community results.
+
+ * My results
+
+ * This page lists all results uploaded by you after you signed in,
+
+ * You can also upload your results on this page.
+
+ * There is a *choose file* button, once you click it, you can choose your reuslt file in your harddisk
+ then click the *upload* button, and you will see a results id once your uploading succeed.
+
+ * Check the *review* box to submit your result to the OPNFV. Uncheck the box to withdraw your result.
+
+ * profile
+
+ * This page shows your account info after you signed in.
Updating Dovetail or a Test Suite
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^