Merge "Test case specification for VIM Image"

1 files changed, 375 insertions, 17 deletions

diff --git a/docs/testing/user/testspecification/vimoperationsimage/index.rst b/docs/testing/user/testspecification/vimoperationsimage/index.rst
index 932c7382..8e5f08ce 100644
--- a/docs/testing/user/testspecification/vimoperationsimage/index.rst
+++ b/docs/testing/user/testspecification/vimoperationsimage/index.rst
@@ -1,6 +1,6 @@
 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
 .. http://creativecommons.org/licenses/by/4.0
-.. (c) Ericsson AB
+.. (c) Ericsson AB, Huawei Technologies Co.,Ltd
 
 =======================================
 VIM image operations test specification
@@ -9,17 +9,191 @@ VIM image operations test specification
 .. toctree::
    :maxdepth: 2
 
-Each test case requires documentation according to:
-* Use case specification
-* Test preconditions
-* Basic test flow execution descriptor
-* Post conditions and pass fail criteria
+Scope
+=====
+
+The VIM image test area evaluates the ability of the system under test to support
+VIM image operations. The test cases documented here are the Image API test cases
+in the Openstack Interop guideline 2016.8 as implemented by the Refstack client.
+These test cases will evaluate basic Openstack (as a VIM) image operations including
+image creation, image list, image update and image deletion capabilities using Glance v2 API.
+
+References
+==========
+
+- Defcore test cases
+
+  - https://github.com/openstack/interop/blob/master/2016.08.json
+
+- Refstack client
+
+  - https://github.com/openstack/refstack-client
+
+Definitions and abbreviations
+=============================
+
+The following terms and abbreviations are used in conjunction with this test area
+
+- API - Application Programming Interface
+- CRUD - Create, Read, Update, and Delete
+- NFVi - Network Functions Virtualization infrastructure
+- VIM - Virtual Infrastructure Manager
+
+System Under Test (SUT)
+=======================
+
+The system under test is assumed to be the NFVi and VIM in operation on a
+Pharos compliant infrastructure.
+
+Test Area Structure
+====================
+
+The test area is structured based on VIM image operations. Each test case is able
+to run independently, i.e. irrelevant of the state created by a previous test.
+
+For brevity, the test cases in this test area are summarized together based on
+the operations they are testing.
+
+Test Descriptions
+=================
+
+API Used and Reference
+----------------------
+
+Images: https://developer.openstack.org/api-ref/image/v2/
+
+- create image
+- delete image
+- show image details
+- show images
+- show image schema
+- show images schema
+- upload binary image data
+- add image tag
+- delete image tag
+
+---------------------------------------
+Image get tests using the Glance v2 API
+---------------------------------------
+
+Test case specification
+-----------------------
 
-tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_delete_image
-tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_update_image
 tempest.api.image.v2.test_images.ListImagesTest.test_get_image_schema
 tempest.api.image.v2.test_images.ListImagesTest.test_get_images_schema
+tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_delete_deleted_image
+tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_image_null_id
+tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_non_existent_image
+
+tempest.api.image.v2.test_images.ListUserImagesTest.test_get_image_schema
+tempest.api.image.v2.test_images.ListUserImagesTest.test_get_images_schema
+
+Note: the latter two test cases are the alias of the former first two, respectively.
+Alias should always be included so that the test run will be tempest version agnostic,
+which can be used to test different version of Openstack.
+
+Test preconditions
+------------------
+
+Glance is available.
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create 6 images and store their ids in a created images list.
+* Test action 2: Use image v2 API to show image schema and check the body of the response.
+* **Test assertion 1:** In the body of the response, the value of the key 'name' is 'image'.
+* Test action 3: Use image v2 API to show images schema and check the body of the response.
+* **Test assertion 2:** In the body of the response, the value of the key 'name' is 'images'.
+* Test action 4: Create an image with name 'test', container_formats 'bare' and
+  disk_formats 'raw'. Delete this image with its id and then try to show it with
+  its id. Delete this deleted image again with its id and check the API's response code.
+* **Test assertion 3:** The operations of showing and deleting a deleted image with its id
+  both get 404 response code.
+* Test action 5: Use a null image id to show a image and check the API's response code.
+* **Test assertion 4:** The API's response code is 404.
+* Test action 6: Generate a random uuid and use it as the image id to show the image.
+* **Test assertion 5:** The API's response code is 404.
+* Test action 7: Delete the 6 images with the stored ids. Show all images and check
+  whether the 6 images' ids are not in the show list.
+* **Test assertion 6:** The 6 images' ids are not found in the show list.
+
+Pass / fail criteria
+''''''''''''''''''''
+
+The first two test cases evaluate the ability to use Glance v2 API to show image
+and images schema. The latter three test cases evaluate the ability to use Glance
+v2 API to show images with a deleted image id, a null image id and a non-existing
+image id. Specifically it verifies that:
+
+* Glance image get API can show the image and images schema.
+* Glance image get API can't show an image with a deleted image id.
+* Glance image get API can't show an image with a null image id.
+* Glance image get API can't show an image with a non-existing image id.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+None
+
+--------------------------------------
+CRUD image operations in Images API v2
+--------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.image.v2.test_images.ListImagesTest.test_list_no_params
+
 tempest.api.image.v2.test_images.ListImagesTest.test_index_no_params
+tempest.api.image.v2.test_images.ListUserImagesTest.test_list_no_params
+
+Note: the latter two test cases are the alias of the former one. Alias should
+always be included so that the test run will be tempest version agnostic,
+which can be used to test different version of Openstack.
+
+Test preconditions
+------------------
+
+Glance is available.
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create 6 images and store their ids in a created images list.
+* Test action 2: List all images and check whether the ids listed are in the created images list.
+* **Test assertion 1:** The ids get from the list images API are in the created images list.
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test case evaluates the ability to use Glance v2 API to list images.
+Specifically it verifies that:
+
+* Glance image API can show the images.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+None
+
+----------------------------------------
+Image list tests using the Glance v2 API
+----------------------------------------
+
+Test case specification
+-----------------------
+
 tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_container_format
 tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_disk_format
 tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_limit
@@ -27,9 +201,7 @@ tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_min_max_s
 tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_size
 tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_status
 tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_visibility
-tempest.api.image.v2.test_images.ListImagesTest.test_list_no_params
-tempest.api.image.v2.test_images.ListUserImagesTest.test_get_image_schema
-tempest.api.image.v2.test_images.ListUserImagesTest.test_get_images_schema
+
 tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_container_format
 tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_disk_format
 tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_limit
@@ -37,12 +209,198 @@ tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_min_m
 tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_size
 tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_status
 tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_visibility
-tempest.api.image.v2.test_images.ListUserImagesTest.test_list_no_params
+
+Note: the latter 7 test cases are the alias of the former 7, respectively.
+Alias should always be included so that the test run will be tempest version agnostic,
+which can be used to test different version of Openstack.
+
+Test preconditions
+------------------
+
+Glance is available.
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create 6 images with a random size ranging from 1024 to 4096 and
+  visibility 'private'; set their (container_format, disk_format) pair to be
+  (ami, ami), (ami, ari), (ami, aki), (ami, vhd), (ami, vmdk) and (ami, raw);
+  store their ids in a list and upload the binary images data.
+* Test action 2: Use Glance v2 API to list all images whose container_format is 'ami'
+  and store the response details in a list.
+* **Test assertion 1:** The list is not empty and all the values of container_format
+  in the list are 'ami'.
+* Test action 3: Use Glance v2 API to list all images whose disk_format is 'raw'
+  and store the response details in a list.
+* **Test assertion 2:** The list is not empty and all the values of disk_format
+  in the list are 'raw'.
+* Test action 4: Use Glance v2 API to list one image by setting limit to be 1 and
+  store the response details in a list.
+* **Test assertion 3:** The length of the list is one.
+* Test action 5: Use Glance v2 API to list images by setting size_min and size_max,
+  and store the response images' sizes in a list. Choose the first image's size as
+  the median, size_min is median-500 and size_max is median+500.
+* **Test assertion 4:** All sizes in the list are no less than size_min and no more
+  than size_max.
+* Test action 6: Use Glance v2 API to show the first created image with its id and
+  get its size from the response. Use Glance v2 API to list images whose size is equal
+  to this size and store the response details in a list.
+* **Test assertion 5:** All sizes of the images in the list are equal to the size
+  used to list the images.
+* Test action 7: Use Glance v2 API to list the images whose status is active and
+  store the response details in a list.
+* **Test assertion 6:** All status of images in the list are active.
+* Test action 8: Use Glance v2 API to list the images whose visibility is private and
+  store the response details in a list.
+* **Test assertion 7:** All images' values of visibility in the list are private.
+* Test action 9: Delete the 6 images with the stored ids. Show images and check whether
+  the 6 ids are not in the show list.
+* **Test assertion 8:** The stored 6 ids are not found in the show list.
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test case evaluates the ability to use Glance v2 API to list images with
+different parameters. Specifically it verifies that:
+
+* Glance image API can show the images with the container_format.
+* Glance image API can show the images with the disk_format.
+* Glance image API can show the images by setting a limit number.
+* Glance image API can show the images with the size_min and size_max.
+* Glance image API can show the images with the size.
+* Glance image API can show the images with the status.
+* Glance image API can show the images with the visibility type.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+None
+
+------------------------------------------
+Image update tests using the Glance v2 API
+------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_update_image
+tempest.api.image.v2.test_images_tags.ImagesTagsTest.test_update_delete_tags_for_image
+tempest.api.image.v2.test_images_tags_negative.ImagesTagsNegativeTest.test_update_tags_for_non_existing_image
+
+Test preconditions
+------------------
+
+Glance is available.
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create an image with container_formats 'ami', disk_formats 'ami'
+  and visibility 'private' and store its id returned in the response. Check whether
+  the status of the created image is 'queued'.
+* **Test assertion 1:** The status of the created image is 'queued'.
+* Test action 2: Use the stored image id to upload the binary image data and update
+  this image's name. Show this image with the stored id. Check if the stored id and
+  name used to update the image are equal to the id and name in the show list.
+* **Test assertion 2:** The id and name returned in the show list are equal to
+  the stored id and name used to update the image.
+* Test action 3: Create an image with container_formats 'bare', disk_formats 'raw'
+  and visibility 'private' and store its id returned in the response.
+* Test action 4: Use the stored id to add a tag. Show the image with the stored id
+  and check if the tag used to add is in the image's tags returned in the show list.
+* **Test assertion 3:** The tag used to add into the image is in the show list.
+* Test action 5: Use the stored id to delete this tag. Show the image with the
+  stored id and check if the tag used to delete is not in the show list.
+* **Test assertion 4:** The tag used to delete from the image is not in the show list.
+* Test action 6: Generate a random uuid as the image id. Use the image id to add a tag
+  into the image's tags.
+* **Test assertion 5:** The API's response code is 404.
+* Test action 7: Delete the images created in test action 1 and 3. Show the images
+  and check whether the ids are not in the show list.
+* **Test assertion 6:** The two ids are not found in the show list.
+
+Pass / fail criteria
+''''''''''''''''''''
+
+This test case evaluates the ability to use Glance v2 API to update images with
+different parameters. Specifically it verifies that:
+
+* Glance image API can update image's name with the existing image id.
+* Glance image API can update image's tags with the existing image id.
+* Glance image API can't update image's tags with a non-existing image id.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+None
+
+--------------------------------------------
+Image deletion tests using the Glance v2 API
+--------------------------------------------
+
+Test case specification
+-----------------------
+
+tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_delete_image
 tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_delete_image_null_id
 tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_delete_non_existing_image
-tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_delete_deleted_image
-tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_image_null_id
-tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_non_existent_image
-tempest.api.image.v2.test_images_tags.ImagesTagsTest.test_update_delete_tags_for_image
 tempest.api.image.v2.test_images_tags_negative.ImagesTagsNegativeTest.test_delete_non_existing_tag
-tempest.api.image.v2.test_images_tags_negative.ImagesTagsNegativeTest.test_update_tags_for_non_existing_image
+
+Test preconditions
+------------------
+
+Glance is available.
+
+Basic test flow execution description and pass/fail criteria
+------------------------------------------------------------
+
+Test execution
+''''''''''''''
+
+* Test action 1: Create an image with container_formats 'ami', disk_formats 'ami'
+  and visibility 'private'. Use the id of the created image to delete the image.
+  List all images and check whether this id is in the list.
+* **Test assertion 1:** The id of the created image is not found in the list
+  of all images after the deletion operation.
+* Test action 2: Delete images with a null id and check the API's response code.
+* **Test assertion 2:** The API's response code is 404.
+* Test action 3: Generate a random uuid and delete images with this uuid as image id.
+  Check the API's response code.
+* **Test assertion 3:** The API's response code is 404.
+* Test action 4: Create an image with container_formats 'bare', disk_formats 'raw'
+  and visibility 'private'. Delete this image's tag with the image id and a random tag
+  Check the API's response code.
+* **Test assertion 4:** The API's response code is 404.
+* Test action 5: Delete the images created in test action 1 and 4. List all images
+  and check whether the ids are in the list.
+* **Test assertion 5:** The two ids are not found in the list.
+
+Pass / fail criteria
+''''''''''''''''''''
+
+The first three test cases evaluate the ability to use Glance v2 API to delete images
+with an existing image id, a null image id and a non-existing image id. The last one
+evaluates the ability to use the API to delete a non-existing image tag.
+Specifically it verifies that:
+
+* Glance image deletion API can delete the image with an existing id.
+* Glance image deletion API can't delete an image with a null image id.
+* Glance image deletion API can't delete an image with a non-existing image id.
+* Glance image deletion API can't delete an image tag with a non-existing image tag.
+
+In order to pass this test, all test assertions listed in the test execution above need to pass.
+
+Post conditions
+---------------
+
+None