path: root/docs/testing/user/testspecification/vping/index.rst

.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
.. (c) Ericsson AB

========================
Vping Test Specification
========================

.. toctree::
   :maxdepth: 2

Scope
=====

The vping test area evaluates basic NFVi capabilities of the system under test.
These capabilities include creating a small number of virtual machines,
establishing basic L3 connectivity between them and verifying connectivity by
means of ICMP packets.


References
==========

- OpenStack Shade

  - https://docs.openstack.org/shade/latest/user/index.html

- SSHClient

  - http://docs.paramiko.org/en/2.2/

- SCPClient

  - https://pypi.org/project/scp/


Definitions and Abbreviations
=============================

The following terms and abbreviations are used in conjunction with this test
area

- ICMP - Internet Control Message Protocol
- L3 - Layer 3
- NFVi - Network Functions Virtualization infrastructure
- SCP - Secure CoPy
- SSH - Secure SHell
- VM - Virtual Machine


System Under Test (SUT)
=======================

The system under test is assumed to be the NFVi and VIM in operation on a
Pharos compliant infrastructure.


Test Area Structure
===================

The test area is structured in two separate tests which are executed
sequentially. The order of the tests is arbitrary as there are no dependencies
across the tests.


Test Descriptions
=================

--------------------------------------------------------------------
Test Case 1 - vPing using userdata provided by nova metadata service
--------------------------------------------------------------------

Short name
----------

functest.vping.userdata


Use case specification
----------------------

This test evaluates the use case where an NFVi tenant boots up two VMs and
requires L3 connectivity between those VMs. The target IP is passed to the VM
that will initiate pings by using a custom userdata script provided by nova metadata service.


Test preconditions
------------------

At least one compute node is available. No further pre-configuration needed.


Basic test flow execution description and pass/fail criteria
------------------------------------------------------------

Methodology for verifying connectivity
''''''''''''''''''''''''''''''''''''''

Connectivity between VMs is tested by sending ICMP ping packets between
selected VMs. The target IP is passed to the VM sending pings by using a
custom userdata script by means of the config driver mechanism provided by
Nova metadata service. Whether or not a ping was successful is determined by
checking the console output of the source VMs.


Test execution
''''''''''''''

* **Test action 1:**
    * Create a private tenant network by using neutron client
    * Create one subnet and one router in the network by neutron client
    * Add one interface between the subnet and router
    * Add one gateway route to the router by neutron client
    * Store the network id in the response
* **Test assertion 1:** The network id, subnet id and router id can be found in the response
* **Test action 2:**
    * Create an security group by using neutron client
    * Store the security group id parameter in the response
* **Test assertion 2:** The security group id can be found in the response
* **Test action 3:**
    * Boot VM1 by using nova client with configured name, image, flavor, private tenant
      network created in test action 1 and security group created in test action 2
* **Test assertion 3:** The VM1 object can be found in the response
* **Test action 4:**
    * Generate ping script with the IP of VM1 to be passed as userdata provided by
      the **nova metadata service**
* **Test action 5:**
    * Boot VM2 by using nova client with configured name, image, flavor, private tenant
      network created in test action 1, security group created in test action 2, and
      userdata created in test action 4
* **Test assertion 4:** The VM2 object can be found in the response
* **Test action 6:**
    * Inside VM2, the ping script is executed automatically when booted and it contains a
      loop doing the ping until the return code is 0 or timeout reached
    * For each ping, when the return code is 0, "vPing OK" is printed in the VM2 console-log,
      otherwise, "vPing KO" is printed
    * Monitoring the console-log of VM2 to see the response generated by the script
* **Test assertion 5:** "vPing OK" is detected, when monitoring the console-log in VM2
* **Test action 7:**
    * Delete VM1, VM2
* **Test assertion 6:** VM1 and VM2 are not present in the VM list
* **Test action 8:**
    * Delete security group, gateway, interface, router, subnet and network
* **Test assertion 7:** The security group, gateway, interface, router, subnet and network are
  no longer present in the lists after deleting


Pass / Fail criteria
''''''''''''''''''''

This test evaluates basic NFVi capabilities of the system under test.
Specifically, the test verifies that:

* Neutron client network, subnet, router, interface create commands return valid "id" parameters
  which are shown in the create response message
* Neutron client interface add command to add between subnet and router returns success code
* Neutron client gateway add command to add to router returns success code
* Neutron client security group create command returns valid "id" parameter which is shown in
  the response message
* Nova client VM create command returns valid VM attributes response message
* Nova metadata server can transfer userdata configuration at nova client VM booting time
* Ping command from one VM to the other in same private tenant network returns valid code
* All items created using neutron client or nova client create commands are able to be removed by
  using the returned identifiers

In order to pass this test, all test assertions listed in the test execution
above need to pass.


Post conditions
---------------

None


----------------------------------------------
Test Case 2 - vPing using SSH to a floating IP
----------------------------------------------

Short name
----------

functest.vping.ssh


Use case specification
----------------------

This test evaluates the use case where an NFVi tenant boots up two VMs and requires
L3 connectivity between those VMs. An SSH connection is establised from the host to
a floating IP associated with VM2 and ``ping`` is executed on VM2 with the IP of VM1 as target.


Test preconditions
------------------

At least one compute node is available. There should exist an OpenStack external network
and can assign floating IP.


Basic test flow execution description and pass/fail criteria
------------------------------------------------------------

Methodology for verifying connectivity
''''''''''''''''''''''''''''''''''''''

Connectivity between VMs is tested by sending ICMP ping packets between
selected VMs. To this end, the test establishes an SSH connection from the host
running the test suite to a floating IP associated with VM2 and executes ``ping``
on VM2 with the IP of VM1 as target.


Test execution
''''''''''''''


* **Test action 1:**
    * Create a private tenant network by neutron client
    * Create one subnet and one router are created in the network by using neutron client
    * Create one interface between the subnet and router
    * Add one gateway route to the router by neutron client
    * Store the network id in the response
* **Test assertion 1:** The network id, subnet id and router id can be found in the response
* **Test action 2:**
    * Create an security group by using neutron client
    * Store the security group id parameter in the response
* **Test assertion 2:** The security group id can be found in the response
* **Test action 3:**
    * Boot VM1 by using nova client with configured name, image, flavor, private tenant
      network created in test action 1 and security group created in test action 2
* **Test assertion 3:** The VM1 object can be found in the response
* **Test action 4:**
    * Boot VM2 by using nova client with configured name, image, flavor, private tenant
      network created in test action 1 and security group created in test action 2
* **Test assertion 4:** The VM2 object can be found in the response
* **Test action 5:**
    * Create one floating IP by using neutron client, storing the floating IP address
      returned in the response
* **Test assertion 5:** Floating IP address can be found in the response
* **Test action 6:**
    * Assign the floating IP address created in test action 5 to VM2 by using nova client
* **Test assertion 6:** The assigned floating IP can be found in the VM2 console log file
* **Test action 7:**
    * Establish SSH connection between the test host and VM2 through the floating IP
* **Test assertion 7:** SSH connection between the test host and VM2 is established within
  300 seconds
* **Test action 8:**
    * Copy the Ping script from the test host to VM2 by using SCPClient
* **Test assertion 8:** The Ping script can be found inside VM2
* **Test action 9:**
    * Inside VM2, to execute the Ping script to ping VM1, the Ping script contains a
      loop doing the ping until the return code is 0 or timeout reached
    * For each ping, when the return code is 0, "vPing OK" is printed in the VM2 console-log,
      otherwise, "vPing KO" is printed
    * Monitoring the console-log of VM2 to see the response generated by the script
* **Test assertion 9:** "vPing OK" is detected, when monitoring the console-log in VM2
* **Test action 10:**
    * Delete VM1, VM2
* **Test assertion 10:** VM1 and VM2 are not present in the VM list
* **Test action 11:**
    * Delete floating IP, security group, gateway, interface, router, subnet and network
* **Test assertion 11:** The security group, gateway, interface, router, subnet and network are
  no longer present in the lists after deleting

Pass / Fail criteria
''''''''''''''''''''

This test evaluates basic NFVi capabilities of the system under test.
Specifically, the test verifies that:

* Neutron client network, subnet, router, interface create commands return valid "id" parameters
  which are shown in the create response message
* Neutron client interface add command to add between subnet and router return success code
* Neutron client gateway add command to add to router return success code
* Neutron client security group create command returns valid "id" parameter which is shown in the
  response message
* Nova client VM create command returns valid VM attributes response message
* Neutron client floating IP create command return valid floating IP address
* Nova client add floating IP command returns valid response message
* SSH connection can be established using a floating IP
* Ping command from one VM to another in same private tenant network returns valid code
* All items created using neutron client or nova client create commands are able to be removed by
  using the returned identifiers

In order to pass this test, all test assertions listed in the test execution
above need to pass.


Post conditions
---------------

None