aboutsummaryrefslogtreecommitdiffstats
path: root/docs/testing
diff options
context:
space:
mode:
Diffstat (limited to 'docs/testing')
-rwxr-xr-xdocs/testing/developer/devguide/devguide.rst140
-rw-r--r--docs/testing/user/userguide/glossary.rst75
2 files changed, 202 insertions, 13 deletions
diff --git a/docs/testing/developer/devguide/devguide.rst b/docs/testing/developer/devguide/devguide.rst
index 91f2c2148..4fe01c12b 100755
--- a/docs/testing/developer/devguide/devguide.rst
+++ b/docs/testing/developer/devguide/devguide.rst
@@ -449,6 +449,10 @@ Verify your patch::
It is used in CI but also by the CLI.
+For more details on ``tox`` and tests, please refer to the `Running tests`_
+and `working with tox`_ sections below, which describe the different available
+environments.
+
Submit the code with Git
++++++++++++++++++++++++
@@ -566,6 +570,142 @@ The process for backporting is as follows:
A backported change needs a ``+1`` and a ``+2`` from a committer who didn’t
propose the change (i.e. minimum 3 people involved).
+Development guidelines
+----------------------
+This section provides guidelines and best practices for feature development
+and bug fixing in Yardstick.
+
+In general, bug fixes should be submitted as a single patch.
+
+When developing larger features, all commits on the local topic branch can be
+submitted together, by running ``git review`` on the tip of the branch. This
+creates a chain of related patches in gerrit.
+
+Each commit should contain one logical change and the author should aim for no
+more than 300 lines of code per commit. This helps to make the changes easier
+to review.
+
+Each feature should have the following:
+
+* Feature/bug fix code
+* Unit tests (both positive and negative)
+* Functional tests (optional)
+* Sample testcases (if applicable)
+* Documentation
+* Update to release notes
+
+Coding style
+~~~~~~~~~~~~
+.. _`OpenStack Style Guidelines`: https://docs.openstack.org/hacking/latest/user/hacking.html
+.. _`OPNFV coding guidelines`: https://wiki.opnfv.org/display/DEV/Contribution+Guidelines
+
+Please follow the `OpenStack Style Guidelines`_ for code contributions (the
+section on Internationalization (i18n) Strings is not applicable).
+
+When writing commit message, the `OPNFV coding guidelines`_ on git commit
+message style should also be used.
+
+Running tests
+~~~~~~~~~~~~~
+Once your patch has been submitted, a number of tests will be run by Jenkins
+CI to verify the patch. Before submitting your patch, you should run these
+tests locally. You can do this using ``tox``, which has a number of different
+test environments defined in ``tox.ini``.
+Calling ``tox`` without any additional arguments runs the default set of
+tests (unit tests, functional tests, coverage and pylint).
+
+If some tests are failing, you can save time and select test environments
+individually, by passing one or more of the following command-line options to
+``tox``:
+
+* ``-e py27``: Unit tests using Python 2.7
+* ``-e py3``: Unit tests using Python 3
+* ``-e pep8``: Linter and style checks on updated files
+* ``-e functional``: Functional tests using Python 2.7
+* ``-e functional-py3``: Functional tests using Python 3
+* ``-e coverage``: Code coverage checks
+
+.. note:: You need to stage your changes prior to running coverage for those
+ changes to be checked.
+
+In addition to the tests run by Jenkins (listed above), there are a number of
+other test environments defined.
+
+* ``-e pep8-full``: Linter and style checks are run on the whole repo (not
+ just on updated files)
+* ``-e os-requirements``: Check that the requirements are compatible with
+ OpenStack requirements.
+
+Working with tox
+++++++++++++++++
+.. _virtualenv: https://virtualenv.pypa.io/en/stable/
+
+``tox`` uses `virtualenv`_ to create isolated Python environments to run the
+tests in. The test environments are located at
+``.tox/<environment_name>`` e.g. ``.tox/py27``.
+
+If requirements are changed, you will need to recreate the tox test
+environment to make sure the new requirements are installed. This is done by
+passing the additional ``-r`` command-line option to ``tox``::
+
+ tox -r -e ...
+
+This can also be achieved by deleting the test environments manually before
+running ``tox``::
+
+ rm -rf .tox/<environment_name>
+ rm -rf .tox/py27
+
+Writing unit tests
+~~~~~~~~~~~~~~~~~~
+For each change submitted, a set of unit tests should be submitted, which
+should include both positive and negative testing.
+
+In order to help identify which tests are needed, follow the guidelines below.
+
+* In general, there should be a separate test for each branching point, return
+ value and input set.
+* Negative tests should be written to make sure exceptions are raised and/or
+ handled appropriately.
+
+The following convention should be used for naming tests::
+
+ test_<method_name>_<some_comment>
+
+The comment gives more information on the nature of the test, the side effect
+being checked, or the parameter being modified::
+
+ test_my_method_runtime_error
+ test_my_method_invalid_credentials
+ test_my_method_param1_none
+
+Mocking
++++++++
+The ``mock`` library is used for unit testing to stub out external libraries.
+
+The following conventions are used in Yardstick:
+
+* Use ``mock.patch.object`` instead of ``mock.patch``.
+
+* When naming mocked classes/functions, use ``mock_<class_and_function_name>``
+ e.g. ``mock_subprocess_call``
+
+* Avoid decorating classes with mocks. Apply the mocking in ``setUp()``::
+
+ @mock.patch.object(ssh, 'SSH')
+ class MyClassTestCase(unittest.TestCase):
+
+ should be::
+
+ class MyClassTestCase(unittest.TestCase):
+ def setUp(self):
+ self._mock_ssh = mock.patch.object(ssh, 'SSH')
+ self.mock_ssh = self._mock_ssh.start()
+
+ self.addCleanup(self._stop_mocks)
+
+ def _stop_mocks(self):
+ self._mock_ssh.stop()
Plugins
-------
diff --git a/docs/testing/user/userguide/glossary.rst b/docs/testing/user/userguide/glossary.rst
index be98aa6c0..6a153943c 100644
--- a/docs/testing/user/userguide/glossary.rst
+++ b/docs/testing/user/userguide/glossary.rst
@@ -13,6 +13,11 @@ Glossary
API
Application Programming Interface
+ Docker
+ Docker provisions and manages containers. Yardstick and many other OPNFV
+ projects are deployed in containers. Docker is required to launch the
+ containerized versions of these projects.
+
DPI
Deep Packet Inspection
@@ -27,36 +32,80 @@ Glossary
IOPS
Input/Output Operations Per Second
+ A performance measurement used to benchmark storage devices.
+
+ KPI
+ Key Performance Indicator
+
+ Kubernetes
+ k8s
+ Kubernetes is an open-source container-orchestration system for automating
+ deployment, scaling and management of containerized applications.
+ It is one of the contexts supported in Yardstick.
+
+ NFV
+ Network Function Virtualization
+ NFV is an initiative to take network services which were traditionally run
+ on proprietary, dedicated hardware, and virtualize them to run on general
+ purpose hardware.
+
+ NFVI
+ Network Function Virtualization Infrastructure
+ The servers, routers, switches, etc on which the NFV system runs.
NIC
Network Interface Controller
+ OpenStack
+ OpenStack is a cloud operating system that controls pools of compute,
+ storage, and networking resources. OpenStack is an open source project
+ licensed under the Apache License 2.0.
+
PBFS
Packet Based per Flow State
+ PROX
+ Packet pROcessing eXecution engine
+
QoS
Quality of Service
+ The ability to guarantee certain network or storage requirements to
+ satisfy a Service Level Agreement (SLA) between an application provider
+ and end users.
+ Typically includes performance requirements like networking bandwidth,
+ latency, jitter correction, and reliability as well as storage
+ performance in Input/Output Operations Per Second (IOPS), throttling
+ agreements, and performance expectations at peak load
+
+ SLA
+ Service Level Agreement
+ An SLA is an agreement between a service provider and a customer to
+ provide a certain level of service/performance.
+
+ SR-IOV
+ Single Root IO Virtualization
+ A specification that, when implemented by a physical PCIe
+ device, enables it to appear as multiple separate PCIe devices. This
+ enables multiple virtualized guests to share direct access to the
+ physical device.
+
+ SUT
+ System Under Test
+
+ ToS
+ Type of Service
VLAN
- Virtual LAN
+ Virtual LAN (Local Area Network)
VM
Virtual Machine
+ An operating system instance that runs on top of a hypervisor.
+ Multiple VMs can run at the same time on the same physical
+ host.
VNF
Virtual Network Function
VNFC
Virtual Network Function Component
-
- NFVI
- Network Function Virtualization Infrastructure
-
- SR-IOV
- Single Root IO Virtualization
-
- SUT
- System Under Test
-
- ToS
- Type of Service