summaryrefslogtreecommitdiffstats
path: root/docs
diff options
context:
space:
mode:
Diffstat (limited to 'docs')
-rw-r--r--docs/testing/user/userguide/test_details.rst98
-rw-r--r--docs/testing/user/userguide/test_results.rst137
2 files changed, 170 insertions, 65 deletions
diff --git a/docs/testing/user/userguide/test_details.rst b/docs/testing/user/userguide/test_details.rst
index fe7e6a9b6..002b3b7e6 100644
--- a/docs/testing/user/userguide/test_details.rst
+++ b/docs/testing/user/userguide/test_details.rst
@@ -302,39 +302,43 @@ Features
Functest has been supporting several feature projects since Brahmaputra:
-+-----------------+---------+----------+--------+-----------+
-| Test | Brahma | Colorado | Danube | Euphrates |
-+=================+=========+==========+========+===========+
-| barometer | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| bgpvpn | | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| copper | | X | | |
-+-----------------+---------+----------+--------+-----------+
-| doctor | X | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| domino | | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| fds | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| moon | | X | | |
-+-----------------+---------+----------+--------+-----------+
-| multisite | | X | X | |
-+-----------------+---------+----------+--------+-----------+
-| netready | | | X | |
-+-----------------+---------+----------+--------+-----------+
-| odl_sfc | | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| opera | | | X | |
-+-----------------+---------+----------+--------+-----------+
-| orchestra | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| parser | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| promise | X | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| security_scan | | X | X | |
-+-----------------+---------+----------+--------+-----------+
++-----------------+---------+----------+--------+-----------+-----------+
+| Test | Brahma | Colorado | Danube | Euphrates | Fraser |
++=================+=========+==========+========+===========+===========+
+| barometer | | | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| bgpvpn | | X | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| copper | | X | | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| doctor | X | X | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| domino | | X | X | X | |
++-----------------+---------+----------+--------+-----------+-----------+
+| fds | | | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| moon | | X | | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| multisite | | X | X | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| netready | | | X | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| odl_sfc | | X | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| opera | | | X | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| orchestra | | | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| parser | | | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| promise | X | X | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| security_scan | | X | X | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| clover | | | | | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| stor4nfv | | | | | X |
++-----------------+---------+----------+--------+-----------+-----------+
Please refer to the dedicated feature user guides for details.
@@ -419,6 +423,33 @@ The vyos-vrouter architecture is described in `[14]`_
juju_epc
^^^^^^^^
+
+Kubernetes (K8s)
+----------------
+
+Kubernetes testing relies on sets of tests, which are part of the Kubernetes
+source tree, such as the Kubernetes End-to-End (e2e) tests `[15]`_.
+
+The kubernetes testcases are distributed across various Tiers:
+
+ * Healthcheck Tier
+
+ * k8s_smoke Test Case: Creates a Guestbook application that contains redis
+ server, 2 instances of redis slave, frontend application, frontend service
+ and redis master service and redis slave service. Using frontend service,
+ the test will write an entry into the guestbook application which will
+ store the entry into the backend redis database. Application flow MUST
+ work as expected and the data written MUST be available to read.
+
+ * Smoke Tier
+
+ * k8s_conformance Test Case: Runs a series of k8s e2e tests expected to
+ pass on any Kubernetes cluster. It is a subset of tests necessary to
+ demonstrate conformance grows with each release. Conformance is thus
+ considered versioned, with backwards compatibility guarantees and are
+ designed to be run with no cloud provider configured.
+
+
.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
.. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml
@@ -429,3 +460,4 @@ juju_epc
.. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html
.. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO
.. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter
+.. _`[15]`: https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md
diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst
index 2d36bdc33..bb28989ae 100644
--- a/docs/testing/user/userguide/test_results.rst
+++ b/docs/testing/user/userguide/test_results.rst
@@ -10,42 +10,115 @@ In manual mode test results are displayed in the console and result files
are put in /home/opnfv/functest/results.
If you want additional logs, you may configure the logging.ini under
-/usr/lib/python2.7/site-packages/functest/ci.
+/usr/lib/python2.7/site-packages/xtesting/ci.
Automated testing
-----------------
-In automated mode, test results are displayed in jenkins logs, a summary is
-provided at the end of the job and can be described as follow::
-
- +-------------------------+----------------------------------------------------------+
- | ENV VAR | VALUE |
- +-------------------------+----------------------------------------------------------+
- | INSTALLER_TYPE | daisy |
- | DEPLOY_SCENARIO | os-nosdn-nofeature-ha |
- | BUILD_TAG | jenkins-functest-daisy-baremetal-daily-master-67 |
- | CI_LOOP | daily |
- +-------------------------+----------------------------------------------------------+
-
-::
-
- +------------------------------+------------------+---------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +------------------------------+------------------+---------------------+------------------+----------------+
- | connection_check | functest | healthcheck | 00:08 | PASS |
- | api_check | functest | healthcheck | 04:22 | PASS |
- | snaps_health_check | functest | healthcheck | 00:35 | PASS |
- | vping_ssh | functest | smoke | 00:54 | PASS |
- | vping_userdata | functest | smoke | 00:27 | PASS |
- | tempest_smoke_serial | functest | smoke | 19:39 | FAIL |
- | rally_sanity | functest | smoke | 15:16 | PASS |
- | refstack_defcore | functest | smoke | 15:55 | PASS |
- | snaps_smoke | functest | smoke | 26:45 | FAIL |
- | cloudify_ims | functest | vnf | 23:56 | PASS |
- | orchestra_openims | orchestra | vnf | 15:07 | PASS |
- | orchestra_clearwaterims | orchestra | vnf | 19:10 | PASS |
- | vyos_vrouter | functest | vnf | 00:00 | SKIP |
- +------------------------------+------------------+---------------------+------------------+----------------+
+In automated mode, tests are run within split Alpine containers, and test
+results are displayed in jenkins logs. The result summary is provided at the
+end of each suite and can be described as follow.
+
+Healthcheck suite::
+
+ +----------------------------+------------------+---------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +----------------------------+------------------+---------------------+------------------+----------------+
+ | connection_check | functest | healthcheck | 00:07 | PASS |
+ | api_check | functest | healthcheck | 07:46 | PASS |
+ | snaps_health_check | functest | healthcheck | 00:36 | PASS |
+ +----------------------------+------------------+---------------------+------------------+----------------+
+
+Smoke suite::
+
+ +------------------------------+------------------+---------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +------------------------------+------------------+---------------+------------------+----------------+
+ | vping_ssh | functest | smoke | 00:57 | PASS |
+ | vping_userdata | functest | smoke | 00:33 | PASS |
+ | tempest_smoke_serial | functest | smoke | 13:22 | PASS |
+ | rally_sanity | functest | smoke | 24:07 | PASS |
+ | refstack_defcore | functest | smoke | 05:21 | PASS |
+ | patrole | functest | smoke | 04:29 | PASS |
+ | snaps_smoke | functest | smoke | 46:54 | PASS |
+ | odl | functest | smoke | 00:00 | SKIP |
+ | neutron_trunk | functest | smoke | 00:00 | SKIP |
+ +------------------------------+------------------+---------------+------------------+----------------+
+
+Features suite::
+
+ +-----------------------------+------------------------+------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-----------------------------+------------------------+------------------+------------------+----------------+
+ | doctor-notification | doctor | features | 00:00 | SKIP |
+ | bgpvpn | sdnvpn | features | 00:00 | SKIP |
+ | functest-odl-sfc | sfc | features | 00:00 | SKIP |
+ | barometercollectd | barometer | features | 00:00 | SKIP |
+ | fds | fastdatastacks | features | 00:00 | SKIP |
+ +-----------------------------+------------------------+------------------+------------------+----------------+
+
+Components suite::
+
+ +-------------------------------+------------------+--------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-------------------------------+------------------+--------------------+------------------+----------------+
+ | tempest_full_parallel | functest | components | 48:28 | PASS |
+ | rally_full | functest | components | 126:02 | PASS |
+ +-------------------------------+------------------+--------------------+------------------+----------------+
+
+Vnf suite::
+
+ +----------------------+------------------+--------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +----------------------+------------------+--------------+------------------+----------------+
+ | cloudify_ims | functest | vnf | 28:15 | PASS |
+ | vyos_vrouter | functest | vnf | 17:59 | PASS |
+ | juju_epc | functest | vnf | 46:44 | PASS |
+ +----------------------+------------------+--------------+------------------+----------------+
+
+Parser testcase::
+
+ +-----------------------+-----------------+------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-----------------------+-----------------+------------------+------------------+----------------+
+ | parser-basics | parser | features | 00:00 | SKIP |
+ +-----------------------+-----------------+------------------+------------------+----------------+
+
+Functest Kubernetes test result::
+
+ +--------------------------------------+------------------------------------------------------------+
+ | ENV VAR | VALUE |
+ +--------------------------------------+------------------------------------------------------------+
+ | INSTALLER_TYPE | compass |
+ | DEPLOY_SCENARIO | k8-nosdn-nofeature-ha |
+ | BUILD_TAG | jenkins-functest-compass-baremetal-daily-master-75 |
+ | CI_LOOP | daily |
+ +--------------------------------------+------------------------------------------------------------+
+
+Kubernetes healthcheck suite::
+
+ +-------------------+------------------+---------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-------------------+------------------+---------------------+------------------+----------------+
+ | k8s_smoke | functest | healthcheck | 01:54 | PASS |
+ +-------------------+------------------+---------------------+------------------+----------------+
+
+Kubernetes smoke suite::
+
+ +-------------------------+------------------+---------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-------------------------+------------------+---------------+------------------+----------------+
+ | k8s_conformance | functest | smoke | 57:47 | PASS |
+ +-------------------------+------------------+---------------+------------------+----------------+
+
+Kubernetes features suite::
+
+ +----------------------+------------------+------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +----------------------+------------------+------------------+------------------+----------------+
+ | stor4nfv_k8s | stor4nfv | stor4nfv | 00:00 | SKIP |
+ | clover_k8s | clover | clover | 00:00 | SKIP |
+ +----------------------+------------------+------------------+------------------+----------------+
Results are automatically pushed to the test results database, some additional
result files are pushed to OPNFV artifact web sites.