summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorCedric Ollivier <cedric.ollivier@orange.com>2018-04-20 07:55:40 +0000
committerGerrit Code Review <gerrit@opnfv.org>2018-04-20 07:55:40 +0000
commit62a8c30ec6ddeff54bb91d54428caa0831bfd38e (patch)
tree0f228a56e4b32e19cf20f4cda7259f6017241ed4
parentf97acea68b7e4be6f4c1f56214c3e2445cba5360 (diff)
parent4b18074ae39972ec9674e4d80d8392f06e6df61a (diff)
Merge "Update test results in userguide"
-rw-r--r--docs/testing/user/userguide/test_details.rst70
-rw-r--r--docs/testing/user/userguide/test_results.rst101
2 files changed, 106 insertions, 65 deletions
diff --git a/docs/testing/user/userguide/test_details.rst b/docs/testing/user/userguide/test_details.rst
index 935298fdd..002b3b7e6 100644
--- a/docs/testing/user/userguide/test_details.rst
+++ b/docs/testing/user/userguide/test_details.rst
@@ -302,39 +302,43 @@ Features
Functest has been supporting several feature projects since Brahmaputra:
-+-----------------+---------+----------+--------+-----------+
-| Test | Brahma | Colorado | Danube | Euphrates |
-+=================+=========+==========+========+===========+
-| barometer | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| bgpvpn | | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| copper | | X | | |
-+-----------------+---------+----------+--------+-----------+
-| doctor | X | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| domino | | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| fds | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| moon | | X | | |
-+-----------------+---------+----------+--------+-----------+
-| multisite | | X | X | |
-+-----------------+---------+----------+--------+-----------+
-| netready | | | X | |
-+-----------------+---------+----------+--------+-----------+
-| odl_sfc | | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| opera | | | X | |
-+-----------------+---------+----------+--------+-----------+
-| orchestra | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| parser | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| promise | X | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| security_scan | | X | X | |
-+-----------------+---------+----------+--------+-----------+
++-----------------+---------+----------+--------+-----------+-----------+
+| Test | Brahma | Colorado | Danube | Euphrates | Fraser |
++=================+=========+==========+========+===========+===========+
+| barometer | | | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| bgpvpn | | X | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| copper | | X | | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| doctor | X | X | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| domino | | X | X | X | |
++-----------------+---------+----------+--------+-----------+-----------+
+| fds | | | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| moon | | X | | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| multisite | | X | X | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| netready | | | X | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| odl_sfc | | X | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| opera | | | X | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| orchestra | | | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| parser | | | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| promise | X | X | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| security_scan | | X | X | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| clover | | | | | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| stor4nfv | | | | | X |
++-----------------+---------+----------+--------+-----------+-----------+
Please refer to the dedicated feature user guides for details.
diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst
index c215e05ec..bb28989ae 100644
--- a/docs/testing/user/userguide/test_results.rst
+++ b/docs/testing/user/userguide/test_results.rst
@@ -10,42 +10,79 @@ In manual mode test results are displayed in the console and result files
are put in /home/opnfv/functest/results.
If you want additional logs, you may configure the logging.ini under
-/usr/lib/python2.7/site-packages/functest/ci.
+/usr/lib/python2.7/site-packages/xtesting/ci.
Automated testing
-----------------
-In automated mode, test results are displayed in jenkins logs, a summary is
-provided at the end of the job and can be described as follow::
-
- +-------------------------+----------------------------------------------------------+
- | ENV VAR | VALUE |
- +-------------------------+----------------------------------------------------------+
- | INSTALLER_TYPE | daisy |
- | DEPLOY_SCENARIO | os-nosdn-nofeature-ha |
- | BUILD_TAG | jenkins-functest-daisy-baremetal-daily-master-67 |
- | CI_LOOP | daily |
- +-------------------------+----------------------------------------------------------+
-
-::
-
- +------------------------------+------------------+---------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +------------------------------+------------------+---------------------+------------------+----------------+
- | connection_check | functest | healthcheck | 00:08 | PASS |
- | api_check | functest | healthcheck | 04:22 | PASS |
- | snaps_health_check | functest | healthcheck | 00:35 | PASS |
- | vping_ssh | functest | smoke | 00:54 | PASS |
- | vping_userdata | functest | smoke | 00:27 | PASS |
- | tempest_smoke_serial | functest | smoke | 19:39 | FAIL |
- | rally_sanity | functest | smoke | 15:16 | PASS |
- | refstack_defcore | functest | smoke | 15:55 | PASS |
- | snaps_smoke | functest | smoke | 26:45 | FAIL |
- | cloudify_ims | functest | vnf | 23:56 | PASS |
- | orchestra_openims | orchestra | vnf | 15:07 | PASS |
- | orchestra_clearwaterims | orchestra | vnf | 19:10 | PASS |
- | vyos_vrouter | functest | vnf | 00:00 | SKIP |
- +------------------------------+------------------+---------------------+------------------+----------------+
+In automated mode, tests are run within split Alpine containers, and test
+results are displayed in jenkins logs. The result summary is provided at the
+end of each suite and can be described as follow.
+
+Healthcheck suite::
+
+ +----------------------------+------------------+---------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +----------------------------+------------------+---------------------+------------------+----------------+
+ | connection_check | functest | healthcheck | 00:07 | PASS |
+ | api_check | functest | healthcheck | 07:46 | PASS |
+ | snaps_health_check | functest | healthcheck | 00:36 | PASS |
+ +----------------------------+------------------+---------------------+------------------+----------------+
+
+Smoke suite::
+
+ +------------------------------+------------------+---------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +------------------------------+------------------+---------------+------------------+----------------+
+ | vping_ssh | functest | smoke | 00:57 | PASS |
+ | vping_userdata | functest | smoke | 00:33 | PASS |
+ | tempest_smoke_serial | functest | smoke | 13:22 | PASS |
+ | rally_sanity | functest | smoke | 24:07 | PASS |
+ | refstack_defcore | functest | smoke | 05:21 | PASS |
+ | patrole | functest | smoke | 04:29 | PASS |
+ | snaps_smoke | functest | smoke | 46:54 | PASS |
+ | odl | functest | smoke | 00:00 | SKIP |
+ | neutron_trunk | functest | smoke | 00:00 | SKIP |
+ +------------------------------+------------------+---------------+------------------+----------------+
+
+Features suite::
+
+ +-----------------------------+------------------------+------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-----------------------------+------------------------+------------------+------------------+----------------+
+ | doctor-notification | doctor | features | 00:00 | SKIP |
+ | bgpvpn | sdnvpn | features | 00:00 | SKIP |
+ | functest-odl-sfc | sfc | features | 00:00 | SKIP |
+ | barometercollectd | barometer | features | 00:00 | SKIP |
+ | fds | fastdatastacks | features | 00:00 | SKIP |
+ +-----------------------------+------------------------+------------------+------------------+----------------+
+
+Components suite::
+
+ +-------------------------------+------------------+--------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-------------------------------+------------------+--------------------+------------------+----------------+
+ | tempest_full_parallel | functest | components | 48:28 | PASS |
+ | rally_full | functest | components | 126:02 | PASS |
+ +-------------------------------+------------------+--------------------+------------------+----------------+
+
+Vnf suite::
+
+ +----------------------+------------------+--------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +----------------------+------------------+--------------+------------------+----------------+
+ | cloudify_ims | functest | vnf | 28:15 | PASS |
+ | vyos_vrouter | functest | vnf | 17:59 | PASS |
+ | juju_epc | functest | vnf | 46:44 | PASS |
+ +----------------------+------------------+--------------+------------------+----------------+
+
+Parser testcase::
+
+ +-----------------------+-----------------+------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-----------------------+-----------------+------------------+------------------+----------------+
+ | parser-basics | parser | features | 00:00 | SKIP |
+ +-----------------------+-----------------+------------------+------------------+----------------+
Functest Kubernetes test result::