Merge "Update test results in userguide"

2 files changed, 106 insertions, 65 deletions

diff --git a/docs/testing/user/userguide/test_details.rst b/docs/testing/user/userguide/test_details.rst
index 935298fdd..002b3b7e6 100644
--- a/docs/testing/user/userguide/test_details.rst
+++ b/docs/testing/user/userguide/test_details.rst
@@ -302,39 +302,43 @@ Features
 Functest has been supporting several feature projects since Brahmaputra:
 
 
-+-----------------+---------+----------+--------+-----------+
-| Test            | Brahma  | Colorado | Danube | Euphrates |
-+=================+=========+==========+========+===========+
-| barometer       |         |          |    X   |     X     |
-+-----------------+---------+----------+--------+-----------+
-| bgpvpn          |         |    X     |    X   |     X     |
-+-----------------+---------+----------+--------+-----------+
-| copper          |         |    X     |        |           |
-+-----------------+---------+----------+--------+-----------+
-| doctor          |    X    |    X     |    X   |     X     |
-+-----------------+---------+----------+--------+-----------+
-| domino          |         |    X     |    X   |     X     |
-+-----------------+---------+----------+--------+-----------+
-| fds             |         |          |    X   |     X     |
-+-----------------+---------+----------+--------+-----------+
-| moon            |         |    X     |        |           |
-+-----------------+---------+----------+--------+-----------+
-| multisite       |         |    X     |    X   |           |
-+-----------------+---------+----------+--------+-----------+
-| netready        |         |          |    X   |           |
-+-----------------+---------+----------+--------+-----------+
-| odl_sfc         |         |    X     |    X   |     X     |
-+-----------------+---------+----------+--------+-----------+
-| opera           |         |          |    X   |           |
-+-----------------+---------+----------+--------+-----------+
-| orchestra       |         |          |    X   |     X     |
-+-----------------+---------+----------+--------+-----------+
-| parser          |         |          |    X   |     X     |
-+-----------------+---------+----------+--------+-----------+
-| promise         |    X    |    X     |    X   |     X     |
-+-----------------+---------+----------+--------+-----------+
-| security_scan   |         |    X     |    X   |           |
-+-----------------+---------+----------+--------+-----------+
++-----------------+---------+----------+--------+-----------+-----------+
+| Test            | Brahma  | Colorado | Danube | Euphrates |  Fraser   |
++=================+=========+==========+========+===========+===========+
+| barometer       |         |          |    X   |     X     |     X     |
++-----------------+---------+----------+--------+-----------+-----------+
+| bgpvpn          |         |    X     |    X   |     X     |     X     |
++-----------------+---------+----------+--------+-----------+-----------+
+| copper          |         |    X     |        |           |           |
++-----------------+---------+----------+--------+-----------+-----------+
+| doctor          |    X    |    X     |    X   |     X     |     X     |
++-----------------+---------+----------+--------+-----------+-----------+
+| domino          |         |    X     |    X   |     X     |           |
++-----------------+---------+----------+--------+-----------+-----------+
+| fds             |         |          |    X   |     X     |     X     |
++-----------------+---------+----------+--------+-----------+-----------+
+| moon            |         |    X     |        |           |           |
++-----------------+---------+----------+--------+-----------+-----------+
+| multisite       |         |    X     |    X   |           |           |
++-----------------+---------+----------+--------+-----------+-----------+
+| netready        |         |          |    X   |           |           |
++-----------------+---------+----------+--------+-----------+-----------+
+| odl_sfc         |         |    X     |    X   |     X     |     X     |
++-----------------+---------+----------+--------+-----------+-----------+
+| opera           |         |          |    X   |           |           |
++-----------------+---------+----------+--------+-----------+-----------+
+| orchestra       |         |          |    X   |     X     |     X     |
++-----------------+---------+----------+--------+-----------+-----------+
+| parser          |         |          |    X   |     X     |     X     |
++-----------------+---------+----------+--------+-----------+-----------+
+| promise         |    X    |    X     |    X   |     X     |     X     |
++-----------------+---------+----------+--------+-----------+-----------+
+| security_scan   |         |    X     |    X   |           |           |
++-----------------+---------+----------+--------+-----------+-----------+
+| clover          |         |          |        |           |     X     |
++-----------------+---------+----------+--------+-----------+-----------+
+| stor4nfv        |         |          |        |           |     X     |
++-----------------+---------+----------+--------+-----------+-----------+
 
 Please refer to the dedicated feature user guides for details.
 
diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst
index c215e05ec..bb28989ae 100644
--- a/docs/testing/user/userguide/test_results.rst
+++ b/docs/testing/user/userguide/test_results.rst
@@ -10,42 +10,79 @@ In manual mode test results are displayed in the console and result files
 are put in /home/opnfv/functest/results.
 
 If you want additional logs, you may configure the logging.ini under
-/usr/lib/python2.7/site-packages/functest/ci.
+/usr/lib/python2.7/site-packages/xtesting/ci.
 
 Automated testing
 -----------------
 
-In automated mode, test results are displayed in jenkins logs, a summary is
-provided at the end of the job and can be described as follow::
-
- +-------------------------+----------------------------------------------------------+
- |         ENV VAR         |                          VALUE                           |
- +-------------------------+----------------------------------------------------------+
- |      INSTALLER_TYPE     |                          daisy                           |
- |     DEPLOY_SCENARIO     |                  os-nosdn-nofeature-ha                   |
- |        BUILD_TAG        |     jenkins-functest-daisy-baremetal-daily-master-67     |
- |         CI_LOOP         |                          daily                           |
- +-------------------------+----------------------------------------------------------+
-
-::
-
- +------------------------------+------------------+---------------------+------------------+----------------+
- |          TEST CASE           |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
- +------------------------------+------------------+---------------------+------------------+----------------+
- |       connection_check       |     functest     |     healthcheck     |      00:08       |      PASS      |
- |          api_check           |     functest     |     healthcheck     |      04:22       |      PASS      |
- |      snaps_health_check      |     functest     |     healthcheck     |      00:35       |      PASS      |
- |          vping_ssh           |     functest     |        smoke        |      00:54       |      PASS      |
- |        vping_userdata        |     functest     |        smoke        |      00:27       |      PASS      |
- |     tempest_smoke_serial     |     functest     |        smoke        |      19:39       |      FAIL      |
- |         rally_sanity         |     functest     |        smoke        |      15:16       |      PASS      |
- |       refstack_defcore       |     functest     |        smoke        |      15:55       |      PASS      |
- |         snaps_smoke          |     functest     |        smoke        |      26:45       |      FAIL      |
- |           cloudify_ims       |     functest     |         vnf         |      23:56       |      PASS      |
- |        orchestra_openims     |     orchestra    |         vnf         |      15:07       |      PASS      |
- |     orchestra_clearwaterims  |     orchestra    |         vnf         |      19:10       |      PASS      |
- |           vyos_vrouter       |     functest     |         vnf         |      00:00       |      SKIP      |
- +------------------------------+------------------+---------------------+------------------+----------------+
+In automated mode, tests are run within split Alpine containers, and test
+results are displayed in jenkins logs. The result summary is provided at the
+end of each suite and can be described as follow.
+
+Healthcheck suite::
+
+ +----------------------------+------------------+---------------------+------------------+----------------+
+ |         TEST CASE          |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
+ +----------------------------+------------------+---------------------+------------------+----------------+
+ |      connection_check      |     functest     |     healthcheck     |      00:07       |      PASS      |
+ |         api_check          |     functest     |     healthcheck     |      07:46       |      PASS      |
+ |     snaps_health_check     |     functest     |     healthcheck     |      00:36       |      PASS      |
+ +----------------------------+------------------+---------------------+------------------+----------------+
+
+Smoke suite::
+
+ +------------------------------+------------------+---------------+------------------+----------------+
+ |          TEST CASE           |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
+ +------------------------------+------------------+---------------+------------------+----------------+
+ |          vping_ssh           |     functest     |     smoke     |      00:57       |      PASS      |
+ |        vping_userdata        |     functest     |     smoke     |      00:33       |      PASS      |
+ |     tempest_smoke_serial     |     functest     |     smoke     |      13:22       |      PASS      |
+ |         rally_sanity         |     functest     |     smoke     |      24:07       |      PASS      |
+ |       refstack_defcore       |     functest     |     smoke     |      05:21       |      PASS      |
+ |           patrole            |     functest     |     smoke     |      04:29       |      PASS      |
+ |         snaps_smoke          |     functest     |     smoke     |      46:54       |      PASS      |
+ |             odl              |     functest     |     smoke     |      00:00       |      SKIP      |
+ |        neutron_trunk         |     functest     |     smoke     |      00:00       |      SKIP      |
+ +------------------------------+------------------+---------------+------------------+----------------+
+
+Features suite::
+
+ +-----------------------------+------------------------+------------------+------------------+----------------+
+ |          TEST CASE          |        PROJECT         |       TIER       |     DURATION     |     RESULT     |
+ +-----------------------------+------------------------+------------------+------------------+----------------+
+ |     doctor-notification     |         doctor         |     features     |      00:00       |      SKIP      |
+ |            bgpvpn           |         sdnvpn         |     features     |      00:00       |      SKIP      |
+ |       functest-odl-sfc      |          sfc           |     features     |      00:00       |      SKIP      |
+ |      barometercollectd      |       barometer        |     features     |      00:00       |      SKIP      |
+ |             fds             |     fastdatastacks     |     features     |      00:00       |      SKIP      |
+ +-----------------------------+------------------------+------------------+------------------+----------------+
+
+Components suite::
+
+ +-------------------------------+------------------+--------------------+------------------+----------------+
+ |           TEST CASE           |     PROJECT      |        TIER        |     DURATION     |     RESULT     |
+ +-------------------------------+------------------+--------------------+------------------+----------------+
+ |     tempest_full_parallel     |     functest     |     components     |      48:28       |      PASS      |
+ |           rally_full          |     functest     |     components     |      126:02      |      PASS      |
+ +-------------------------------+------------------+--------------------+------------------+----------------+
+
+Vnf suite::
+
+ +----------------------+------------------+--------------+------------------+----------------+
+ |      TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
+ +----------------------+------------------+--------------+------------------+----------------+
+ |     cloudify_ims     |     functest     |     vnf      |      28:15       |      PASS      |
+ |     vyos_vrouter     |     functest     |     vnf      |      17:59       |      PASS      |
+ |       juju_epc       |     functest     |     vnf      |      46:44       |      PASS      |
+ +----------------------+------------------+--------------+------------------+----------------+
+
+Parser testcase::
+
+ +-----------------------+-----------------+------------------+------------------+----------------+
+ |       TEST CASE       |     PROJECT     |       TIER       |     DURATION     |     RESULT     |
+ +-----------------------+-----------------+------------------+------------------+----------------+
+ |     parser-basics     |      parser     |     features     |      00:00       |      SKIP      |
+ +-----------------------+-----------------+------------------+------------------+----------------+
 
 Functest Kubernetes test result::