summaryrefslogtreecommitdiffstats
path: root/docs/testing/user/userguide/test_results.rst
blob: 2d36bdc330f555a780a3eadc1f39941f41ee441e (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
.. SPDX-License-Identifier: CC-BY-4.0

Test results
============

Manual testing
--------------

In manual mode test results are displayed in the console and result files
are put in /home/opnfv/functest/results.

If you want additional logs, you may configure the logging.ini under
/usr/lib/python2.7/site-packages/functest/ci.

Automated testing
-----------------

In automated mode, test results are displayed in jenkins logs, a summary is
provided at the end of the job and can be described as follow::

 +-------------------------+----------------------------------------------------------+
 |         ENV VAR         |                          VALUE                           |
 +-------------------------+----------------------------------------------------------+
 |      INSTALLER_TYPE     |                          daisy                           |
 |     DEPLOY_SCENARIO     |                  os-nosdn-nofeature-ha                   |
 |        BUILD_TAG        |     jenkins-functest-daisy-baremetal-daily-master-67     |
 |         CI_LOOP         |                          daily                           |
 +-------------------------+----------------------------------------------------------+

::

 +------------------------------+------------------+---------------------+------------------+----------------+
 |          TEST CASE           |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
 +------------------------------+------------------+---------------------+------------------+----------------+
 |       connection_check       |     functest     |     healthcheck     |      00:08       |      PASS      |
 |          api_check           |     functest     |     healthcheck     |      04:22       |      PASS      |
 |      snaps_health_check      |     functest     |     healthcheck     |      00:35       |      PASS      |
 |          vping_ssh           |     functest     |        smoke        |      00:54       |      PASS      |
 |        vping_userdata        |     functest     |        smoke        |      00:27       |      PASS      |
 |     tempest_smoke_serial     |     functest     |        smoke        |      19:39       |      FAIL      |
 |         rally_sanity         |     functest     |        smoke        |      15:16       |      PASS      |
 |       refstack_defcore       |     functest     |        smoke        |      15:55       |      PASS      |
 |         snaps_smoke          |     functest     |        smoke        |      26:45       |      FAIL      |
 |           cloudify_ims       |     functest     |         vnf         |      23:56       |      PASS      |
 |        orchestra_openims     |     orchestra    |         vnf         |      15:07       |      PASS      |
 |     orchestra_clearwaterims  |     orchestra    |         vnf         |      19:10       |      PASS      |
 |           vyos_vrouter       |     functest     |         vnf         |      00:00       |      SKIP      |
 +------------------------------+------------------+---------------------+------------------+----------------+

Results are automatically pushed to the test results database, some additional
result files are pushed to OPNFV artifact web sites.

Based on the results stored in the result database, a `Functest reporting`_
portal is also automatically updated. This portal provides information on the
overall status per scenario and per installer

.. _`Functest reporting`: http://testresults.opnfv.org/reporting/master/functest/status-apex.html