aboutsummaryrefslogtreecommitdiffstats
path: root/docs/testing/user/userguide/test_results.rst
blob: 3941ba0a17ae7deaa52b92759bcdcc69c550b768 (plain) 
1
2
3
4
5
6
7
8

@media only all and (prefers-color-scheme: dark) {
.highlight .hll { background-color: #49483e }
.highlight .c { color: #75715e } /* Comment */
.highlight .err { color: #960050; background-color: #1e0010 } /* Error */
.highlight .k { color: #66d9ef } /* Keyword */
.highlight .l { color: #ae81ff } /* Literal */
.highlight .n { color: #f8f8f2 } /* Name */
.highlight .o { color: #f92672 } /* Operator */
.highlight .p { color: #f8f8f2 } /* Punctuation */
.highlight .ch { color: #75715e } /* Comment.Hashbang */
.highlight .cm { color: #75715e } /* Comment.Multiline */
.highlight .cp { color: #75715e } /* Comment.Preproc */
.highlight .cpf { color: #75715e } /* Comment.PreprocFile */
.highlight .c1 { color: #75715e } /* Comment.Single */
.highlight .cs { color: #75715e } /* Comment.Special */
.highlight .gd { color: #f92672 } /* Generic.Deleted */
.highlight .ge { font-style: italic } /* Generic.Emph */
.highlight .gi { color: #a6e22e } /* Generic.Inserted */
.highlight .gs { font-weight: bold } /* Generic.Strong */
.highlight .gu { color: #75715e } /* Generic.Subheading */
.highlight .kc { color: #66d9ef } /* Keyword.Constant */
.highlight .kd { color: #66d9ef } /* Keyword.Declaration */
.highlight .kn { color: #f92672 } /* Keyword.Namespace */
.highlight .kp { color: #66d9ef } /* Keyword.Pseudo */
.highlight .kr { color: #66d9ef } /* Keyword.Reserved */
.highlight .kt { color: #66d9ef 
Test results
============

Manual testing
--------------

In manual mode test results are displayed in the console and result files
are put in /home/opnfv/functest/results.

If you want additional logs, you may configure the logging.ini under
/usr/lib/python2.7/site-packages/functest/ci.

Automated testing
--------------

In automated mode, test results are displayed in jenkins logs, a summary is provided
at the end of the job and can be described as follow::

 +-------------------------+----------------------------------------------------------+
 |         ENV VAR         |                          VALUE                           |
 +-------------------------+----------------------------------------------------------+
 |      INSTALLER_TYPE     |                          daisy                           |
 |     DEPLOY_SCENARIO     |                  os-nosdn-nofeature-ha                   |
 |        BUILD_TAG        |     jenkins-functest-daisy-baremetal-daily-master-67     |
 |         CI_LOOP         |                          daily                           |
 +-------------------------+----------------------------------------------------------+

 +------------------------------+------------------+---------------------+------------------+----------------+
 |          TEST CASE           |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
 +------------------------------+------------------+---------------------+------------------+----------------+
 |       connection_check       |     functest     |     healthcheck     |      00:08       |      PASS      |
 |          api_check           |     functest     |     healthcheck     |      04:22       |      PASS      |
 |      snaps_health_check      |     functest     |     healthcheck     |      00:35       |      PASS      |
 |          vping_ssh           |     functest     |        smoke        |      00:54       |      PASS      |
 |        vping_userdata        |     functest     |        smoke        |      00:27       |      PASS      |
 |     tempest_smoke_serial     |     functest     |        smoke        |      19:39       |      FAIL      |
 |         rally_sanity         |     functest     |        smoke        |      15:16       |      PASS      |
 |       refstack_defcore       |     functest     |        smoke        |      15:55       |      PASS      |
 |         snaps_smoke          |     functest     |        smoke        |      26:45       |      FAIL      |
 |           cloudify_ims       |     functest     |         vnf         |      23:56       |      PASS      |
 |        orchestra_openims     |     orchestra    |         vnf         |      15:07       |      PASS      |
 |     orchestra_clearwaterims  |     orchestra    |         vnf         |      19:10       |      PASS      |
 |           vyos_vrouter       |     functest     |         vnf         |      00:00       |      SKIP      |
 +------------------------------+------------------+---------------------+------------------+----------------+

Results are automatically pushed to the test results database, some additional
result files are pushed to OPNFV artifact web sites.

Based on the results stored in the result database, a `Functest reporting`_
portal is also automatically updated. This portal provides information on the
overall status per scenario and per installer

.. _`Functest reporting`: http://testresults.opnfv.org/reporting/master/functest/status-apex.html