diff options
| -rw-r--r-- | docs/testing/user/userguide/test_details.rst | 70 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_results.rst | 101 |
2 files changed, 106 insertions, 65 deletions
diff --git a/docs/testing/user/userguide/test_details.rst b/docs/testing/user/userguide/test_details.rst index 935298fdd..002b3b7e6 100644 --- a/docs/testing/user/userguide/test_details.rst +++ b/docs/testing/user/userguide/test_details.rst @@ -302,39 +302,43 @@ Features Functest has been supporting several feature projects since Brahmaputra: -+-----------------+---------+----------+--------+-----------+ -| Test | Brahma | Colorado | Danube | Euphrates | -+=================+=========+==========+========+===========+ -| barometer | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| bgpvpn | | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| copper | | X | | | -+-----------------+---------+----------+--------+-----------+ -| doctor | X | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| domino | | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| fds | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| moon | | X | | | -+-----------------+---------+----------+--------+-----------+ -| multisite | | X | X | | -+-----------------+---------+----------+--------+-----------+ -| netready | | | X | | -+-----------------+---------+----------+--------+-----------+ -| odl_sfc | | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| opera | | | X | | -+-----------------+---------+----------+--------+-----------+ -| orchestra | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| parser | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| promise | X | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| security_scan | | X | X | | -+-----------------+---------+----------+--------+-----------+ ++-----------------+---------+----------+--------+-----------+-----------+ +| Test | Brahma | Colorado | Danube | Euphrates | Fraser | ++=================+=========+==========+========+===========+===========+ +| barometer | | | X | X | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| bgpvpn | | X | X | X | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| copper | | X | | | | ++-----------------+---------+----------+--------+-----------+-----------+ +| doctor | X | X | X | X | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| domino | | X | X | X | | ++-----------------+---------+----------+--------+-----------+-----------+ +| fds | | | X | X | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| moon | | X | | | | ++-----------------+---------+----------+--------+-----------+-----------+ +| multisite | | X | X | | | ++-----------------+---------+----------+--------+-----------+-----------+ +| netready | | | X | | | ++-----------------+---------+----------+--------+-----------+-----------+ +| odl_sfc | | X | X | X | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| opera | | | X | | | ++-----------------+---------+----------+--------+-----------+-----------+ +| orchestra | | | X | X | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| parser | | | X | X | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| promise | X | X | X | X | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| security_scan | | X | X | | | ++-----------------+---------+----------+--------+-----------+-----------+ +| clover | | | | | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| stor4nfv | | | | | X | ++-----------------+---------+----------+--------+-----------+-----------+ Please refer to the dedicated feature user guides for details. diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst index c215e05ec..bb28989ae 100644 --- a/docs/testing/user/userguide/test_results.rst +++ b/docs/testing/user/userguide/test_results.rst @@ -10,42 +10,79 @@ In manual mode test results are displayed in the console and result files are put in /home/opnfv/functest/results. If you want additional logs, you may configure the logging.ini under -/usr/lib/python2.7/site-packages/functest/ci. +/usr/lib/python2.7/site-packages/xtesting/ci. Automated testing ----------------- -In automated mode, test results are displayed in jenkins logs, a summary is -provided at the end of the job and can be described as follow:: - - +-------------------------+----------------------------------------------------------+ - | ENV VAR | VALUE | - +-------------------------+----------------------------------------------------------+ - | INSTALLER_TYPE | daisy | - | DEPLOY_SCENARIO | os-nosdn-nofeature-ha | - | BUILD_TAG | jenkins-functest-daisy-baremetal-daily-master-67 | - | CI_LOOP | daily | - +-------------------------+----------------------------------------------------------+ - -:: - - +------------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +------------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:08 | PASS | - | api_check | functest | healthcheck | 04:22 | PASS | - | snaps_health_check | functest | healthcheck | 00:35 | PASS | - | vping_ssh | functest | smoke | 00:54 | PASS | - | vping_userdata | functest | smoke | 00:27 | PASS | - | tempest_smoke_serial | functest | smoke | 19:39 | FAIL | - | rally_sanity | functest | smoke | 15:16 | PASS | - | refstack_defcore | functest | smoke | 15:55 | PASS | - | snaps_smoke | functest | smoke | 26:45 | FAIL | - | cloudify_ims | functest | vnf | 23:56 | PASS | - | orchestra_openims | orchestra | vnf | 15:07 | PASS | - | orchestra_clearwaterims | orchestra | vnf | 19:10 | PASS | - | vyos_vrouter | functest | vnf | 00:00 | SKIP | - +------------------------------+------------------+---------------------+------------------+----------------+ +In automated mode, tests are run within split Alpine containers, and test +results are displayed in jenkins logs. The result summary is provided at the +end of each suite and can be described as follow. + +Healthcheck suite:: + + +----------------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +----------------------------+------------------+---------------------+------------------+----------------+ + | connection_check | functest | healthcheck | 00:07 | PASS | + | api_check | functest | healthcheck | 07:46 | PASS | + | snaps_health_check | functest | healthcheck | 00:36 | PASS | + +----------------------------+------------------+---------------------+------------------+----------------+ + +Smoke suite:: + + +------------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +------------------------------+------------------+---------------+------------------+----------------+ + | vping_ssh | functest | smoke | 00:57 | PASS | + | vping_userdata | functest | smoke | 00:33 | PASS | + | tempest_smoke_serial | functest | smoke | 13:22 | PASS | + | rally_sanity | functest | smoke | 24:07 | PASS | + | refstack_defcore | functest | smoke | 05:21 | PASS | + | patrole | functest | smoke | 04:29 | PASS | + | snaps_smoke | functest | smoke | 46:54 | PASS | + | odl | functest | smoke | 00:00 | SKIP | + | neutron_trunk | functest | smoke | 00:00 | SKIP | + +------------------------------+------------------+---------------+------------------+----------------+ + +Features suite:: + + +-----------------------------+------------------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-----------------------------+------------------------+------------------+------------------+----------------+ + | doctor-notification | doctor | features | 00:00 | SKIP | + | bgpvpn | sdnvpn | features | 00:00 | SKIP | + | functest-odl-sfc | sfc | features | 00:00 | SKIP | + | barometercollectd | barometer | features | 00:00 | SKIP | + | fds | fastdatastacks | features | 00:00 | SKIP | + +-----------------------------+------------------------+------------------+------------------+----------------+ + +Components suite:: + + +-------------------------------+------------------+--------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------------+------------------+--------------------+------------------+----------------+ + | tempest_full_parallel | functest | components | 48:28 | PASS | + | rally_full | functest | components | 126:02 | PASS | + +-------------------------------+------------------+--------------------+------------------+----------------+ + +Vnf suite:: + + +----------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +----------------------+------------------+--------------+------------------+----------------+ + | cloudify_ims | functest | vnf | 28:15 | PASS | + | vyos_vrouter | functest | vnf | 17:59 | PASS | + | juju_epc | functest | vnf | 46:44 | PASS | + +----------------------+------------------+--------------+------------------+----------------+ + +Parser testcase:: + + +-----------------------+-----------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-----------------------+-----------------+------------------+------------------+----------------+ + | parser-basics | parser | features | 00:00 | SKIP | + +-----------------------+-----------------+------------------+------------------+----------------+ Functest Kubernetes test result:: |