diff options
| -rw-r--r-- | docs/testing/user/userguide/test_details.rst | 98 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_results.rst | 137 |
2 files changed, 170 insertions, 65 deletions
diff --git a/docs/testing/user/userguide/test_details.rst b/docs/testing/user/userguide/test_details.rst index fe7e6a9b6..002b3b7e6 100644 --- a/docs/testing/user/userguide/test_details.rst +++ b/docs/testing/user/userguide/test_details.rst @@ -302,39 +302,43 @@ Features Functest has been supporting several feature projects since Brahmaputra: -+-----------------+---------+----------+--------+-----------+ -| Test | Brahma | Colorado | Danube | Euphrates | -+=================+=========+==========+========+===========+ -| barometer | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| bgpvpn | | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| copper | | X | | | -+-----------------+---------+----------+--------+-----------+ -| doctor | X | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| domino | | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| fds | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| moon | | X | | | -+-----------------+---------+----------+--------+-----------+ -| multisite | | X | X | | -+-----------------+---------+----------+--------+-----------+ -| netready | | | X | | -+-----------------+---------+----------+--------+-----------+ -| odl_sfc | | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| opera | | | X | | -+-----------------+---------+----------+--------+-----------+ -| orchestra | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| parser | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| promise | X | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| security_scan | | X | X | | -+-----------------+---------+----------+--------+-----------+ ++-----------------+---------+----------+--------+-----------+-----------+ +| Test | Brahma | Colorado | Danube | Euphrates | Fraser | ++=================+=========+==========+========+===========+===========+ +| barometer | | | X | X | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| bgpvpn | | X | X | X | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| copper | | X | | | | ++-----------------+---------+----------+--------+-----------+-----------+ +| doctor | X | X | X | X | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| domino | | X | X | X | | ++-----------------+---------+----------+--------+-----------+-----------+ +| fds | | | X | X | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| moon | | X | | | | ++-----------------+---------+----------+--------+-----------+-----------+ +| multisite | | X | X | | | ++-----------------+---------+----------+--------+-----------+-----------+ +| netready | | | X | | | ++-----------------+---------+----------+--------+-----------+-----------+ +| odl_sfc | | X | X | X | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| opera | | | X | | | ++-----------------+---------+----------+--------+-----------+-----------+ +| orchestra | | | X | X | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| parser | | | X | X | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| promise | X | X | X | X | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| security_scan | | X | X | | | ++-----------------+---------+----------+--------+-----------+-----------+ +| clover | | | | | X | ++-----------------+---------+----------+--------+-----------+-----------+ +| stor4nfv | | | | | X | ++-----------------+---------+----------+--------+-----------+-----------+ Please refer to the dedicated feature user guides for details. @@ -419,6 +423,33 @@ The vyos-vrouter architecture is described in `[14]`_ juju_epc ^^^^^^^^ + +Kubernetes (K8s) +---------------- + +Kubernetes testing relies on sets of tests, which are part of the Kubernetes +source tree, such as the Kubernetes End-to-End (e2e) tests `[15]`_. + +The kubernetes testcases are distributed across various Tiers: + + * Healthcheck Tier + + * k8s_smoke Test Case: Creates a Guestbook application that contains redis + server, 2 instances of redis slave, frontend application, frontend service + and redis master service and redis slave service. Using frontend service, + the test will write an entry into the guestbook application which will + store the entry into the backend redis database. Application flow MUST + work as expected and the data written MUST be available to read. + + * Smoke Tier + + * k8s_conformance Test Case: Runs a series of k8s e2e tests expected to + pass on any Kubernetes cluster. It is a subset of tests necessary to + demonstrate conformance grows with each release. Conformance is thus + considered versioned, with backwards compatibility guarantees and are + designed to be run with no cloud provider configured. + + .. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html .. _`[3]`: https://rally.readthedocs.org/en/latest/index.html .. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml @@ -429,3 +460,4 @@ juju_epc .. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html .. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter +.. _`[15]`: https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst index 2d36bdc33..bb28989ae 100644 --- a/docs/testing/user/userguide/test_results.rst +++ b/docs/testing/user/userguide/test_results.rst @@ -10,42 +10,115 @@ In manual mode test results are displayed in the console and result files are put in /home/opnfv/functest/results. If you want additional logs, you may configure the logging.ini under -/usr/lib/python2.7/site-packages/functest/ci. +/usr/lib/python2.7/site-packages/xtesting/ci. Automated testing ----------------- -In automated mode, test results are displayed in jenkins logs, a summary is -provided at the end of the job and can be described as follow:: - - +-------------------------+----------------------------------------------------------+ - | ENV VAR | VALUE | - +-------------------------+----------------------------------------------------------+ - | INSTALLER_TYPE | daisy | - | DEPLOY_SCENARIO | os-nosdn-nofeature-ha | - | BUILD_TAG | jenkins-functest-daisy-baremetal-daily-master-67 | - | CI_LOOP | daily | - +-------------------------+----------------------------------------------------------+ - -:: - - +------------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +------------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:08 | PASS | - | api_check | functest | healthcheck | 04:22 | PASS | - | snaps_health_check | functest | healthcheck | 00:35 | PASS | - | vping_ssh | functest | smoke | 00:54 | PASS | - | vping_userdata | functest | smoke | 00:27 | PASS | - | tempest_smoke_serial | functest | smoke | 19:39 | FAIL | - | rally_sanity | functest | smoke | 15:16 | PASS | - | refstack_defcore | functest | smoke | 15:55 | PASS | - | snaps_smoke | functest | smoke | 26:45 | FAIL | - | cloudify_ims | functest | vnf | 23:56 | PASS | - | orchestra_openims | orchestra | vnf | 15:07 | PASS | - | orchestra_clearwaterims | orchestra | vnf | 19:10 | PASS | - | vyos_vrouter | functest | vnf | 00:00 | SKIP | - +------------------------------+------------------+---------------------+------------------+----------------+ +In automated mode, tests are run within split Alpine containers, and test +results are displayed in jenkins logs. The result summary is provided at the +end of each suite and can be described as follow. + +Healthcheck suite:: + + +----------------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +----------------------------+------------------+---------------------+------------------+----------------+ + | connection_check | functest | healthcheck | 00:07 | PASS | + | api_check | functest | healthcheck | 07:46 | PASS | + | snaps_health_check | functest | healthcheck | 00:36 | PASS | + +----------------------------+------------------+---------------------+------------------+----------------+ + +Smoke suite:: + + +------------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +------------------------------+------------------+---------------+------------------+----------------+ + | vping_ssh | functest | smoke | 00:57 | PASS | + | vping_userdata | functest | smoke | 00:33 | PASS | + | tempest_smoke_serial | functest | smoke | 13:22 | PASS | + | rally_sanity | functest | smoke | 24:07 | PASS | + | refstack_defcore | functest | smoke | 05:21 | PASS | + | patrole | functest | smoke | 04:29 | PASS | + | snaps_smoke | functest | smoke | 46:54 | PASS | + | odl | functest | smoke | 00:00 | SKIP | + | neutron_trunk | functest | smoke | 00:00 | SKIP | + +------------------------------+------------------+---------------+------------------+----------------+ + +Features suite:: + + +-----------------------------+------------------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-----------------------------+------------------------+------------------+------------------+----------------+ + | doctor-notification | doctor | features | 00:00 | SKIP | + | bgpvpn | sdnvpn | features | 00:00 | SKIP | + | functest-odl-sfc | sfc | features | 00:00 | SKIP | + | barometercollectd | barometer | features | 00:00 | SKIP | + | fds | fastdatastacks | features | 00:00 | SKIP | + +-----------------------------+------------------------+------------------+------------------+----------------+ + +Components suite:: + + +-------------------------------+------------------+--------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------------+------------------+--------------------+------------------+----------------+ + | tempest_full_parallel | functest | components | 48:28 | PASS | + | rally_full | functest | components | 126:02 | PASS | + +-------------------------------+------------------+--------------------+------------------+----------------+ + +Vnf suite:: + + +----------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +----------------------+------------------+--------------+------------------+----------------+ + | cloudify_ims | functest | vnf | 28:15 | PASS | + | vyos_vrouter | functest | vnf | 17:59 | PASS | + | juju_epc | functest | vnf | 46:44 | PASS | + +----------------------+------------------+--------------+------------------+----------------+ + +Parser testcase:: + + +-----------------------+-----------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-----------------------+-----------------+------------------+------------------+----------------+ + | parser-basics | parser | features | 00:00 | SKIP | + +-----------------------+-----------------+------------------+------------------+----------------+ + +Functest Kubernetes test result:: + + +--------------------------------------+------------------------------------------------------------+ + | ENV VAR | VALUE | + +--------------------------------------+------------------------------------------------------------+ + | INSTALLER_TYPE | compass | + | DEPLOY_SCENARIO | k8-nosdn-nofeature-ha | + | BUILD_TAG | jenkins-functest-compass-baremetal-daily-master-75 | + | CI_LOOP | daily | + +--------------------------------------+------------------------------------------------------------+ + +Kubernetes healthcheck suite:: + + +-------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------+------------------+---------------------+------------------+----------------+ + | k8s_smoke | functest | healthcheck | 01:54 | PASS | + +-------------------+------------------+---------------------+------------------+----------------+ + +Kubernetes smoke suite:: + + +-------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------+------------------+---------------+------------------+----------------+ + | k8s_conformance | functest | smoke | 57:47 | PASS | + +-------------------------+------------------+---------------+------------------+----------------+ + +Kubernetes features suite:: + + +----------------------+------------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +----------------------+------------------+------------------+------------------+----------------+ + | stor4nfv_k8s | stor4nfv | stor4nfv | 00:00 | SKIP | + | clover_k8s | clover | clover | 00:00 | SKIP | + +----------------------+------------------+------------------+------------------+----------------+ Results are automatically pushed to the test results database, some additional result files are pushed to OPNFV artifact web sites. |