diff options
Diffstat (limited to 'docs/release/release-notes/functest-release.rst')
| -rw-r--r-- | docs/release/release-notes/functest-release.rst | 637 |
1 files changed, 637 insertions, 0 deletions
diff --git a/docs/release/release-notes/functest-release.rst b/docs/release/release-notes/functest-release.rst new file mode 100644 index 000000000..ce03047f6 --- /dev/null +++ b/docs/release/release-notes/functest-release.rst @@ -0,0 +1,637 @@ +======= +License +======= + +OPNFV Colorado release note for Functest Docs +(c) by Morgan Richomme (Orange) + +OPNFV Colorado release note for Functest Docs +are licensed under a Creative Commons Attribution 4.0 International License. +You should have received a copy of the license along with this. +If not, see <http://creativecommons.org/licenses/by/4.0/>. + +=========================================== +OPNFV Colorado1.0 release note for Functest +=========================================== + +Abstract +======== + +This document describes the release note of Functest project. + + +Version history +=============== + ++------------+----------+------------------+------------------------+ +| **Date** | **Ver.** | **Author** | **Comment** | +| | | | | ++------------+----------+------------------+------------------------+ +| 2016-08-17 | 1.0.0 | Morgan Richomme | Functest for Colorado | +| | | (Orange) | release | ++------------+----------+------------------+------------------------+ + +OPNFV Colorado Release +====================== + +Functest deals with functional testing of the OPNFV solution. +It includes test cases developed within the project and test cases developed in +other OPNFV projects and other upstream communities. + +The internal test cases are: + + * healthcheck + * vPing ssh + * vPing userdata + * Tempest Smoke Serial + * Rally Sanity + * ODL + * Tempest full parallel + * Rally full + * vIMS + +The OPNFV projects integrated into Functest framework for automation are: + + * bgpvpn + * Copper + * Doctor + * Domino + * Moon + * Multisite + * ONOSFW + * ONOS-sfc + * ODL-sfc + * Parser + * Promise + * Security scan + +The validation of a scenario requires a subset of these tests depending +on the installer and the scenario. + +The 3 last internal test cases (tempest full parallel, Rally full and +vIMS) are not considered for scenario validation. + +Release Data +============ + ++--------------------------------------+--------------------------------------+ +| **Project** | functest | +| | | ++--------------------------------------+--------------------------------------+ +| **Repo/tag** | colorado.1.0 | +| | | ++--------------------------------------+--------------------------------------+ +| **Release designation** | Colorado base release | +| | | ++--------------------------------------+--------------------------------------+ +| **Release date** | September 22 2016 | +| | | ++--------------------------------------+--------------------------------------+ +| **Purpose of the delivery** | Colorado base release | +| | | ++--------------------------------------+--------------------------------------+ + +Deliverables +============ + +Software +-------- + + - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: colorado.1.0) + + - The TestAPI Docker image: https://hub.docker.com/r/opnfv/testapi (tag:colorado.1.0) + + +Documents +--------- + + - Installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html + + - User Guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html + + - Developer Guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html + + +Version change +============== + +Feature evolution +----------------- + + - refactoring of ODL functional tests (with upstream modifications) + + - refactoring of TestAPI (update, swagger documentation, dockerization) + + - jenkins logs improvement + + - update integration of Doctor, Promise and SDNVPN projects + + - split Tempest and rally into 2 different tests: smoke and full + + - vIMS test suite integration + + - adoption of Kibana for dashboarding + + +New features +------------ + + - Functest CLI to prepare and run the tests + + - creation of the healthcheck test case + + - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, fdio, multisite) + + - integration of new OPNFV feature projects (copper, domino, multisite, + moon, parser, onos-sfc, odl-sfc, security scan) + + - introduction of test tiers in functest framework + + - automatic reporting + + - introduction of a jenkins summary table + + - support of ARM architecture + + +Scenario Matrix +=============== + +For Colorado 1.0, Functest was tested on the following scenarios (if not +precised, the scenario is a HA scenario): + ++---------------------+---------+---------+---------+---------+ +| Scenario | Apex | Compass | Fuel | Joid | ++=====================+=========+=========+=========+=========+ +| nosdn | X | X | X | X | ++---------------------+---------+---------+---------+---------+ +| nosdn-noha | | | X | X | ++---------------------+---------+---------+---------+---------+ +| kvm | | | X | | ++---------------------+---------+---------+---------+---------+ +| kvm-noha | | | X | | ++---------------------+---------+---------+---------+---------+ +| lxd | | | | X | ++---------------------+---------+---------+---------+---------+ +| lxd-noha | | | | X | ++---------------------+---------+---------+---------+---------+ +| ovs-noha (dpdk) | | | X | | ++---------------------+---------+---------+---------+---------+ +| odl_l2 | X | X | X | X | ++---------------------+---------+---------+---------+---------+ +| odl_l2-noha | | | X | | ++---------------------+---------+---------+---------+---------+ +| odl_l2-bgpvpn | X | | X | | ++---------------------+---------+---------+---------+---------+ +| odl_l2-bgpvpn-noha| | | X | | ++---------------------+---------+---------+---------+---------+ +| odl_l2-fdio-noha | X | | | | ++---------------------+---------+---------+---------+---------+ +| odl_l2-moon | | X | | | ++---------------------+---------+---------+---------+---------+ +| odl_l2-sfc-noha | X | | X | | ++---------------------+---------+---------+---------+---------+ +| odl_l3 | X | X | X | | ++---------------------+---------+---------+---------+---------+ +| odl_l3-noha | | | X | | ++---------------------+---------+---------+---------+---------+ +| onos | | X | X | X | ++---------------------+---------+---------+---------+---------+ +| onos-noha | | | X | | ++---------------------+---------+---------+---------+---------+ +| onos-sfc | | X | X | X | ++---------------------+---------+---------+---------+---------+ +| onos-sfc-noha | | | X | | ++---------------------+---------+---------+---------+---------+ +| multisite | | | X | | ++---------------------+---------+---------+---------+---------+ + +For Colorado 2.0, the following scenarios have been re-released: + ++---------------------+---------+---------+---------+---------+ +| Scenario | Apex | Compass | Fuel | Joid | ++=====================+=========+=========+=========+=========+ +| onos | X | X | X | X | ++---------------------+---------+---------+---------+---------+ +| odl_l2-sfc-ha | | | X | | ++---------------------+---------+---------+---------+---------+ + + +In Colorado, the functional tests have been sliced in 6 different +categories: + ++----------------+-----------------------------------------------+ +| Category | Description | ++================+===============================================+ +| healthcheck | Basic OpenStack commands | ++----------------+-----------------------------------------------+ +| smoke | vPings, Tempest and rally smoke tests | ++----------------+-----------------------------------------------+ +| sdn_suites | Specific SDN feature tests | ++----------------+-----------------------------------------------+ +| features | OPNFV feature project functional test suites | ++----------------+-----------------------------------------------+ +| openstack | Advanced, long duration OpenStack tests | +| | (Tempest and Rally full suite). Each test may | +| | last several hours | ++----------------+-----------------------------------------------+ +| vnf | Complex scenarios dealing with orchestration, | +| | VNF deployment and tests (e.g. vIMS) | ++----------------+-----------------------------------------------+ + +For the scenario validation, we consider only the healthcheck, smoke, +sdn_suites and features categories. These tests are run systematically +in the CI daily loops. + +Success criteria have been defined for these test cases, they can be +PASS/FAIL or a success rate may be declared (100%, > 90%) +All the criteria, as well as the test dependencies are declared in the +ci/testcases.yaml file. + +The scoring for the Colorado release per installer can be described as +follows. + +The scoring is an indicator showing how many feature project test suites +have been integrated on the scenario. + +The scoring equals the number of tests * succesful iteration of each +test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617 + + e.g. + apex/odl_l2-nofeature-ha + tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper + Scoring = 21/21 = 7 * 3 + +By default, if not specified, the scenarios are HA. +HA means OpenStack High Availability (main services). Note that not +all VIM (e.g. OpenStack) services are deployed in HA mode, as that +depends upon support of the specific service for HA deployment. +For example, in the Colorado release, the Congress service +is deployed in non-HA mode even for HA OPNFV scenarios, as explicit +support for HA operation has not yet been verified. +See the release notes of the installers for more details. + + +Apex +---- + ++------------------+---------+---------+-------------------+ +| Scenario | Scoring | Success | Results | +| | | rate | | ++==================+=========+=========+===================+ +| nosdn | 17/18 | 95% | `apex-res-174`_ | ++------------------+---------+---------+-------------------+ +| odl_l2 | 21/21 | 100% | `apex-res-175`_ | ++------------------+---------+---------+-------------------+ +| odl_l3 | 15/18 | 83% | `apex-res-176`_ | ++------------------+---------+---------+-------------------+ +| odl_l2-bgpvpn | 14/18 | 78% | `apex-res-235`_ | ++------------------+---------+---------+-------------------+ +| odl_l2-fdio-noha | 12/15 | 80% | `apex-res-6`_ | ++------------------+---------+---------+-------------------+ +| odl_l2-sfc-noha | 18/24 | 75% | `apex-res-217`_ | ++------------------+---------+---------+-------------------+ +| onos-nofeature-ha| 20/21 | 95% | `apex-res-423`_ | ++------------------+---------+---------+-------------------+ + + +Compass +------- + ++------------------+---------+---------+-------------------+ +| Scenario | Scoring | Success | Results | +| | | rate | | ++==================+=========+=========+===================+ +| nosdn | 12/12 | 100% | `compass-res-55`_ | ++------------------+---------+---------+-------------------+ +| odl_l2 | 15/15 | 100% | `compass-res-59`_ | ++------------------+---------+---------+-------------------+ +| odl_l3 | 9/12 | 75% | `compass-res-73`_ | ++------------------+---------+---------+-------------------+ +| odl_l2-moon | 15/18 | 83% | `compass-res-567`_| ++------------------+---------+---------+-------------------+ +| onos-ha | 15/15 | 100% | `compass-res-285`_| ++------------------+---------+---------+-------------------+ +| onos-sfc-ha | 17/18 | 95% | `compass-res-76`_ | ++------------------+---------+---------+-------------------+ + +Note: all the Compass tests for Colorado have been executed on virtual +environment. Bare metal resources were used for Master branch. + + +Fuel +---- + ++---------------------+---------+---------+-------------------+ +| Scenario | Scoring | Success | Results | +| | | rate | | ++=====================+=========+=========+===================+ +| nosdn | 18/18 | 100% | `fuel-res-129`_ | ++---------------------+---------+---------+-------------------+ +| nosdn-noha | 15/15 | 100% | `fuel-res-154`_ | ++---------------------+---------+---------+-------------------+ +| nosdn-kvm | 18/18 | 100% | `fuel-res-128`_ | ++---------------------+---------+---------+-------------------+ +| nosdn-kvm-noha | 15/15 | 100% | `fuel-res-161`_ | ++---------------------+---------+---------+-------------------+ +| nosdn-ovs | 12/18 | 67%* | `fuel-res-213`_ | ++---------------------+---------+---------+-------------------+ +| nosdn-ovs-noha | 15/15 | 100% | `fuel-res-162`_ | ++---------------------+---------+---------+-------------------+ +| odl_l2 | 21/21 | 100% | `fuel-res-123`_ | ++---------------------+---------+---------+-------------------+ +| odl_l2-noha | 17/18 | 94% | `fuel-res-155`_ | ++---------------------+---------+---------+-------------------+ +| odl_l2-bgpvpn | 14/18 | 78% | `fuel-res-119`_ | ++---------------------+---------+---------+-------------------+ +| odl_l2-bgpvpn-noha | 14/15 | 93% | `fuel-res-160`_ | ++---------------------+---------+---------+-------------------+ +| odl_l2-sfc-noha | 6/21 | 29% | `fuel-res-219`_ | ++---------------------+---------+---------+-------------------+ +| odl_l2-sfc-ha | 16/21 | 76% | `fuel-res-376`_ | ++---------------------+---------+---------+-------------------+ +| odl_l3 | 15/18 | 83% | `fuel-res-115`_ | ++---------------------+---------+---------+-------------------+ +| odl_l3-noha | 12/15 | 80% | `fuel-res-164`_ | ++---------------------+---------+---------+-------------------+ +| onos | 20/21 | 95% | `fuel-res-492`_ | ++---------------------+---------+---------+-------------------+ +| onos-noha | 18/18 | 100% | `fuel-res-166`_ | ++---------------------+---------+---------+-------------------+ +| onos-sfc | 24/24 | 100% | `fuel-res-124`_ | ++---------------------+---------+---------+-------------------+ +| onos-sfc-noha | 21/21 | 100% | `fuel-res-129`_ | ++---------------------+---------+---------+-------------------+ +| multisite | N.R | 100% | `fuel-res-8`_ | ++---------------------+---------+---------+-------------------+ + +*: all results passed, lacking iterations to reach the full score + +Results of Functest on AArch64 Colorado 3.0 + ++---------------------+---------+---------+----------------------+ +| Scenario | Scoring | Success | Results | +| | | rate | | ++=====================+=========+=========+======================+ +| nosdn | 18/18 | 100% | `fuel-arm-res-128`_ | ++---------------------+---------+---------+----------------------+ +| odl_l2 | 21/21 | 100% | `fuel-arm-res-122`_ | ++---------------------+---------+---------+----------------------+ +| odl_l2-noha | 18/18 | 100% | `fuel-arm-res-129`_ | ++---------------------+---------+---------+----------------------+ +| odl_l3 | 17/18 | 95% | `fuel-arm-res-135`_ | ++---------------------+---------+---------+----------------------+ + + +Joid +---- + ++---------------------+---------+---------+-----------------+ +| Scenario | Scoring | Success | Results | +| | | rate | | ++=====================+=========+=========+=================+ +| nosdn | 18/18 | 100% | `joid-res-102`_ | ++---------------------+---------+---------+-----------------+ +| nosdn-noha | 17/18 | 95% | `joid-res-93`_ | ++---------------------+---------+---------+-----------------+ +| nosdn-lxd | 12/12 | 100% | `joid-res-104`_ | ++---------------------+---------+---------+-----------------+ +| nosdn-lxd-noha | 12/12 | 100% | `joid-res-91`_ | ++---------------------+---------+---------+-----------------+ +| odl_l2 | 21/21 | 100% | `joid-res-103`_ | ++---------------------+---------+---------+-----------------+ +| onos | 21/21 | 100% | `joid-res-345`_ | ++---------------------+---------+---------+-----------------+ +| onos-sfc | 24/24 | 100% | `joid-res-97`_ | ++---------------------+---------+---------+-----------------+ + +It is highly recommended to install a json viewer in your browser +(e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/) + +You can get additional details through test logs on http://artifacts.opnfv.org/. +As no search engine is available on the OPNFV artifact web site you must +retrieve the pod identifier on which the tests have been executed (see +field pod in any of the results) then click on the selected POD and look +for the date of the test you are interested in. + +The reporting pages can be found at: + + * apex: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html + * compass: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-compass.html + * fuel: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html + * joid: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-joid.html + +Colorado known restrictions/issues +================================== + ++-----------+-----------+----------------------------------------------+ +| Installer | Scenario | Issue | ++===========+===========+==============================================+ +| any | odl_l3-* | Tempest cases related to using floating IP | +| | | addresses fail because of a known ODL bug. | +| | | vPing_ssh test case is excluded for the same | +| | | reason. | +| | | https://jira.opnfv.org/browse/APEX-112 | +| | | https://jira.opnfv.org/browse/FUNCTEST-445 | ++-----------+-----------+----------------------------------------------+ +| apex/fuel | *-bgpvpn | vPing_ssh (floating ips not supported) and | +| | | vIMS excluded. Some Tempest cases related to | +| | | floating ips also excluded. Some performance | +| | | issues have been detected in this scenario | +| | | (i.e. BGPVPN extension enabled) when running | +| | | commands against the OpenStack APIs, thus | +| | | Rally sanity test case has been disabled. | +| | | Performance issues seem to be connected to | +| | | the ODL version. It is planned to reintroduce| +| | | Rally sanity in Colorado 2.0 with the | +| | | adoption of ODL Boron release. | ++-----------+-----------+----------------------------------------------+ +| apex | *-fdio | Due to late integration, fdio decided to | +| | | focus on mandatory tests and exclude feature | +| | | tests (copper, doctor, security_scan) from | +| | | its scenarios | ++-----------+-----------+----------------------------------------------+ +| compass | moon | First ODL test FAILS because ODL/Openstack | +| | | federation done in moon is partial. Only | +| | | MD-SAL is federated (not AD-SAL) | ++-----------+-----------+----------------------------------------------+ +| fuel | any | TestServerBasicOps test case skipped | +| | | https://gerrit.opnfv.org/gerrit/#/c/19635/ | ++-----------+-----------+----------------------------------------------+ +| fuel | kvm | vPing_ssh and vIMS excluded (metadata related| +| | | scenarios) | ++-----------+-----------+----------------------------------------------+ +| fuel | multisite | random errors running multisite. A known bug | +| | | in keystone mitaka, due to which memcache | +| | | raises exception and keystone becomes | +| | | unresponsive | +| | | bugs.launchpad.net/keystone/+bug/1600394 | +| | | workaround consists in restarting memcache on| +| | | server | ++-----------+-----------+----------------------------------------------+ +| joid | any | Tempest cases related to object storage | +| | | excluded | +| | | https://gerrit.opnfv.org/gerrit/#/c/17871/ | ++-----------+-----------+----------------------------------------------+ +| joid | domino | Domino tests are skipped in CI. However the | +| | | test case can be run by manually setting | +| | | IS_IPandKEY_CONFIGURED=true after manually | +| | | setting properly the IP addresses of the 3 | +| | | Controller nodes in the configuration file | +| | | /repos/domino/tests/run_multinode.sh | ++-----------+-----------+----------------------------------------------+ + + +Test and installer/scenario dependencies +======================================== + +It is not always possible to run all the test cases on all the scenarios. +The following table details the dependencies of the test cases per +scenario. The scenario dependencies (installer or scenario) are detailed +in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml + +Test results +============ + +Test results are available in: + + - test results document: http://artifacts.opnfv.org/functest + + - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/ + + - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/ + + + +Open JIRA tickets +================= + ++------------------+-----------------------------------------------+ +| JIRA | Description | ++==================+===============================================+ +| `FUNCTEST-446`_ | Cleanup ODL-SFC output in Functest execution | +| | Impact on odl_l2-sfc scenarios | ++------------------+-----------------------------------------------+ +| `FUNCTEST-454`_ | Cleanup failures when using HA networks in | +| | Neutron | ++------------------+-----------------------------------------------+ + +Useful links +============ + + - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing + + - wiki Functest Colorado page: https://wiki.opnfv.org/display/functest/Functest+Colorado + + - Functest repo: https://git.opnfv.org/cgit/functest + + - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/ + + - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611 + + - Functest IRC chan: #opnfv-functest + + - Functest reporting: http://testresults.opnfv.org/reporting + + - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml + + - Functest Colorado user guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html + + - Functest installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html + + - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html + +.. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446 + +.. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454 + +.. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6 + +.. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174 + +.. _`apex-res-175`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-175 + +.. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176 + +.. _`apex-res-217`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-217 + +.. _`apex-res-235`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-235 + +.. _`apex-res-423`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-423 + +.. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55 + +.. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59 + +.. _`compass-res-73`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-73 + +.. _`compass-res-76`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-76 + +.. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77 + +.. _`compass-res-567`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-567 + +.. _`compass-res-285`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-285 + +.. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console + +.. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115 + +.. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117 + +.. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119 + +.. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123 + +.. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124 + +.. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128 + +.. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129 + +.. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154 + +.. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155 + +.. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160 + +.. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161 + +.. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162 + +.. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164 + +.. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166 + +.. _`fuel-res-213`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-213 + +.. _`fuel-res-219`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-219 + +.. _`fuel-res-376`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-376 + +.. _`fuel-res-492`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-492 + +.. _`fuel-arm-res-128`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/128/console + +.. _`fuel-arm-res-122`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/122/console + +.. _`fuel-arm-res-129`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/129/console + +.. _`fuel-arm-res-135`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/135/console + +.. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93 + +.. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91 + +.. _`joid-res-99`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-99 + +.. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97 + +.. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102 + +.. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103 + +.. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104 + +.. _`joid-res-345`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-345 + |