From Arno to Colorado

Functest: Many rivers to cross


Functest contributors

OPNFV Design Summit, 20/6/2016, Berlin

“Bad programmers have all the answers. Good testers have all the questions. Gil Zilberfeld - ”
# Agenda * Functest * Arno * Brahmaputra * Colorado * D, E, F rivers and beyond
# The dream team ![team](../img/teamSummitBerlin.png)
# Functest: what for?
## Our goals ### make sure OPNFV works......
## so concretely... * Validate scenarios for the release * Give confidence on OPNFV fresh releases * Ensure consistancy towards installers * Ensure End to End interoperability
## and also * Provide functional test suites to be replayed * ~ post install sanity check * usable for plugfests * coherent with ETSI and Telco needs * Provide test tools * run tests * collect info * create dashboard
## Our goals are not.... * Performance testing (yardstick, Xperf ) * Re-do tests done in specific projects * storage/storeperf * infra audit/qtip * Compare results
## Our rules * Re-use as much as possible upstream suites (Rally, Tempest, Robot, testOn..) * Keep it simple... * Automate (if not it is not a test...) * Create new suites only if tests not covered * Consider fields not addressed by existing test projects
## Our rules * Installer neutral * Controller neutral * Black box approach
# Arno
## Proof of concept * 4 Testcases: vPing (userdata), Tempest, Rally, ODL * 1 scenario: os-odl_l2-nofeature-ha * 2 installers: Foreman/Fuel * Lots of manual operations (reporting, documentation)
# Brahmaputra
## More everything.... * 7 Testcases: vPing (ssh), ONOS, vIMS, Tempest and Rally extended * 3 feature projects: doctor, promise, sdnvpn * 13 Scenarios * 4 Installers (Apex, Compass, Fuel, Joid) * Test result collection * Automatic dashboard

Jiras

Jiras Jiras

Brahmaputra Scenario matrix

Jiras
Comparison odl_l2-nofeature-ha scenario
Project Apex Compass Fuel Joid
POD LF POD1 Huawei US Ericsson POD2 Orange POD2
Functest 2h50 1h45 2h35 2h30
Yardstick 1h05 1h00 1h00 0h55
* difference of duration due to POD and test suites
## Lessons learned * Tempest/Rally * harmonizing installer related OpenStack configuration differences * creating customized test lists * main challenge was resolving SDN controller interworking problems * vIMS * complete but complex test case * very interesting to automate (pre MANO, most of Telco needs met there)
# Colorado
## What's new? * New internal test cases: healthcheck, security * New controler: OpenContrail? * New feature projects (domino, multisites, movie, parser, moon, copper, models, onos-sfc...) * ARM Support (congratulations to ANEA team!) * Simplified feature project integration journey
## What's new? * Slicing of the tests (healthcheck / Smoke / SDN controllers / Features / Components / VNFs) * Better Test duration management * Refactoring (repo, case management, test API) * Automatic reporting * Dashboard evolution to ELK * CLI

Automatic reporting

Reporting
Validated scenarios for Colorado
# Many rivers to cross ## Functest beyond Colorado
## More test cases * New VNFs (vEPC, vCDN, vWhatever...) * Multi sites (e.g bgpvpn with different back ends) * Better test coverage
## A Functional testing As a Service framework * Scenario owner can select their relevant test cases * Test duration estimation * Agile dashboarding * Analytics
## Upstream * Rally: time to give back more * Functest description for ETSI?

Thank you

tests