From Arno to Colorado

Functest: Many rivers to cross

-
-

Functest contributors

OPNFV Design Summit, 20/6/2016, Berlin

-
- -

- -

- “Bad programmers have all the answers. Good testers have all the questions. Gil Zilberfeld - ” -

- -

- # Agenda - * Functest - * Arno - * Brahmaputra - * Colorado - * D, E, F rivers and beyond -

- -

- # The dream team - ![team](../img/teamSummitBerlin.png) -

- -

- # Functest: what for? -

- ## Our goals - ### make sure OPNFV works...... -

- ## so concretely... - * Validate scenarios for the release - * Give confidence on OPNFV fresh releases - * Ensure consistancy towards installers - * Ensure End to End interoperability -

- ## and also - * Provide functional test suites to be replayed - * ~ post install sanity check - * usable for plugfests - * coherent with ETSI and Telco needs - * Provide test tools - * run tests - * collect info - * create dashboard -

- ## Our goals are not.... - * Performance testing (yardstick, Xperf ) - * Re-do tests done in specific projects - * storage/storeperf - * infra audit/qtip - * Compare results -

- -

- ## Our rules - * Re-use as much as possible upstream suites (Rally, Tempest, Robot, testOn..) - * Keep it simple... - * Automate (if not it is not a test...) - * Create new suites only if tests not covered - * Consider fields not addressed by existing test projects -

- -

- ## Our rules - * Installer neutral - * Controller neutral - * Black box approach -

- -

- # Arno -

- ## Proof of concept - * 4 Testcases: vPing (userdata), Tempest, Rally, ODL - * 1 scenario: os-odl_l2-nofeature-ha - * 2 installers: Foreman/Fuel - * Lots of manual operations (reporting, documentation) -

- # Brahmaputra -

- ## More everything.... - * 7 Testcases: vPing (ssh), ONOS, vIMS, Tempest and Rally extended - * 3 feature projects: doctor, promise, sdnvpn - * 13 Scenarios - * 4 Installers (Apex, Compass, Fuel, Joid) - * Test result collection - * Automatic dashboard -

Jiras

- -

Brahmaputra Scenario matrix

- -

- Comparison odl_l2-nofeature-ha scenario - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Project	Apex	Compass	Fuel	Joid
POD	LF POD1	Huawei US	Ericsson POD2	Orange POD2
Functest	2h50	1h45	2h35	2h30
Yardstick	1h05	1h00	1h00	0h55

- * difference of duration due to POD and test suites -

- -

- ## Lessons learned - * Tempest/Rally - * harmonizing installer related OpenStack configuration differences - * creating customized test lists - * main challenge was resolving SDN controller interworking problems - * vIMS - * complete but complex test case - * very interesting to automate (pre MANO, most of Telco needs met there) -

- -

- # Colorado -

- -

- ## What's new? - * New internal test cases: healthcheck, security - * New controler: OpenContrail? - * New feature projects (domino, multisites, movie, parser, moon, copper, models, onos-sfc...) - * ARM Support (congratulations to ENEA team!) - * Simplified feature project integration journey -

- -

- ## What's new? - * Slicing of the tests (healthcheck / Smoke / SDN controllers / Features / Components / VNFs) - * Better Test duration management - * Refactoring (repo, case management, test API) - * Automatic reporting - * Dashboard evolution to ELK - * CLI -

- - -

Automatic reporting

-
Validated scenarios for Colorado -

- -

- # Many rivers to cross - ## Functest beyond Colorado -

- ## More test cases - * New VNFs (vEPC, vCDN, vWhatever...) - * Multi sites (e.g bgpvpn with different back ends) - * Better test coverage -

- ## A Functional testing As a Service framework - * Scenario owner can select their relevant test cases - * Test duration estimation - * Agile dashboarding - * Analytics -

- ## Upstream - * Rally: time to give back more - * Functest description for ETSI? -

- -

Thank you

- -