<!doctype html> <html lang="en"> <head> <meta charset="utf-8"> <title>OPNFV presentation</title> <meta name="description" content="Functest: Many rivers to cross"> <meta name="author" content="Morgan Richomme and Functest contributors"> <meta name="apple-mobile-web-app-capable" content="yes" /> <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" /> <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no, minimal-ui"> <link rel="stylesheet" href="../css/reveal.css"> <link rel="stylesheet" href="../css/theme/OPNFV.css" id="theme"> <!-- Code syntax highlighting --> <link rel="stylesheet" href="../lib/css/zenburn.css"> <!-- Printing and PDF exports --> <script> var link = document.createElement( 'link' ); link.rel = 'stylesheet'; link.type = 'text/css'; link.href = window.location.search.match( /print-pdf/gi ) ? '../css/print/pdf.css' : '../css/print/paper.css'; document.getElementsByTagName( 'head' )[0].appendChild( link ); </script> <!--[if lt IE 9]> <script src="lib/js/html5shiv.js"></script> <![endif]--> </head> <body> <div class="reveal"> <!-- Any section element inside of this container is displayed as a slide --> <div class="slides"> <section data-background="../img/title-bg.png" data-background-transition="none"> <br> <h1>From Arno to Colorado</h1> <h3>Functest: Many rivers to cross</h3> <br><br> <h4>Functest contributors</h4> <h5>OPNFV Design Summit, 20/6/2016, Berlin</h5> <br> </section> <section> <blockquote> “Bad programmers have all the answers. Good testers have all the questions. Gil Zilberfeld - ” </blockquote> </section> <section data-markdown> # Agenda * Functest * Arno * Brahmaputra * Colorado * D, E, F rivers and beyond </section> <section data-markdown> # The dream team ![team](../img/teamSummitBerlin.png) </section> <section> <section data-markdown> # Functest: what for? </section> <section data-markdown> ## Our goals ### make sure OPNFV works...... </section> <section data-markdown> ## so concretely... * Validate scenarios for the release * Give confidence on OPNFV fresh releases * Ensure consistancy towards installers * Ensure End to End interoperability </section> <section data-markdown> ## and also * Provide functional test suites to be replayed * ~ post install sanity check * usable for plugfests * coherent with ETSI and Telco needs * Provide test tools * run tests * collect info * create dashboard </section> <section data-markdown> ## Our goals are not.... * Performance testing (yardstick, Xperf ) * Re-do tests done in specific projects * storage/storeperf * infra audit/qtip * Compare results </section> <section data-markdown> ## Our rules * Re-use as much as possible upstream suites (Rally, Tempest, Robot, testOn..) * Keep it simple... * Automate (if not it is not a test...) * Create new suites only if tests not covered * Consider fields not addressed by existing test projects </section> <section data-markdown> ## Our rules * Installer neutral * Controller neutral * Black box approach </section> </section> <section> <section data-markdown> # Arno </section> <section data-markdown> ## Proof of concept * 4 Testcases: vPing (userdata), Tempest, Rally, ODL * 1 scenario: os-odl_l2-nofeature-ha * 2 installers: Foreman/Fuel * Lots of manual operations (reporting, documentation) </section> </section> <section> <section data-markdown> # Brahmaputra </section> <section data-markdown> ## More everything.... * 7 Testcases: vPing (ssh), ONOS, vIMS, Tempest and Rally extended * 3 feature projects: doctor, promise, sdnvpn * 13 Scenarios * 4 Installers (Apex, Compass, Fuel, Joid) * Test result collection * Automatic dashboard </section> <section> <h3>Jiras</h3> <img height="200" data-src="../img/jiras.png" alt="Jiras"> <img height="200" data-src="../img/jiras2.png" alt="Jiras"> </section> <section> <h3>Brahmaputra Scenario matrix</h3> <img width="800" data-src="../img/matrix.png" alt="Jiras"> </section> <section> Comparison odl_l2-nofeature-noha scenario <table> <thead> <tr> <th>Project</th> <th>Apex</th> <th>Compass</th> <th>Fuel</th> <th>Joid</th> </tr> </thead> <tbody> <tr> <td>POD</td> <td>LF POD1</td> <td>Huawei US</td> <td>Ericsson POD2</td> <td>Orange POD2</td> </tr> <tr> <td>Functest</td> <td>2h50</td> <td>1h45</td> <td>2h35</td> <td>2h30</td> </tr> <tr> <td>Yardstick</td> <td>1h05</td> <td>1h00</td> <td>1h00</td> <td>0h55</td> </tr> </tbody> </table> * difference of duration due to POD and test suites </section> <section data-markdown> ## Lessons learned * Tempest: * Rally: * vIMS: complete but complex test case, very interesting to automate (pre MANO, most of Telco needs met there) </section> </section> <section> <section data-markdown> # Colorado </section> <section data-markdown> ## What's new? * Slicing of the tests (healthcheck / Smoke / SDN controllers / Features / VNFs) * Test duration management * New test cases: healthcheck, security * Refactoring (API, structure) * ARM Support * Automatic reporting * Dashboard evolution to ELK * CLI </section> <section> <h3>Automatic reporting</h3> <img height="400" data-src="../img/reporting.png" alt="Reporting"> <br><a href="http://testresults.opnfv.org/reporting/functest/release/master/index-status-fuel.html">Validated scenarios for Colorado</a> </section> </section> <section> <section data-markdown> # Many rivers to cross ## Functest beyond Colorado </section> <section data-markdown> ## More test cases * New controler: OpenContrail * New feature projects * New VNFs (vEPC, vCDN, vWhatever...) * Multi sites (e.g bgpvpn with different back ends) </section> <section data-markdown> ## A Functional testing As a Service framework * Scenarios can select their relevant test cases * Test duration estimation * Agile dashboarding * Analytics </section> <section data-markdown> ## Upstream * Rally * Functest description for ETSI? </section> </section> <section> <h3>Thank you</h3> <img width="600" data-src="../img/colorado.png" alt="tests"> </section> </div> <div class='footer'> <img src="../img/logo-OPNFV.png" alt="OPNFV logo"> </div> </div> <script src="../lib/js/head.min.js"></script> <script src="../js/reveal.js"></script> <script> // Full list of configuration options available at: // https://github.com/hakimel/reveal.js#configuration Reveal.initialize({ controls: true, progress: true, history: true, center: true, transition: 'slide', // none/fade/slide/convex/concave/zoom // Optional reveal.js plugins dependencies: [ { src: '../lib/js/classList.js', condition: function() { return !document.body.classList; } }, { src: '../plugin/markdown/marked.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } }, { src: '../plugin/markdown/markdown.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } }, { src: '../plugin/highlight/highlight.js', async: true, condition: function() { return !!document.querySelector( 'pre code' ); }, callback: function() { hljs.initHighlightingOnLoad(); } }, { src: '../plugin/zoom-js/zoom.js', async: true }, { src: '../plugin/notes/notes.js', async: true } ] }); </script> </body> </html>