<!doctype html>
<html lang="en">

	<head>
		<meta charset="utf-8">

		<title>OPNFV presentation</title>

		<meta name="description" content="Functest: Many rivers to cross">
		<meta name="author" content="Morgan Richomme and Functest contributors">

		<meta name="apple-mobile-web-app-capable" content="yes" />
		<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />

		<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no, minimal-ui">

		<link rel="stylesheet" href="../css/reveal.css">
		<link rel="stylesheet" href="../css/theme/OPNFV.css" id="theme">

		<!-- Code syntax highlighting -->
		<link rel="stylesheet" href="../lib/css/zenburn.css">

		<!-- Printing and PDF exports -->
		<script>
			var link = document.createElement( 'link' );
			link.rel = 'stylesheet';
			link.type = 'text/css';
			link.href = window.location.search.match( /print-pdf/gi ) ? '../css/print/pdf.css' : '../css/print/paper.css';
			document.getElementsByTagName( 'head' )[0].appendChild( link );
		</script>

		<!--[if lt IE 9]>
		<script src="lib/js/html5shiv.js"></script>
		<![endif]-->
	</head>

	<body>

		<div class="reveal">
			<!-- Any section element inside of this container is displayed as a slide -->
			<div class="slides">

				<section data-background="../img/title-bg.png" data-background-transition="none">
                     <br>
                     <h1>From Arno to Colorado</h1>
                     <h3>Functest: Many rivers to cross</h3>
                     <br><br>
					 <h4>Functest contributors</h4>
					 <h5>OPNFV Design Summit, 20/6/2016, Berlin</h5>
					 <br>

				</section>

				<section>
					<blockquote>
						&ldquo;Bad programmers have all the answers. Good testers have all the questions. Gil Zilberfeld - &rdquo;
					</blockquote>
				</section>

				<section data-markdown>
					# Agenda
					* Functest
					* Arno
					* Brahmaputra
					* Colorado
					* D, E, F rivers and beyond
				</section>

				<section data-markdown>
					# The dream team
					![team](../img/teamSummitBerlin.png)
				</section>

				<section>
					<section data-markdown>
						# Functest: what for?
					</section>
					<section data-markdown>
						## Our goals
						### make sure OPNFV works......
					</section>
					<section data-markdown>
						## so concretely...
						* Validate scenarios for the release	
						* Give confidence on OPNFV fresh releases
						* Ensure consistancy towards installers
						* Ensure End to End interoperability
					</section>
					<section data-markdown>
						## and also
						* Provide functional test suites to be replayed
						  * ~ post install sanity check
						  * usable for plugfests
						  * coherent with ETSI and Telco needs
						* Provide test tools
						  * run tests
						  * collect info
						  * create dashboard
					</section>
                    <section data-markdown>
						## Our goals are not....
						* Performance testing (yardstick, Xperf )
						* Re-do tests done in specific projects
						  * storage/storeperf
						  * infra audit/qtip
						* Compare results
					</section>

                    <section data-markdown>
						## Our rules
						* Re-use as much as possible upstream suites (Rally, Tempest, Robot, testOn..)
						* Keep it simple...
						* Automate (if not it is not a test...)
						* Create new suites only if tests not covered
						* Consider fields not addressed by existing test projects
					</section>

                    <section data-markdown>
						## Our rules
						* Installer neutral
						* Controller neutral
						* Black box approach
					</section>

				</section>

                <section>
                    <section data-markdown>
						# Arno
					</section>
                    <section data-markdown>
						## Proof of concept
						* 4 Testcases: vPing (userdata), Tempest, Rally, ODL
						* 1 scenario: os-odl_l2-nofeature-ha
						* 2 installers: Foreman/Fuel
						* Lots of manual operations (reporting, documentation)
					</section>
                </section>				
				<section>
                    <section data-markdown>
						# Brahmaputra
					</section>
                    <section data-markdown>
						## More everything....
						* 7 Testcases: vPing (ssh), ONOS, vIMS, Tempest and Rally extended
						* 3 feature projects: doctor, promise, sdnvpn
						* 13 Scenarios
						* 4 Installers (Apex, Compass, Fuel, Joid)
						* Test result collection 
						* Automatic dashboard 
					</section>
				<section>
					<h3>Jiras</h3>
					<img height="200"  data-src="../img/jiras.png" alt="Jiras">
					<img height="200"  data-src="../img/jiras2.png" alt="Jiras">
				</section>

				<section>
					<h3>Brahmaputra Scenario matrix</h3>
					<img width="800"  data-src="../img/matrix.png" alt="Jiras">
				</section>

				<section>
					Comparison odl_l2-nofeature-noha scenario
					<table>
						<thead>
							<tr>
								<th>Project</th>
								<th>Apex</th>
								<th>Compass</th>
								<th>Fuel</th>
								<th>Joid</th>
							</tr>
						</thead>
						<tbody>
							<tr>
								<td>POD</td>
								<td>LF POD1</td>
								<td>Huawei US</td>
								<td>Ericsson POD2</td>
								<td>Orange POD2</td>
							</tr>
							<tr>
								<td>Functest</td>
								<td>2h50</td>
								<td>1h45</td>
								<td>2h35</td>
								<td>2h30</td>
							</tr>
							<tr>
								<td>Yardstick</td>
								<td>1h05</td>
								<td>1h00</td>
								<td>1h00</td>
								<td>0h55</td>
							</tr>
						</tbody>
					</table>
                   * difference of duration due to POD and test suites 
				</section>

				<section data-markdown>
					## Lessons learned
					* Tempest: 
					* Rally:
					* vIMS: complete but complex test case, very interesting to automate (pre MANO, most of Telco needs met there)
				</section>

				</section>

				<section>				
					<section data-markdown>
						# Colorado
					</section>

					<section data-markdown>
						## What's new?
						* Slicing of the tests (healthcheck / Smoke / SDN controllers / Features / VNFs)
						* Test duration management
						* New test cases: healthcheck, security
						* Refactoring (API, structure)
						* ARM Support 
						* Automatic reporting
						* Dashboard evolution to ELK
						* CLI
					</section>

				<section>
					<h3>Automatic reporting</h3>
					<img height="400"  data-src="../img/reporting.png" alt="Reporting">
					<br><a href="http://testresults.opnfv.org/reporting/functest/release/master/index-status-fuel.html">Validated scenarios for Colorado</a>
				</section>

				</section>

                <section>
					<section data-markdown>
						# Many rivers to cross
						## Functest beyond Colorado
					</section>
					<section data-markdown>
						## More test cases
						* New controler: OpenContrail
						* New feature projects
						* New VNFs (vEPC, vCDN, vWhatever...)
						* Multi sites (e.g bgpvpn with different back ends)
					</section>					
					<section data-markdown>
						## A Functional testing As a Service framework
						* Scenarios can select their relevant test cases
						* Test duration estimation
						* Agile dashboarding
						* Analytics
					</section>	
                    <section data-markdown>
						## Upstream
						* Rally
						* Functest description for ETSI?   
					</section>					
                </section>

				<section>
					<h3>Thank you</h3>
					<img width="600"  data-src="../img/colorado.png" alt="tests">
				</section>

			</div>
            <div class='footer'>
				 <img src="../img/logo-OPNFV.png" alt="OPNFV logo"> 
	        </div>
		</div>

		<script src="../lib/js/head.min.js"></script>
		<script src="../js/reveal.js"></script>

		<script>

			// Full list of configuration options available at:
			// https://github.com/hakimel/reveal.js#configuration
			Reveal.initialize({
				controls: true,
				progress: true,
				history: true,
				center: true,

				transition: 'slide', // none/fade/slide/convex/concave/zoom

				// Optional reveal.js plugins
				dependencies: [
					{ src: '../lib/js/classList.js', condition: function() { return !document.body.classList; } },
					{ src: '../plugin/markdown/marked.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } },
					{ src: '../plugin/markdown/markdown.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } },
					{ src: '../plugin/highlight/highlight.js', async: true, condition: function() { return !!document.querySelector( 'pre code' ); }, callback: function() { hljs.initHighlightingOnLoad(); } },
					{ src: '../plugin/zoom-js/zoom.js', async: true },
					{ src: '../plugin/notes/notes.js', async: true }
				]
			});

		</script>

	</body>
</html>