From 6bb45e4d1ba014c1dd33bedff49be8afa9426d17 Mon Sep 17 00:00:00 2001 From: Morgan Richomme Date: Thu, 13 Oct 2016 18:02:06 +0200 Subject: remove 3rd part files with MIT or BSD license Change-Id: I941093e91897d1425720b5acdbf072cf620f131d Signed-off-by: Morgan Richomme --- docs/com/pres/Summit/Berlin-2016/conversation.html | 251 ++++++++++++++ .../com/pres/Summit/Berlin-2016/summit-Berlin.html | 313 +++++++++++++++++ docs/com/pres/Summit/Berlin-2016/testapi.html | 274 +++++++++++++++ docs/com/pres/conversation.html | 251 -------------- docs/com/pres/index.template.html | 378 --------------------- docs/com/pres/summit-Berlin.html | 313 ----------------- docs/com/pres/testapi.html | 274 --------------- 7 files changed, 838 insertions(+), 1216 deletions(-) create mode 100755 docs/com/pres/Summit/Berlin-2016/conversation.html create mode 100755 docs/com/pres/Summit/Berlin-2016/summit-Berlin.html create mode 100755 docs/com/pres/Summit/Berlin-2016/testapi.html delete mode 100755 docs/com/pres/conversation.html delete mode 100755 docs/com/pres/index.template.html delete mode 100755 docs/com/pres/summit-Berlin.html delete mode 100755 docs/com/pres/testapi.html (limited to 'docs/com/pres') diff --git a/docs/com/pres/Summit/Berlin-2016/conversation.html b/docs/com/pres/Summit/Berlin-2016/conversation.html new file mode 100755 index 00000000..b56b1e10 --- /dev/null +++ b/docs/com/pres/Summit/Berlin-2016/conversation.html @@ -0,0 +1,251 @@ + + + + + + + OPNFV presentation + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+




+

Conversation with the Testing Community

+

+

OPNFV testing community

+
OPNFV Design Summit, 20/6/2016, Berlin
+
+ +
+ +
+ >“You make experiments and I make theories. Do you know the difference? A theory is something nobody believes, except the person who made it. An experiment is something everybody believes, except the person who made it." A.Einstein +
+ +
+ # Agenda + * Who are we? + * Upstream, Upstream, Upstream + * Towards Telco Cloud KPI +
+ +
+ # Who are we? + ![team](../../../img/OPNFV_testing_group.png) + This is a representation...not a governance! +
+
+
+

The test projects

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Test caseDescription
FunctestUmbrella project for Functional testing
YardstickUmbrella project for performance/qualification testing
CPerfSDN controller performance testing
StorPerfStorage performance testing
VSPerfVSwitch qualification
BottlenecksDetect possible bottlenecks
QTIPBenchmark to boost performance
+
+ +
+ # Functest + * Functional testing to validate scenarios + * Contributors (raise your hands) + * Functest presentation (here...1h ago :)) + * breakout sessions + * Today 3.10PM-3.40PM (vIMS evolution) 4PM (feature project integration), 5PM (Colorado status) + * tomorrow 11AM (API, test collection), 1PM (work Upstream), 3.15PM (D Release)(to be confirmed...) +
+
+ # Yardstick + * Performance testing framework and NFVI Validation + * [Contributors](https://wiki.opnfv.org/display/yardstick/People) + * Yardstick presentation + * How to Do a Pre-deployment NFVI Validation Quickly and Efficiently? Wednesday, June 22, 4:00 PM - 4:30 PM + * Yardstick: A Pre-deployment NFVI Validation Tool. Thursday, June 23, 3:10 PM - 3:30 PM + * breakout sessions + * Tomorrow 2PM (Yardstick in C) +
+
+ # CPerf + * SDN Controller Performance Testing + * Upstream test experts + standards/metrics experts + * Goal: Cross-controller perf tests in CI against realistic deployments +
+
+ # StorPerf + * Cinder Volume Performance Testing + * Based on SNIA's SSD Test Specification + * Presentation + * StorPerf: Cinder Storage Performance Measurement. + Wednesday, June 22, 3:40 PM - 3:55 PM + * Breakout Session + * Tomorrow 10:15 (StorPerf Test Results Working Session) +
+
+ # VSPerf +
+
+ # Bottlenecks +
+
+ # QTIP + + ## OPNFV Platform Benchmark + + - Bottom up + - testing baremetal components first + - More than result + - comparing against reference + - Setup-Test-Diagnose-Improve + - a closed loop to boost platform performance + + ## Breakout session + + - 14:30~15:45, June 21st/Tuesday + - Room Tegel +
+
+
+
+ # Upstream, upstream, upstream.. +
+
+ ## We are on the shoulders of the giants + * Rally (OpenStack) + * ODL, ONOS, ... + * RobotFramework, Teston + * ..... +
+
+ ## Time to give back + ### How to improve work with testing upstream community +
+ +
+ +
+ +
+ # Towards Telco Cloud KPI? +
+ +
+ # Testing group goals + * Define common needs (naming, rules, tools...) + * Unify result collection, reporting, dashboarding, documentation + * Share best practice: docker, CLI, .. + * Identify testing domains + * Encourage testing in poorly covered areas +
+ +
+ # The ultimate goal: define Telco Cloud KPIs + * Provide methodology and test suites to qualify a Telco Cloud + * Provide reference benchs for plugfest/third party integration: compare apple with apple +
+ +
+ +
+ # Thank you +
+ +
+
+ OPNFV logo +
+
+ + + + + + + + diff --git a/docs/com/pres/Summit/Berlin-2016/summit-Berlin.html b/docs/com/pres/Summit/Berlin-2016/summit-Berlin.html new file mode 100755 index 00000000..4173a86a --- /dev/null +++ b/docs/com/pres/Summit/Berlin-2016/summit-Berlin.html @@ -0,0 +1,313 @@ + + + + + + + OPNFV presentation + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+




+

From Arno to Colorado

+

Functest: Many rivers to cross

+
+

Functest contributors

+
OPNFV Design Summit, 20/6/2016, Berlin
+
+ +
+ +
+
+ “Bad programmers have all the answers. Good testers have all the questions. Gil Zilberfeld - ” +
+
+ +
+ # Agenda + * Functest + * Arno + * Brahmaputra + * Colorado + * D, E, F rivers and beyond +
+ +
+ # The dream team + ![team](../../../img/teamSummitBerlin.png) +
+ +
+
+ # Functest: what for? +
+
+ ## Our goals + ### make sure OPNFV works...... +
+
+ ## so concretely... + * Validate scenarios for the release + * Give confidence on OPNFV fresh releases + * Ensure consistancy towards installers + * Ensure End to End interoperability +
+
+ ## and also + * Provide functional test suites to be replayed + * ~ post install sanity check + * usable for plugfests + * coherent with ETSI and Telco needs + * Provide test tools + * run tests + * collect info + * create dashboard +
+
+ ## Our goals are not.... + * Performance testing (yardstick, Xperf ) + * Re-do tests done in specific projects + * storage/storeperf + * infra audit/qtip + * Compare results +
+ +
+ ## Our rules + * Re-use as much as possible upstream suites (Rally, Tempest, Robot, testOn..) + * Keep it simple... + * Automate (if not it is not a test...) + * Create new suites only if tests not covered + * Consider fields not addressed by existing test projects +
+ +
+ ## Our rules + * Installer neutral + * Controller neutral + * Black box approach +
+ +
+ +
+
+ # Arno +
+
+ ## Proof of concept + * 4 Testcases: vPing (userdata), Tempest, Rally, ODL + * 1 scenario: os-odl_l2-nofeature-ha + * 2 installers: Foreman/Fuel + * Lots of manual operations (reporting, documentation) +
+
+
+
+ # Brahmaputra +
+
+ ## More everything.... + * 7 Testcases: vPing (ssh), ONOS, vIMS, Tempest and Rally extended + * 3 feature projects: doctor, promise, sdnvpn + * 13 Scenarios + * 4 Installers (Apex, Compass, Fuel, Joid) + * Test result collection + * Automatic dashboard +
+
+

Jiras

+ Jiras + Jiras +
+ +
+

Brahmaputra Scenario matrix

+ Jiras +
+ +
+ Comparison odl_l2-nofeature-ha scenario + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ProjectApexCompassFuelJoid
PODLF POD1Huawei USEricsson POD2Orange POD2
Functest2h501h452h352h30
Yardstick1h051h001h000h55
+ * difference of duration due to POD and test suites +
+ +
+ ## Lessons learned + * Tempest/Rally + * harmonizing installer related OpenStack configuration differences + * creating customized test lists + * main challenge was resolving SDN controller interworking problems + * vIMS + * complete but complex test case + * very interesting to automate (pre MANO, most of Telco needs met there) +
+ +
+ +
+
+ # Colorado +
+ +
+ ## What's new? + * New internal test cases: healthcheck, security + * New controler: OpenContrail? + * New feature projects (domino, multisites, movie, parser, moon, copper, models, onos-sfc...) + * ARM Support (congratulations to ENEA team!) + * Simplified feature project integration journey +
+ +
+ ## What's new? + * Slicing of the tests (healthcheck / Smoke / SDN controllers / Features / Components / VNFs) + * Better Test duration management + * Refactoring (repo, case management, test API) + * Automatic reporting + * Dashboard evolution to ELK + * CLI +
+ + +
+

Automatic reporting

+ Reporting +
Validated scenarios for Colorado +
+ +
+ +
+
+ # Many rivers to cross + ## Functest beyond Colorado +
+
+ ## More test cases + * New VNFs (vEPC, vCDN, vWhatever...) + * Multi sites (e.g bgpvpn with different back ends) + * Better test coverage +
+
+ ## A Functional testing As a Service framework + * Scenario owner can select their relevant test cases + * Test duration estimation + * Agile dashboarding + * Analytics +
+
+ ## Upstream + * Rally: time to give back more + * Functest description for ETSI? +
+
+ +
+

Thank you

+ tests +
+ +
+
+ OPNFV logo +
+
+ + + + + + + + diff --git a/docs/com/pres/Summit/Berlin-2016/testapi.html b/docs/com/pres/Summit/Berlin-2016/testapi.html new file mode 100755 index 00000000..7035d71d --- /dev/null +++ b/docs/com/pres/Summit/Berlin-2016/testapi.html @@ -0,0 +1,274 @@ + + + + + + + OPNFV presentation + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+




+

Test API

+

Clean, Easy, Complete

+
+

OPNFV testing community

+
OPNFV Design Summit, 20/6/2016, Berlin
+
+
+ +
+ # Agenda + * testAPI: what for? + * API overview + * API evolution + * Roadmap +
+ +
+
+ # test API: what for? +
+
+ ## Consistant view for + * Resources (Pods) + * Projects + * Test cases + * Results +
+ +
+ ## Status Currently + * From many projects + * Many formats (log, html, json, ..) + * Many locations (in VM, Jumphost, external DB,...) +
+
+ +
+ +
+ ## Help building + * Dashboards + * Automatic reporting +
+
+ +
+
+ # API overview +
+
+ ## Storage structure + ![testapi](https://wiki.opnfv.org/download/attachments/2926452/results_collection_structure.png?version=1&modificationDate=1459196347000&api=v2 "OPNFV API page") +
+
+ ## API in Brahmaputra + ![testapi](../../../img/testapi0.png) + https://wiki.opnfv.org/display/functest/Collection+Of+Test+Results +
+ +
+ ## API status in Brahmaputra + * Simple data models aggreed with the testing group + * No unit tests + * Wiki and rst documentation +
+ +
+ ## Lessons learned in B. + * Painful wiki documentation + * Simple result model + * Misuse version as scenario + * No version actually + * No overall criteria (PASS/FAIL) + * Difficult refactoring +
+
+

Colorado refactoring

+
    +
  • Unit tests
    • +
  • Update of the data model
    • +
  • Tornado_swagger for the documentation
    • +
  • URL and MongoDB collection renaming
    • +
  • Scripts to support databse backup/restore/update
    • +
  • Support installation
    • + +
+

All done in Colorado!

+
+
+ +
+
+ # API Evoluation +
+ +
+

URI changes...

+
+

testresults.opnfv.org/testapi =>
testresults.opnfv.org/test/api/v1

+ +

/test/api/v1/pods

+

/test/api/v1/projects

+

/test/api/v1/projects/qtip/cases

+

/test/api/v1/results

+
+
+
+ ## One new field for pods + | Field | Colorado | + | :--- | :--- | + | role | community_pod, ci_pod, .. | +
+
+ ## Changed fields for results + | Item | Brahmaputra | Colorado | + | :--- | :--- | :--- | + | date | creation_date | start_date and stop_date | + | version | used as scenario | opnfv release version | +
+
+ +
+
+ ## One new filter for results + | Item | Colorado | + | :--- | :--- | + | last | get last N results| +
+
+ http://testresults.opnfv.org/test/swagger/spec.html + ![alt text](../../../img/testapi1.png "Test API swagger interface") +
+
+

unit tests

+ 

+umry8364@umry8364-Latitude-E6400:~/Dev/OPNFV/releng/utils/test/result_collection_api$ ./run_test.sh 
+Tests running...
+WARNING:tornado.general:404 GET /dashboard/v1/results?case=vPing&pod=zte-pod1&version=C&installer=fuel&period=5 (127.0.0.1): Project name missing
+WARNING:tornado.access:404 GET /dashboard/v1/results?case=vPing&pod=zte-pod1&version=C&installer=fuel&period=5 (127.0.0.1) 2.30ms
+WARNING:tornado.general:400 POST /api/v1/projects (127.0.0.1): name missing
+............
+WARNING:tornado.access:400 POST /api/v1/projects (127.0.0.1) 1.13ms
+WARNING:tornado.access:403 PUT /api/v1/projects/functest/cases/vping_1 (127.0.0.1) 2.95ms
+WARNING:tornado.general:404 PUT /api/v1/projects/functest/cases/notFound (127.0.0.1): {'project_name': u'functest', 'name': u'notFound'} could not be found in table [testcases]
+WARNING:tornado.access:404 PUT /api/v1/projects/functest/cases/notFound (127.0.0.1) 2.85ms
+
+Ran 74 tests in 1.848s
+OK
+
+
+
+ +
+
+ # Roadmap +
+
+ ## Roadmap + * Dockerize testAPI + * Automatic update of testAPI + * Command Line Interface + * Automatic update for pods/projects/testcases +
+
+ # suggestions ... +
+
+

Thank you

+ tests +
+
+
+
+ OPNFV logo +
+
+ + + + + + + + diff --git a/docs/com/pres/conversation.html b/docs/com/pres/conversation.html deleted file mode 100755 index df14f514..00000000 --- a/docs/com/pres/conversation.html +++ /dev/null @@ -1,251 +0,0 @@ - - - - - - - OPNFV presentation - - - - - - - - - - - - - - - - - - - - - - - -
- -
- -
-




-

Conversation with the Testing Community

-

-

OPNFV testing community

-
OPNFV Design Summit, 20/6/2016, Berlin
-
- -
- -
- >“You make experiments and I make theories. Do you know the difference? A theory is something nobody believes, except the person who made it. An experiment is something everybody believes, except the person who made it." A.Einstein -
- -
- # Agenda - * Who are we? - * Upstream, Upstream, Upstream - * Towards Telco Cloud KPI -
- -
- # Who are we? - ![team](../img/OPNFV_testing_group.png) - This is a representation...not a governance! -
-
-
-

The test projects

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Test caseDescription
FunctestUmbrella project for Functional testing
YardstickUmbrella project for performance/qualification testing
CPerfSDN controller performance testing
StorPerfStorage performance testing
VSPerfVSwitch qualification
BottlenecksDetect possible bottlenecks
QTIPBenchmark to boost performance
-
- -
- # Functest - * Functional testing to validate scenarios - * Contributors (raise your hands) - * Functest presentation (here...1h ago :)) - * breakout sessions - * Today 3.10PM-3.40PM (vIMS evolution) 4PM (feature project integration), 5PM (Colorado status) - * tomorrow 11AM (API, test collection), 1PM (work Upstream), 3.15PM (D Release)(to be confirmed...) -
-
- # Yardstick - * Performance testing framework and NFVI Validation - * [Contributors](https://wiki.opnfv.org/display/yardstick/People) - * Yardstick presentation - * How to Do a Pre-deployment NFVI Validation Quickly and Efficiently? Wednesday, June 22, 4:00 PM - 4:30 PM - * Yardstick: A Pre-deployment NFVI Validation Tool. Thursday, June 23, 3:10 PM - 3:30 PM - * breakout sessions - * Tomorrow 2PM (Yardstick in C) -
-
- # CPerf - * SDN Controller Performance Testing - * Upstream test experts + standards/metrics experts - * Goal: Cross-controller perf tests in CI against realistic deployments -
-
- # StorPerf - * Cinder Volume Performance Testing - * Based on SNIA's SSD Test Specification - * Presentation - * StorPerf: Cinder Storage Performance Measurement. - Wednesday, June 22, 3:40 PM - 3:55 PM - * Breakout Session - * Tomorrow 10:15 (StorPerf Test Results Working Session) -
-
- # VSPerf -
-
- # Bottlenecks -
-
- # QTIP - - ## OPNFV Platform Benchmark - - - Bottom up - - testing baremetal components first - - More than result - - comparing against reference - - Setup-Test-Diagnose-Improve - - a closed loop to boost platform performance - - ## Breakout session - - - 14:30~15:45, June 21st/Tuesday - - Room Tegel -
-
-
-
- # Upstream, upstream, upstream.. -
-
- ## We are on the shoulders of the giants - * Rally (OpenStack) - * ODL, ONOS, ... - * RobotFramework, Teston - * ..... -
-
- ## Time to give back - ### How to improve work with testing upstream community -
- -
- -
- -
- # Towards Telco Cloud KPI? -
- -
- # Testing group goals - * Define common needs (naming, rules, tools...) - * Unify result collection, reporting, dashboarding, documentation - * Share best practice: docker, CLI, .. - * Identify testing domains - * Encourage testing in poorly covered areas -
- -
- # The ultimate goal: define Telco Cloud KPIs - * Provide methodology and test suites to qualify a Telco Cloud - * Provide reference benchs for plugfest/third party integration: compare apple with apple -
- -
- -
- # Thank you -
- -
-
- OPNFV logo -
-
- - - - - - - - diff --git a/docs/com/pres/index.template.html b/docs/com/pres/index.template.html deleted file mode 100755 index f9dd5a2d..00000000 --- a/docs/com/pres/index.template.html +++ /dev/null @@ -1,378 +0,0 @@ - - - - - - - OPNFV presentation - - - - - - - - - - - - - - - - - - - - - - - -
- -
- -
-
-

Project

-

subtitle

-


-
John Doe
-
-
whenever you want
-
- -
- -
-
- “Good job. Chris Price - ” -
-
- -
-

item

-
    -
  • another one
    • -
  • one more time
    • -
-
- - - -
-
- # Cool in markdow - ## really cool - - yep -
-
-

OPNFV

- OPNFV -
-
- # Markdown - ![OPNFV](https://www.opnfv.org/sites/all/themes/opnfv/logo.png "OPNFV") -
-
- - -
-

THE END

-

- - Try the online editor
- - Source code & documentation -

-
- -
-

Slides

-

- Not a coder? Not a problem. There's a fully-featured visual editor for authoring these, try it out at http://slides.com. -

-
- -
-

Point of View

-

- Press ESC to enter the slide overview. -

-

- Hold down alt and click on any element to zoom in on it using zoom.js. Alt + click anywhere to zoom back out. -

-
- -
-

Touch Optimized

-

- Presentations look great on touch devices, like mobile phones and tablets. Simply swipe through your slides. -

-
- -
-

Transition Styles

-

- You can select from different transitions, like:
- None - - Fade - - Slide - - Convex - - Concave - - Zoom -

-
- -
-

Themes

-

- reveal.js comes with a few themes built in:
- - Black (default) - - White - - League - - Sky - - Beige - - Simple
- Serif - - Blood - - Night - - Moon - - Solarized -

-
- -
-
-

Slide Backgrounds

-

- Set data-background="#dddddd" on a slide to change the background color. All CSS color formats are supported. -

- - Down arrow - -
-
-

Image Backgrounds

- 
<section data-background="image.png">
-
-
-

Tiled Backgrounds

- 
<section data-background="image.png" data-background-repeat="repeat" data-background-size="100px">
-
-
-
-

Video Backgrounds

- 
<section data-background-video="video.mp4,video.webm">
-
-
-
-

... and GIFs!

-
-
- -
-

Background Transitions

-

- Different background transitions are available via the backgroundTransition option. This one's called "zoom". -

- 
Reveal.configure({ backgroundTransition: 'zoom' })
-
- -
-

Background Transitions

-

- You can override background transitions per-slide. -

- 
<section data-background-transition="zoom">
-
- -
-

Pretty Code

- 

-function linkify( selector ) {
-  if( supports3DTransforms ) {
-
-    var nodes = document.querySelectorAll( selector );
-
-    for( var i = 0, len = nodes.length; i < len; i++ ) {
-      var node = nodes[i];
-
-      if( !node.className ) {
-        node.className += ' roll';
-      }
-    }
-  }
-}
-
-

Code syntax highlighting courtesy of highlight.js.

-
- -
-

Marvelous List

-
    -
  • No order here
    • -
  • Or here
    • -
  • Or here
    • -
  • Or here
    • -
-
- -
-

Fantastic Ordered List

-
    -
  1. One is smaller than...
    2. -
  2. Two is smaller than...
    3. -
  3. Three!
    4. -
-
- -
-

Tabular Tables

- - - - - - - - - - - - - - - - - - - - - - - - - -
ItemValueQuantity
Apples$17
Lemonade$218
Bread$32
-
- -
-

Clever Quotes

-

- These guys come in two forms, inline: - “The nice thing about standards is that there are so many to choose from” and block: -

-
- “For years there has been a theory that millions of monkeys typing at random on millions of typewriters would - reproduce the entire works of Shakespeare. The Internet has proven this theory to be untrue.” -
-
- -
-

Intergalactic Interconnections

-

- You can link between slides internally, - like this. -

-
- -
-

Speaker View

-

There's a speaker view. It includes a timer, preview of the upcoming slide as well as your speaker notes.

-

Press the S key to try it out.

- - -
- -
-

Export to PDF

-

Presentations can be exported to PDF, here's an example:

- -
- -
-

Global State

-

- Set data-state="something" on a slide and "something" - will be added as a class to the document element when the slide is open. This lets you - apply broader style changes, like switching the page background. -

-
- -
-

State Events

-

- Additionally custom events can be triggered on a per slide basis by binding to the data-state name. -

- 

-Reveal.addEventListener( 'customevent', function() {
-	console.log( '"customevent" has fired' );
-} );
-
-
- -
-

Take a Moment

-

- Press B or . on your keyboard to pause the presentation. This is helpful when you're on stage and want to take distracting slides off the screen. -

-
- -
-

Much more

- -
- - -
-
- OPNFV logo -
-
- - - - - - - - diff --git a/docs/com/pres/summit-Berlin.html b/docs/com/pres/summit-Berlin.html deleted file mode 100755 index 3d1f1253..00000000 --- a/docs/com/pres/summit-Berlin.html +++ /dev/null @@ -1,313 +0,0 @@ - - - - - - - OPNFV presentation - - - - - - - - - - - - - - - - - - - - - - - -
- -
- -
-




-

From Arno to Colorado

-

Functest: Many rivers to cross

-
-

Functest contributors

-
OPNFV Design Summit, 20/6/2016, Berlin
-
- -
- -
-
- “Bad programmers have all the answers. Good testers have all the questions. Gil Zilberfeld - ” -
-
- -
- # Agenda - * Functest - * Arno - * Brahmaputra - * Colorado - * D, E, F rivers and beyond -
- -
- # The dream team - ![team](../img/teamSummitBerlin.png) -
- -
-
- # Functest: what for? -
-
- ## Our goals - ### make sure OPNFV works...... -
-
- ## so concretely... - * Validate scenarios for the release - * Give confidence on OPNFV fresh releases - * Ensure consistancy towards installers - * Ensure End to End interoperability -
-
- ## and also - * Provide functional test suites to be replayed - * ~ post install sanity check - * usable for plugfests - * coherent with ETSI and Telco needs - * Provide test tools - * run tests - * collect info - * create dashboard -
-
- ## Our goals are not.... - * Performance testing (yardstick, Xperf ) - * Re-do tests done in specific projects - * storage/storeperf - * infra audit/qtip - * Compare results -
- -
- ## Our rules - * Re-use as much as possible upstream suites (Rally, Tempest, Robot, testOn..) - * Keep it simple... - * Automate (if not it is not a test...) - * Create new suites only if tests not covered - * Consider fields not addressed by existing test projects -
- -
- ## Our rules - * Installer neutral - * Controller neutral - * Black box approach -
- -
- -
-
- # Arno -
-
- ## Proof of concept - * 4 Testcases: vPing (userdata), Tempest, Rally, ODL - * 1 scenario: os-odl_l2-nofeature-ha - * 2 installers: Foreman/Fuel - * Lots of manual operations (reporting, documentation) -
-
-
-
- # Brahmaputra -
-
- ## More everything.... - * 7 Testcases: vPing (ssh), ONOS, vIMS, Tempest and Rally extended - * 3 feature projects: doctor, promise, sdnvpn - * 13 Scenarios - * 4 Installers (Apex, Compass, Fuel, Joid) - * Test result collection - * Automatic dashboard -
-
-

Jiras

- Jiras - Jiras -
- -
-

Brahmaputra Scenario matrix

- Jiras -
- -
- Comparison odl_l2-nofeature-ha scenario - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ProjectApexCompassFuelJoid
PODLF POD1Huawei USEricsson POD2Orange POD2
Functest2h501h452h352h30
Yardstick1h051h001h000h55
- * difference of duration due to POD and test suites -
- -
- ## Lessons learned - * Tempest/Rally - * harmonizing installer related OpenStack configuration differences - * creating customized test lists - * main challenge was resolving SDN controller interworking problems - * vIMS - * complete but complex test case - * very interesting to automate (pre MANO, most of Telco needs met there) -
- -
- -
-
- # Colorado -
- -
- ## What's new? - * New internal test cases: healthcheck, security - * New controler: OpenContrail? - * New feature projects (domino, multisites, movie, parser, moon, copper, models, onos-sfc...) - * ARM Support (congratulations to ENEA team!) - * Simplified feature project integration journey -
- -
- ## What's new? - * Slicing of the tests (healthcheck / Smoke / SDN controllers / Features / Components / VNFs) - * Better Test duration management - * Refactoring (repo, case management, test API) - * Automatic reporting - * Dashboard evolution to ELK - * CLI -
- - -
-

Automatic reporting

- Reporting -
Validated scenarios for Colorado -
- -
- -
-
- # Many rivers to cross - ## Functest beyond Colorado -
-
- ## More test cases - * New VNFs (vEPC, vCDN, vWhatever...) - * Multi sites (e.g bgpvpn with different back ends) - * Better test coverage -
-
- ## A Functional testing As a Service framework - * Scenario owner can select their relevant test cases - * Test duration estimation - * Agile dashboarding - * Analytics -
-
- ## Upstream - * Rally: time to give back more - * Functest description for ETSI? -
-
- -
-

Thank you

- tests -
- -
-
- OPNFV logo -
-
- - - - - - - - diff --git a/docs/com/pres/testapi.html b/docs/com/pres/testapi.html deleted file mode 100755 index c27c21de..00000000 --- a/docs/com/pres/testapi.html +++ /dev/null @@ -1,274 +0,0 @@ - - - - - - - OPNFV presentation - - - - - - - - - - - - - - - - - - - - - - - -
- -
- -
-




-

Test API

-

Clean, Easy, Complete

-
-

OPNFV testing community

-
OPNFV Design Summit, 20/6/2016, Berlin
-
-
- -
- # Agenda - * testAPI: what for? - * API overview - * API evolution - * Roadmap -
- -
-
- # test API: what for? -
-
- ## Consistant view for - * Resources (Pods) - * Projects - * Test cases - * Results -
- -
- ## Status Currently - * From many projects - * Many formats (log, html, json, ..) - * Many locations (in VM, Jumphost, external DB,...) -
-
- -
- -
- ## Help building - * Dashboards - * Automatic reporting -
-
- -
-
- # API overview -
-
- ## Storage structure - ![testapi](https://wiki.opnfv.org/download/attachments/2926452/results_collection_structure.png?version=1&modificationDate=1459196347000&api=v2 "OPNFV API page") -
-
- ## API in Brahmaputra - ![testapi](../img/testapi0.png) - https://wiki.opnfv.org/display/functest/Collection+Of+Test+Results -
- -
- ## API status in Brahmaputra - * Simple data models aggreed with the testing group - * No unit tests - * Wiki and rst documentation -
- -
- ## Lessons learned in B. - * Painful wiki documentation - * Simple result model - * Misuse version as scenario - * No version actually - * No overall criteria (PASS/FAIL) - * Difficult refactoring -
-
-

Colorado refactoring

-
    -
  • Unit tests
    • -
  • Update of the data model
    • -
  • Tornado_swagger for the documentation
    • -
  • URL and MongoDB collection renaming
    • -
  • Scripts to support databse backup/restore/update
    • -
  • Support installation
    • - -
-

All done in Colorado!

-
-
- -
-
- # API Evoluation -
- -
-

URI changes...

-
-

testresults.opnfv.org/testapi =>
testresults.opnfv.org/test/api/v1

- -

/test/api/v1/pods

-

/test/api/v1/projects

-

/test/api/v1/projects/qtip/cases

-

/test/api/v1/results

-
-
-
- ## One new field for pods - | Field | Colorado | - | :--- | :--- | - | role | community_pod, ci_pod, .. | -
-
- ## Changed fields for results - | Item | Brahmaputra | Colorado | - | :--- | :--- | :--- | - | date | creation_date | start_date and stop_date | - | version | used as scenario | opnfv release version | -
-
- -
-
- ## One new filter for results - | Item | Colorado | - | :--- | :--- | - | last | get last N results| -
-
- http://testresults.opnfv.org/test/swagger/spec.html - ![alt text](../img/testapi1.png "Test API swagger interface") -
-
-

unit tests

- 

-umry8364@umry8364-Latitude-E6400:~/Dev/OPNFV/releng/utils/test/result_collection_api$ ./run_test.sh 
-Tests running...
-WARNING:tornado.general:404 GET /dashboard/v1/results?case=vPing&pod=zte-pod1&version=C&installer=fuel&period=5 (127.0.0.1): Project name missing
-WARNING:tornado.access:404 GET /dashboard/v1/results?case=vPing&pod=zte-pod1&version=C&installer=fuel&period=5 (127.0.0.1) 2.30ms
-WARNING:tornado.general:400 POST /api/v1/projects (127.0.0.1): name missing
-............
-WARNING:tornado.access:400 POST /api/v1/projects (127.0.0.1) 1.13ms
-WARNING:tornado.access:403 PUT /api/v1/projects/functest/cases/vping_1 (127.0.0.1) 2.95ms
-WARNING:tornado.general:404 PUT /api/v1/projects/functest/cases/notFound (127.0.0.1): {'project_name': u'functest', 'name': u'notFound'} could not be found in table [testcases]
-WARNING:tornado.access:404 PUT /api/v1/projects/functest/cases/notFound (127.0.0.1) 2.85ms
-
-Ran 74 tests in 1.848s
-OK
-
-
-
- -
-
- # Roadmap -
-
- ## Roadmap - * Dockerize testAPI - * Automatic update of testAPI - * Command Line Interface - * Automatic update for pods/projects/testcases -
-
- # suggestions ... -
-
-

Thank you

- tests -
-
-
-
- OPNFV logo -
-
- - - - - - - - -- cgit 1.2.3-korg