--- docs/com/pres/Summit/Berlin-2016/conversation.html | 251 ----------------- .../com/pres/Summit/Berlin-2016/summit-Berlin.html | 313 --------------------- docs/com/pres/Summit/Berlin-2016/testapi.html | 274 ------------------ 3 files changed, 838 deletions(-) delete mode 100644 docs/com/pres/Summit/Berlin-2016/conversation.html delete mode 100644 docs/com/pres/Summit/Berlin-2016/summit-Berlin.html delete mode 100644 docs/com/pres/Summit/Berlin-2016/testapi.html (limited to 'docs/com/pres/Summit') diff --git a/docs/com/pres/Summit/Berlin-2016/conversation.html b/docs/com/pres/Summit/Berlin-2016/conversation.html deleted file mode 100644 index 356c2ad1..00000000 --- a/docs/com/pres/Summit/Berlin-2016/conversation.html +++ /dev/null @@ -1,251 +0,0 @@ - - - - - - - OPNFV presentation - - - - - - - - - - - - - - - - - - - - - - - -

- -

-

-

Conversation with the Testing Community

-

-

OPNFV testing community

OPNFV Design Summit, 20/6/2016, Berlin

-
- -

- -

- >“You make experiments and I make theories. Do you know the difference? A theory is something nobody believes, except the person who made it. An experiment is something everybody believes, except the person who made it." A.Einstein -

- -

- # Agenda - * Who are we? - * Upstream, Upstream, Upstream - * Towards Telco Cloud KPI -

- -

- # Who are we? - ![team](../../../img/OPNFV_testing_group.png) - This is a representation...not a governance! -

The test projects

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Test case	Description
Functest	Umbrella project for Functional testing
Yardstick	Umbrella project for performance/qualification testing
CPerf	SDN controller performance testing
StorPerf	Storage performance testing
VSPerf	VSwitch qualification
Bottlenecks	Detect possible bottlenecks
QTIP	Benchmark to boost performance

- -

- # Functest - * Functional testing to validate scenarios - * Contributors (raise your hands) - * Functest presentation (here...1h ago :)) - * breakout sessions - * Today 3.10PM-3.40PM (vIMS evolution) 4PM (feature project integration), 5PM (Colorado status) - * tomorrow 11AM (API, test collection), 1PM (work Upstream), 3.15PM (D Release)(to be confirmed...) -

- # Yardstick - * Performance testing framework and NFVI Validation - * [Contributors](https://wiki.opnfv.org/display/yardstick/People) - * Yardstick presentation - * How to Do a Pre-deployment NFVI Validation Quickly and Efficiently? Wednesday, June 22, 4:00 PM - 4:30 PM - * Yardstick: A Pre-deployment NFVI Validation Tool. Thursday, June 23, 3:10 PM - 3:30 PM - * breakout sessions - * Tomorrow 2PM (Yardstick in C) -

- # CPerf - * SDN Controller Performance Testing - * Upstream test experts + standards/metrics experts - * Goal: Cross-controller perf tests in CI against realistic deployments -

- # StorPerf - * Cinder Volume Performance Testing - * Based on SNIA's SSD Test Specification - * Presentation - * StorPerf: Cinder Storage Performance Measurement. - Wednesday, June 22, 3:40 PM - 3:55 PM - * Breakout Session - * Tomorrow 10:15 (StorPerf Test Results Working Session) -

- # VSPerf -

- # Bottlenecks -

- # QTIP - - ## OPNFV Platform Benchmark - - - Bottom up - - testing baremetal components first - - More than result - - comparing against reference - - Setup-Test-Diagnose-Improve - - a closed loop to boost platform performance - - ## Breakout session - - - 14:30~15:45, June 21st/Tuesday - - Room Tegel -

- # Upstream, upstream, upstream.. -

- ## We are on the shoulders of the giants - * Rally (OpenStack) - * ODL, ONOS, ... - * RobotFramework, Teston - * ..... -

- ## Time to give back - ### How to improve work with testing upstream community -

- -

- # Towards Telco Cloud KPI? -

- -

- # Testing group goals - * Define common needs (naming, rules, tools...) - * Unify result collection, reporting, dashboarding, documentation - * Share best practice: docker, CLI, .. - * Identify testing domains - * Encourage testing in poorly covered areas -

- -

- # The ultimate goal: define Telco Cloud KPIs - * Provide methodology and test suites to qualify a Telco Cloud - * Provide reference benchs for plugfest/third party integration: compare apple with apple -

- -

- # Thank you -

- -

- - - - - - - - diff --git a/docs/com/pres/Summit/Berlin-2016/summit-Berlin.html b/docs/com/pres/Summit/Berlin-2016/summit-Berlin.html deleted file mode 100644 index 97fa66c1..00000000 --- a/docs/com/pres/Summit/Berlin-2016/summit-Berlin.html +++ /dev/null @@ -1,313 +0,0 @@ - - - - - - - OPNFV presentation - - - - - - - - - - - - - - - - - - - - - - - -

- -

-

-

From Arno to Colorado

Functest: Many rivers to cross

-
-

Functest contributors

OPNFV Design Summit, 20/6/2016, Berlin

-
- -

- -

- “Bad programmers have all the answers. Good testers have all the questions. Gil Zilberfeld - ” -

- -

- # Agenda - * Functest - * Arno - * Brahmaputra - * Colorado - * D, E, F rivers and beyond -

- -

- # The dream team - ![team](../../../img/teamSummitBerlin.png) -

- -

- # Functest: what for? -

- ## Our goals - ### make sure OPNFV works...... -

- ## so concretely... - * Validate scenarios for the release - * Give confidence on OPNFV fresh releases - * Ensure consistancy towards installers - * Ensure End to End interoperability -

- ## and also - * Provide functional test suites to be replayed - * ~ post install sanity check - * usable for plugfests - * coherent with ETSI and Telco needs - * Provide test tools - * run tests - * collect info - * create dashboard -

- ## Our goals are not.... - * Performance testing (yardstick, Xperf ) - * Re-do tests done in specific projects - * storage/storeperf - * infra audit/qtip - * Compare results -

- -

- ## Our rules - * Re-use as much as possible upstream suites (Rally, Tempest, Robot, testOn..) - * Keep it simple... - * Automate (if not it is not a test...) - * Create new suites only if tests not covered - * Consider fields not addressed by existing test projects -

- -

- ## Our rules - * Installer neutral - * Controller neutral - * Black box approach -

- -

- # Arno -

- ## Proof of concept - * 4 Testcases: vPing (userdata), Tempest, Rally, ODL - * 1 scenario: os-odl_l2-nofeature-ha - * 2 installers: Foreman/Fuel - * Lots of manual operations (reporting, documentation) -

- # Brahmaputra -

- ## More everything.... - * 7 Testcases: vPing (ssh), ONOS, vIMS, Tempest and Rally extended - * 3 feature projects: doctor, promise, sdnvpn - * 13 Scenarios - * 4 Installers (Apex, Compass, Fuel, Joid) - * Test result collection - * Automatic dashboard -

Jiras

- -

Brahmaputra Scenario matrix

- -

- Comparison odl_l2-nofeature-ha scenario - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Project	Apex	Compass	Fuel	Joid
POD	LF POD1	Huawei US	Ericsson POD2	Orange POD2
Functest	2h50	1h45	2h35	2h30
Yardstick	1h05	1h00	1h00	0h55

- * difference of duration due to POD and test suites -

- -

- ## Lessons learned - * Tempest/Rally - * harmonizing installer related OpenStack configuration differences - * creating customized test lists - * main challenge was resolving SDN controller interworking problems - * vIMS - * complete but complex test case - * very interesting to automate (pre MANO, most of Telco needs met there) -

- -

- # Colorado -

- -

- ## What's new? - * New internal test cases: healthcheck, security - * New controler: OpenContrail? - * New feature projects (domino, multisites, movie, parser, moon, copper, models, onos-sfc...) - * ARM Support (congratulations to ENEA team!) - * Simplified feature project integration journey -

- -

- ## What's new? - * Slicing of the tests (healthcheck / Smoke / SDN controllers / Features / Components / VNFs) - * Better Test duration management - * Refactoring (repo, case management, TestAPI) - * Automatic reporting - * Dashboard evolution to ELK - * CLI -

- - -

Automatic reporting

-
Validated scenarios for Colorado -

- -

- # Many rivers to cross - ## Functest beyond Colorado -

- ## More test cases - * New VNFs (vEPC, vCDN, vWhatever...) - * Multi sites (e.g bgpvpn with different back ends) - * Better test coverage -

- ## A Functional testing As a Service framework - * Scenario owner can select their relevant test cases - * Test duration estimation - * Agile dashboarding - * Analytics -

- ## Upstream - * Rally: time to give back more - * Functest description for ETSI? -

- -

Thank you

- -

- - - - - - - - diff --git a/docs/com/pres/Summit/Berlin-2016/testapi.html b/docs/com/pres/Summit/Berlin-2016/testapi.html deleted file mode 100644 index c40637cb..00000000 --- a/docs/com/pres/Summit/Berlin-2016/testapi.html +++ /dev/null @@ -1,274 +0,0 @@ - - - - - - - OPNFV presentation - - - - - - - - - - - - - - - - - - - - - - - -

- -

-

-

TestAPI

Clean, Easy, Complete

-
-

OPNFV testing community

OPNFV Design Summit, 20/6/2016, Berlin

-
-

- -

- # Agenda - * TestAPI: what for? - * API overview - * API evolution - * Roadmap -

- -

- # TestAPI: what for? -

- ## Consistant view for - * Resources (Pods) - * Projects - * Test cases - * Results -

- -

- ## Status Currently - * From many projects - * Many formats (log, html, json, ..) - * Many locations (in VM, Jumphost, external DB,...) -

- -

- ## Help building - * Dashboards - * Automatic reporting -

- -

- # API overview -

- ## Storage structure - ![TestAPI](https://wiki.opnfv.org/download/attachments/2926452/results_collection_structure.png?version=1&modificationDate=1459196347000&api=v2 "OPNFV API page") -

- ## API in Brahmaputra - ![TestAPI](../../../img/testapi0.png) - https://wiki.opnfv.org/display/functest/Collection+Of+Test+Results -

- -

- ## API status in Brahmaputra - * Simple data models aggreed with the testing group - * No unit tests - * Wiki and rst documentation -

- -

- ## Lessons learned in B. - * Painful wiki documentation - * Simple result model - * Misuse version as scenario - * No version actually - * No overall criteria (PASS/FAIL) - * Difficult refactoring -

Colorado refactoring

Unit tests
Update of the data model
Tornado_swagger for the documentation
URL and MongoDB collection renaming
Scripts to support databse backup/restore/update
Support installation

All done in Colorado!

- -

- # API Evoluation -

- -

URI changes...

testresults.opnfv.org/TestAPI =>
testresults.opnfv.org/test/api/v1

- -

/test/api/v1/pods

/test/api/v1/projects

/test/api/v1/projects/qtip/cases

/test/api/v1/results

- ## One new field for pods - | Field | Colorado | - | :--- | :--- | - | role | community_pod, ci_pod, .. | -

- ## Changed fields for results - | Item | Brahmaputra | Colorado | - | :--- | :--- | :--- | - | date | creation_date | start_date and stop_date | - | version | used as scenario | opnfv release version | -

- -

- ## One new filter for results - | Item | Colorado | - | :--- | :--- | - | last | get last N results| -

- http://testresults.opnfv.org/test/swagger/spec.html - ![alt text](../../../img/testapi1.png "TestAPI swagger interface") -

unit tests


-umry8364@umry8364-Latitude-E6400:~/Dev/OPNFV/releng/utils/test/result_collection_api$ ./run_test.sh 
-Tests running...
-WARNING:tornado.general:404 GET /dashboard/v1/results?case=vPing&pod=zte-pod1&version=C&installer=fuel&period=5 (127.0.0.1): Project name missing
-WARNING:tornado.access:404 GET /dashboard/v1/results?case=vPing&pod=zte-pod1&version=C&installer=fuel&period=5 (127.0.0.1) 2.30ms
-WARNING:tornado.general:400 POST /api/v1/projects (127.0.0.1): name missing
-............
-WARNING:tornado.access:400 POST /api/v1/projects (127.0.0.1) 1.13ms
-WARNING:tornado.access:403 PUT /api/v1/projects/functest/cases/vping_1 (127.0.0.1) 2.95ms
-WARNING:tornado.general:404 PUT /api/v1/projects/functest/cases/notFound (127.0.0.1): {'project_name': u'functest', 'name': u'notFound'} could not be found in table [testcases]
-WARNING:tornado.access:404 PUT /api/v1/projects/functest/cases/notFound (127.0.0.1) 2.85ms
-
-Ran 74 tests in 1.848s
-OK
-

- -

- # Roadmap -

- ## Roadmap - * Dockerize TestAPI - * Automatic update of TestAPI - * Command Line Interface - * Automatic update for pods/projects/testcases -

- # suggestions ... -

Thank you

- -

- - - - - - - - -- cgit 1.2.3-korg