blob: 65480a23ac728743159bc5ea0936921ed80c288c (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
|
==============================================
OPNFV Brahmaputra3.0 release note for Functest
==============================================
Abstract
========
This document describes the release note of Functest project.
License
=======
OPNFV Brahmaputra release note for Functest Docs
(c) by Morgan Richomme (Orange)
OPNFV Brahmaputra release note for Functest Docs
are licensed under a Creative Commons Attribution 4.0 International License.
You should have received a copy of the license along with this.
If not, see <http://creativecommons.org/licenses/by/4.0/>.
Version history
===============
+------------+----------+------------------+------------------------+
| **Date** | **Ver.** | **Author** | **Comment** |
| | | | |
+------------+----------+------------------+------------------------+
| 2016-02-25 | 1.0.0 | Morgan Richomme | Functest for B release |
| | | (Orange) | |
+------------+----------+------------------+------------------------+
| 2016-04-27 | 3.0.0 | Morgan Richomme | Add scenarios |
| | | (Orange) | |
+------------+----------+------------------+------------------------+
OPNFV Brahmaputra Release
=========================
Functest deals with functional testing of the OPNFV solution.
It includes test cases developed within the project and test cases developed in
other OPNFV projects and other upstream communities.
The internal test cases are:
* vPing ssh
* vPing userdata
* Tempest
* Rally
* vIMS
* ODL
The OPNFV projects integrated into Functest framework for automation are:
* Promise
* Doctor
* ONOSFW
* bgpvpn
Release Data
============
+--------------------------------------+--------------------------------------+
| **Project** | functest |
| | |
+--------------------------------------+--------------------------------------+
| **Repo/tag** | brahmaputra.1.0 |
| | |
+--------------------------------------+--------------------------------------+
| **Release designation** | Brahmaputra base release |
| | |
+--------------------------------------+--------------------------------------+
| **Release date** | February 26 2016 |
| | |
+--------------------------------------+--------------------------------------+
| **Purpose of the delivery** | Brahmaputra base release |
| | |
+--------------------------------------+--------------------------------------+
Deliverables
============
Software
--------
- The Functest Docker image
Documents
---------
- Installation/configuration guide
- User Guide
- Developer Guide
- Test results per scenario
Version change
==============
Feature evolution
-----------------
- support new scenarios
New features
------------
- minor bug fixes (formating)
- Modification of the configuration to support vPing_userdata on ONOS scenario
- Use serial option in Tempest to improve success rate
Scenario Matrix
===============
For Brahmaputra 3.0, Functest was succesfully tested on the following scenarios:
+----------------+---------+---------+---------+---------+
| Scenario | Apex | Compass | Fuel | Joid |
+================+=========+=========+=========+=========+
| odl_l2 | X | X | X | X |
+----------------+---------+---------+---------+---------+
| onos | | X | X | |
+----------------+---------+---------+---------+---------+
| nosdn | | X | X | |
+----------------+---------+---------+---------+---------+
| ovs (dpdk) | | | X | |
+----------------+---------+---------+---------+---------+
| kvm | | | X | |
+----------------+---------+---------+---------+---------+
| bgpvpn | X | | X | |
+----------------+---------+---------+---------+---------+
| sfc | | | X | |
+----------------+---------+---------+---------+---------+
Functest defines the success criteria when having at least 4 consecutive
successful runs of a given scenario from the Continuous Integration.
The success criteria is defined as follows:
* vPing SSH 100% OK
* vPing userdata 100% OK
* Tempest success rate > 90%
* Rally success rate > 90%
* ODL success rate = 100%
* ONOSFW success rate = 100%
* Promise success rate = 100%
* Bgpvpn success rate = 100%
* vIMS: deployement of the orchestrator and the vIMS VNF successful
Other scenarios are currently available but did not meet success criteria for
the release but might be added in the incremental scenario update of the
release.
Brahmaputra limitations
-----------------------
- Fuel and Apex Tempest success rate was below 90% but above 80% on some
scenarios. Some of the error causes were identified (workers, lack of IP)
- vIMS failed in CI for joid/odl_l2 scenario
- vPing userdata and vIMS excluded from onos scenario
- vPing_ssh and vIMS excluded from bgpvpn and kvm scenario
- None of the odl_l3 scenarios has been successful due to vPing ssh issue (ODL
bug reported https://bugs.opendaylight.org/show_bug.cgi?id=5582)
- apex/nosdn never run (not a target scenario) but probably succesful
- vPing SSH and vPing userdata no more run on CI since modification of bgpvpn
configuration regex.
See known issues section for details
Test and installer/scenario dependencies
========================================
It is not always possible to run all the test cases on all the scenarios. The
following table details the dependencies of the test cases per scenario.
+----------------+-------------+-------------+-------------+-------------+
| Test cases | Apex | Compass | Fuel | Joid |
+================+=============+=============+=============+=============+
| vPing SSH | all | all | all | all |
+----------------+-------------+-------------+-------------+-------------+
| vPing userdata | all | all | all | all |
+----------------+-------------+-------------+-------------+-------------+
| Tempest | all | all | all | all |
+----------------+-------------+-------------+-------------+-------------+
| Rally | all | all | all | all |
+----------------+-------------+-------------+-------------+-------------+
| ODL | all ODL | all ODL | all ODL | all ODL |
+----------------+-------------+-------------+-------------+-------------+
| ONOS | ONOS | ONOS | ONOS | ONOS |
+----------------+-------------+-------------+-------------+-------------+
| Promise | no | no | all | all |
+----------------+-------------+-------------+-------------+-------------+
| vIMS | all except | all except | all except | all except |
| | ONOS | ONOS | ONOS | ONOS |
+----------------+-------------+-------------+-------------+-------------+
| Doctor | all | no | no | no |
+----------------+-------------+-------------+-------------+-------------+
| Bgpvpn | all | no | all | no |
+----------------+-------------+-------------+-------------+-------------+
Test results
============
Test results are available in:
- test results document: http://artifacts.opnfv.org/functest/docs/results/index.html
- jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
- Test dashboards: http://testresults.opnfv.org/dashboard
Known issues
------------
- IPv6 issues in tempest suite:
- tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools
- Tempest code which doesn't reserve big enough allocation
- https://bugs.launchpad.net/tempest/+bug/1514457
- Lack of IP addresses available lead to several errors in different test cases
- vIMS (http://testresults.opnfv.org/reporting/vims/):
- the VM needs to have access to OpenStack API.
- Technical architecture may not allow this access (for security reasons)
- Orchestrator can be deployed but the vIMS VNF cannot
- That is the reason why it fails on joid/odl_l2 scenario on Orange POD 2
- case needs to be consolidated on new scenaios (bgpvpn, sfc, ovs)
Open JIRA tickets
=================
+------------------+-----------------------------------------+
| JIRA | Description |
+==================+=========================================+
| FUNCTEST-231 | vPing SSH no more run systematically |
| | in CI |
+------------------+-----------------------------------------+
| FUNCTEST-230 | Heat issues in Rally scenarios |
+------------------+-----------------------------------------+
| FUNCTEST-229 | Extend reporting to brahmaputra |
+------------------+-----------------------------------------+
| FUNCTEST-139 | prepare_env failed due to |
| | https://pypi.python.org/samples is not |
| | accessible |
+------------------+-----------------------------------------+
| FUNCTEST-135 | vPing scenario failed in odl_l3 scenario|
+------------------+-----------------------------------------+
Useful links
============
- wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
- Functest repo: https://git.opnfv.org/cgit/functest
- Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
- JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
- Wiki page for B Release: https://wiki.opnfv.org/functest_release_2
- Functest IRC chan: #opnfv-testperf
- Test dashboard: https://www.opnfv.org/opnfvtestgraphs/summary
- Functest dashboard: http://testresults.opnfv.org/dashboard
|
