1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
|
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. SPDX-License-Identifier: CC-BY-4.0
******************************
OPNFV FUNCTEST developer guide
******************************
.. toctree::
:numbered:
:maxdepth: 2
Version history
===============
+------------+----------+------------------+----------------------------------+
| **Date** | **Ver.** | **Author** | **Comment** |
| | | | |
+------------+----------+------------------+----------------------------------+
| 2017-01-23 | 1.0.0 | Morgan Richomme | Creation for Danube |
+------------+----------+------------------+----------------------------------+
| 2017-08-16 | 1.0.1 | Morgan Richomme | Adaptations for Euphrates |
| | | | - move generic part to Testing |
| | | | developer guide |
| | | | - move reporting part to functest|
| | | | user guide |
| | | | - update test case list |
| | | | - include auto generated core |
| | | | documentation |
+------------+----------+------------------+----------------------------------+
============
Introduction
============
Functest is a project dealing with functional testing.
Functest produces its own internal test cases but can also be considered
as a framework to support feature and VNF onboarding project testing.
Therefore there are many ways to contribute to Functest. You can:
* Develop new internal test cases
* Integrate the tests from your feature project
* Develop the framework to ease the integration of external test cases
Additional tasks involving Functest but addressing all the test projects
may also be mentioned:
* Develop the API / Test collection framework
* Develop dashboards or automatic reporting portals
This document describes how, as a developer, you may interact with the
Functest project. The first section details the main working areas of
the project. The Second part is a list of "How to" to help you to join
the Functest family whatever your field of interest is.
========================
Functest developer areas
========================
Functest High level architecture
================================
Functest is a project delivering test containers dedicated to OPNFV.
It includes the tools, the scripts and the test scenarios.
Until Danube, Functest produced 2 docker files based on Ubuntu 14.04:
* x86 Functest: https://hub.docker.com/r/opnfv/functest/
* aarch64 Functest: https://hub.docker.com/r/opnfv/functest_aarch64/
In Euphrates Alpine containers have been introduced in order to lighten the
container and manage testing slicing, the new containers are created according
to the different tiers:
* functest-core: https://hub.docker.com/r/opnfv/functest-core/
* functest-healthcheck: https://hub.docker.com/r/opnfv/functest-healthcheck/
* functest-smoke: https://hub.docker.com/r/opnfv/functest-smoke/
* functest-features: TODO
* functest-components: TODO
* functest-vnf: TODO
Functest can be described as follow::
+----------------------+
| |
| +--------------+ | +-------------------+
| | | | Public | |
| | Tools | +------------------+ OPNFV |
| | Scripts | | | System Under Test |
| | Scenarios | +------------------+ |
| | | | Management | |
| +--------------+ | +-------------------+
| |
| Functest Docker |
| |
+----------------------+
Functest internal test cases
============================
The internal test cases in Euphrates are:
* api_check
* cloudify_ims
* connection_check
* vping_ssh
* vping_userdata
* odl
* odl-netvirt
* odl-fds
* rally_full
* rally_sanity
* snaps_health_check
* tempest_full_parallel
* tempest_smoke_serial
* cloudify_ims
By internal, we mean that this particular test cases have been developed and/or
integrated by functest contributors and the associated code is hosted in the
Functest repository.
An internal case can be fully developed or a simple integration of
upstream suites (e.g. Tempest/Rally developed in OpenStack, or odl suites are
just integrated in Functest).
The structure of this repository is detailed in `[1]`_.
The main internal test cases are in the opnfv_tests subfolder of the
repository, the internal test cases are:
* sdn: odl, odl_netvirt, odl_fds, onos
* openstack: api_check, connection_check, snaps_health_check, vping_ssh, vping_userdata, tempest_*, rally_*
* vnf: cloudify_ims
If you want to create a new test case you will have to create a new folder under
the testcases directory (See next section for details).
Functest external test cases
============================
The external test cases are inherited from other OPNFV projects, especially the
feature projects.
The external test cases are:
* barometer
* bgpvpn
* doctor
* domino
* onos
* fds
* orchestra_ims
* parser
* promise
* refstack_defcore
* snaps_smoke
* functest-odl-sfc
* vyos_vrouter
External test cases integrated in previous versions but not released in
Euphrates:
* copper
* netready
* security_scan
The code to run these test cases is hosted in the repository of the project.
Functest framework
==================
Functest is a framework.
Historically Functest is released as a docker file, including tools, scripts and
a CLI to prepare the environment and run tests.
It simplifies the integration of external test suites in CI pipeline and provide
commodity tools to collect and display results.
Since Colorado, test categories also known as tiers have been created to
group similar tests, provide consistent sub-lists and at the end optimize
test duration for CI (see How To section).
The definition of the tiers has been agreed by the testing working group.
The tiers are:
* healthcheck
* smoke
* features
* components
* performance
* vnf
* stress
Note Functest deals with healthcheck, smoke, features, components and vnf tiers.
Performance and stress tiers are out of scope.
Functest abstraction classes
============================
In order to harmonize test integration, abstraction classes have been
introduced:
* testcase: base for any test case
* unit: run unit tests as test case
* feature: abstraction for feature project
* vnf: abstraction for vnf onboarding
The goal is to unify the way to run tests in Functest.
Feature, unit and vnf_base inherit from testcase::
+-----------------------------------------+
| |
| TestCase |
| |
| - init() |
| - run() |
| - publish_report() |
| - check_criteria() |
| |
+-----------------------------------------+
| |
V V
+--------------------+ +--------------+ +--------------------------+
| | | | | |
| feature | | unit | | vnf |
| | | | | |
| | | | | - prepare() |
| - execute() | | | | - deploy_orchestrator() |
| BashFeature class | | | | - deploy_vnf() |
| | | | | - test_vnf() |
| | | | | - clean() |
+--------------------+ +--------------+ +--------------------------+
Testcase
========
.. raw:: html
:url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.testcase.html
Feature
=======
.. raw:: html
:url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.feature.html
Unit
====
.. raw:: html
:url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.unit.html
VNF
===
.. raw:: html
:url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.vnf.html
see `Functest framework overview`_ to get code samples
Functest util classes
=====================
In order to simplify the creation of test cases, Functest develops also some
functions that can be used by any feature or internal test cases.
Several features are supported such as logger, configuration management and
Openstack capabilities (snapshot, clean, tacker,..).
These functions can be found under <repo>/functest/utils and can be described as
follows::
functest/utils/
|-- config.py
|-- constants.py
|-- decoratos.py
|-- env.py
|-- functest_utils.py
|-- openstack_clean.py
|-- openstack_snapshot.py
|-- openstack_tacker.py
`-- openstack_utils.py
Please note that it is possible to use snaps utils. SNAPS `[4]`_ is an OPNFV
project providing OpenStack utils.
TestAPI
=======
Functest is using the Test collection framework and the TestAPI developed by
the OPNFV community. See `OPNFV Test collection framework`_ for details.
Reporting
=========
A web page is automatically generated every day to display the status based on
jinja2 templates `[3]`_.
Dashboard
=========
Additional dashboarding is managed at the testing group level, see
`OPNFV Testing dashboard`_
=======
How TOs
=======
See `How to section`_ on Functest wiki
==========
References
==========
_`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest configuration guide
_`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide
_`[3]`: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting
_`[4]`: https://git.opnfv.org/snaps/
_`Functest framework overview` : http://testresults.opnfv.org/functest/framework/index.html
_`OPNFV Test collection framework`: TODO
_`OPNFV Testing dashboard`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba
_`How to section`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932
IRC support chan: #opnfv-functest
|
