1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
|
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
Overview of the Functest suites
===============================
Functest is the OPNFV project primarily targeting function testing.
In the Continuous Integration pipeline, it is launched after an OPNFV fresh
installation to validate and verify the basic functions of the
infrastructure.
The current list of test suites can be distributed over 5 main domains: VIM
(Virtualised Infrastructure Manager), Controllers (i.e. SDN Controllers),
Features, VNF (Virtual Network Functions) and MANO stacks.
Functest test suites are also distributed in the OPNFV testing categories:
healthcheck, smoke, features, components, performance, VNF, Stress tests.
All the Healthcheck and smoke tests of a given scenario must be succesful to
validate the scenario for the release.
+-------------+---------------+----------------+----------------------------------+
| Domain | Tier | Test case | Comments |
+=============+===============+================+==================================+
| VIM | healthcheck | healthcheck | Verify basic operation in VIM |
| | +----------------+----------------------------------+
| | | connection | Check OpenStack connectivity |
| | | _check | through SNAPS framework |
| | +----------------+----------------------------------+
| | | api_check | Check OpenStack API through |
| | | | SNAPS framework |
| +---------------+----------------+----------------------------------+
| | smoke | vPing_SSH | NFV "Hello World" using an SSH |
| | | | connection to a destination VM |
| | | | over a created floating IP |
| | | | address on the SUT Public / |
| | | | External network. Using the SSH |
| | | | connection a test script is then |
| | | | copied to the destination |
| | | | VM and then executed via SSH. |
| | | | The script will ping another |
| | | | VM on a specified IP address over|
| | | | the SUT Private Tenant network. |
| | +----------------+----------------------------------+
| | | vPing_userdata | Uses Ping with given userdata |
| | | | to test intra-VM connectivity |
| | | | over the SUT Private Tenant |
| | | | network. The correct operation |
| | | | of the NOVA Metadata service is |
| | | | also verified in this test. |
| | +----------------+----------------------------------+
| | | tempest_smoke | Generate and run a relevant |
| | | \_serial | Tempest Test Suite in smoke mode.|
| | | | The generated test set is |
| | | | dependent on the OpenStack |
| | | | deployment environment. |
| | +----------------+----------------------------------+
| | | rally_sanity | Run a subset of the OpenStack |
| | | | Rally Test Suite in smoke mode |
| | +----------------+----------------------------------+
| | | snaps_smoke | Run a subset of the OpenStack |
| | | | Rally Test Suite in smoke mode |
| +---------------+----------------+----------------------------------+
| | components | tempest_full | Generate and run a full set of |
| | | \_parallel | the OpenStack Tempest Test Suite.|
| | | | See the OpenStack reference test |
| | | | suite `[2]`_. The generated |
| | | | test set is dependent on the |
| | | | OpenStack deployment environment.|
| | +----------------+----------------------------------+
| | | rally_full | Run the OpenStack testing tool |
| | | | benchmarking OpenStack modules |
| | | | See the Rally documents `[3]`_. |
+-------------+---------------+----------------+----------------------------------+
| Controllers | smoke | odl | Opendaylight Test suite |
| | | | Limited test suite to check the |
| | | | basic neutron (Layer 2) |
| | | | operations mainly based on |
| | | | upstream testcases. See below |
| | | | for details |
| | +----------------+----------------------------------+
| | | onos | Test suite of ONOS L2 and L3 |
| | | | functions. |
| | | | See `ONOSFW User Guide`_ for |
| | | | details. |
+-------------+---------------+----------------+----------------------------------+
| Features | features | promise | Resource reservation and |
| | | | management project to identify |
| | | | NFV related requirements and |
| | | | realize resource reservation for |
| | | | future usage by capacity |
| | | | management of resource pools |
| | | | regarding compute, network and |
| | | | storage. |
| | | | See `Promise User Guide`_ for |
| | | | details. |
| | +----------------+----------------------------------+
| | | doctor | Doctor platform, as of Colorado |
| | | | release, provides the three |
| | | | features: |
| | | | * Immediate Notification |
| | | | * Consistent resource state |
| | | | awareness for compute host down |
| | | | * Valid compute host status |
| | | | given to VM owner |
| | | | See `Doctor User Guide`_ for |
| | | | details |
| | +----------------+----------------------------------+
| | | bgpvpn | Implementation of the OpenStack |
| | | | bgpvpn API from the SDNVPN |
| | | | feature project. It allows for |
| | | | the creation of BGP VPNs. |
| | | | See `SDNVPN User Guide`_ for |
| | | | details |
| | +----------------+----------------------------------+
| | | security_scan | Implementation of a simple |
| | | | security scan. (Currently |
| | | | available only for the Apex |
| | | | installer environment) |
| | +----------------+----------------------------------+
| | | onos-sfc | SFC testing for onos scenarios |
| | | | See `ONOSFW User Guide`_ for |
| | | | details |
| | +----------------+----------------------------------+
| | | odl-sfc | SFC testing for odl scenarios |
| | | | See `SFC User Guide`_ for details|
| | +----------------+----------------------------------+
| | | domino | Domino provides TOSCA template |
| | | | distribution service for network |
| | | | service and VNF descriptors |
| | | | among MANO components e.g., |
| | | | NFVO, VNFM, VIM, SDN-C, etc., |
| | | | as well as OSS/BSS functions. |
| | | | See `Domino User Guide`_ for |
| | | | details |
| | +----------------+----------------------------------+
| | | copper | Copper develops OPNFV platform |
| | | | support for policy management, |
| | | | using open source projects such |
| | | | as OpenStack Congress, focused |
| | | | on helping ensure that virtual |
| | | | infrastructure and the apps that |
| | | | execute on it comply with the |
| | | | configuration policy intent of |
| | | | service providers, developers, |
| | | | and end users. See more detail |
| | | | in the `Copper User Guide`_. |
| | +----------------+----------------------------------+
| | | multisites | Multisites |
| | | | See `Multisite User Guide`_ for |
| | | | details |
+-------------+---------------+----------------+----------------------------------+
| VNF | vnf | cloudify_ims | Example of a real VNF deployment |
| | | | to show the NFV capabilities of |
| | | | the platform. The IP Multimedia |
| | | | Subsytem is a typical Telco test |
| | | | case, referenced by ETSI. |
| | | | It provides a fully functional |
| | | | VoIP System |
| | +----------------+----------------------------------+
| | | opera_ims | vIMS deployment using openBaton |
| | +----------------+----------------------------------+
| | | orchestra_ims | vIMS deployment using open-O |
+ +---------------+----------------+----------------------------------+
| | | parser | Parser is an integration project |
| | | | which aims to provide |
| | | | placement/deployment templates |
| | | | translation for OPNFV platform, |
| | | | including TOSCA -> HOT, POLICY ->|
| | | | TOSCA and YANG -> TOSCA. |
| | | | See `Parser User Guide`_ for |
| | | | details |
+-------------+---------------+----------------+----------------------------------+
As shown in the above table, Functest is structured into different 'domains',
'tiers' and 'test cases'. Each 'test case' usually represents an actual
'Test Suite' comprised -in turn- of several test cases internally.
Test cases also have an implicit execution order. For example, if the early
'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke'
Tier testcases, there is little point to launch a full testcase execution round.
In Danube, we merged smoke and sdn controller tiers in smoke tier.
An overview of the Functest Structural Concept is depicted graphically below:
.. figure:: ../images/concepts_mapping_final.png
:align: center
:alt: Functest Concepts Structure
Some of the test cases are developed by Functest team members, whereas others
are integrated from upstream communities or other OPNFV projects. For example,
`Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the
OpenStack integration test suite and Functest is in charge of the selection,
integration and automation of those tests that fit suitably to OPNFV.
The Tempest test suite is the default OpenStack smoke test suite but no new test
cases have been created in OPNFV Functest.
The results produced by the tests run from CI are pushed and collected into a
NoSQL database. The goal is to populate the database with results from different
sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot
of a live Functest Dashboard is shown below:
** TODO **
.. figure:: ../images/FunctestDashboardDanube.png
:align: center
:alt: Functest Dashboard
Basic components (VIM, SDN controllers) are tested through their own suites.
Feature projects also provide their own test suites with different ways of
running their tests.
The notion of domain has been introduced in the description of the test cases
stored in the Database.
This parameters as well as possible tags can be used for the Test case catalog.
vIMS test case was integrated to demonstrate the capability to deploy a
relatively complex NFV scenario on top of the OPNFV infrastructure.
Functest considers OPNFV as a black box. As of Danube release the OPNFV
offers a lot of potential combinations:
* 3 controllers (OpenDaylight, ONOS, OpenContrail)
* 4 installers (Apex, Compass, Fuel, Joid)
Most of the tests are runnable by any combination, but some tests might have
restrictions imposed by the utilized installers or due to the available
deployed features. The system uses the environment variables (INSTALLER_IP and
DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given
environment.
A convenience Functest CLI utility is also available to simplify setting up the
Functest evironment, management of the OpenStack environment (e.g. resource
clean-up) and for executing tests.
The Functest CLI organised the testcase into logical Tiers, which contain in
turn one or more testcases. The CLI allows execution of a single specified
testcase, all test cases in a specified Tier, or the special case of execution
of **ALL** testcases. The Functest CLI is introduced in more detail in the
section `Executing the functest suites`_ of this document.
.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
.. _`Copper User Guide`: http://artifacts.opnfv.org/copper/colorado/docs/userguide/index.html
.. _`Doctor User Guide`: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html
.. _`Promise User Guide`: http://artifacts.opnfv.org/promise/colorado/docs/userguide/index.html
.. _`ONOSFW User Guide`: http://artifacts.opnfv.org/onosfw/colorado/userguide/index.html
.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html
.. _`Domino User Guide`: http://artifacts.opnfv.org/domino/docs/userguide-single/index.html
.. _`Parser User Guide`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html
.. _`Functest Dashboard`: http://testresults.opnfv.org/kibana_dashboards/
.. _`SFC User Guide`: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html
.. _`Multisite User Guide`: http://artifacts.opnfv.org/multisite/docs/userguide/index.html
|
