1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
|
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
Executing the functest suites
=============================
Manual testing
--------------
This section assumes the following:
* The Functest Docker container is running
* The docker prompt is shown
* The Functest environment is ready (Functest CLI command 'functest env prepare'
has been executed)
If any of the above steps are missing please refer to the Functest Config Guide
as they are a prerequisite and all the commands explained in this section **must** be
performed **inside the container**.
Note: In Colorado release, the scripts **run_tests.sh** is now replaced with a
new Functest CLI. One difference, is that tests run through the Functest CLI
will always clean-up OpenStack resources. See the `Troubleshooting`_ section of this
document, where this difference is discussed.
The Functest CLI offers two commands (functest tier ...) and (functest testcase ... )
for the execution of Test Tiers or Test Cases::
root@22e436918db0:~/repos/functest/ci# functest tier --help
Usage: functest tier [OPTIONS] COMMAND [ARGS]...
Options:
-h, --help Show this message and exit.
Commands:
get-tests Prints the tests in a tier.
list Lists the available tiers.
run Executes all the tests within a tier.
show Shows information about a tier.
root@22e436918db0:~/repos/functest/ci# functest testcase --help
Usage: functest testcase [OPTIONS] COMMAND [ARGS]...
Options:
-h, --help Show this message and exit.
Commands:
list Lists the available testcases.
run Executes a test case.
show Shows information about a test case.
More details on the existing Tiers and Test Cases can be seen with the 'list'
command::
root@22e436918db0:~/repos/functest/ci# functest tier list
- 0. healthcheck:
['healthcheck']
- 1. smoke:
['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity']
- 2. sdn_suites:
['odl']
- 3. features:
['doctor', 'security_scan']
- 4. openstack:
['tempest_full_parallel', 'rally_full']
- 5. vnf:
['vims']
and
root@22e436918db0:~/repos/functest/ci# functest testcase list
healthcheck
vping_ssh
vping_userdata
tempest_smoke_serial
rally_sanity
odl
doctor
security_scan
tempest_full_parallel
rally_full
vims
More specific details on specific Tiers or Test Cases can be seen wih the
'show' command::
root@22e436918db0:~/repos/functest/ci# functest tier show smoke
+======================================================================+
| Tier: smoke |
+======================================================================+
| Order: 1 |
| CI Loop: (daily)|(weekly) |
| Description: |
| Set of basic Functional tests to validate the OpenStack |
| deployment. |
| Test cases: |
| - vping_ssh |
| - vping_userdata |
| - tempest_smoke_serial |
| - rally_sanity |
| |
+----------------------------------------------------------------------+
and
root@22e436918db0:~/repos/functest/ci# functest testcase show tempest_smoke_serial
+======================================================================+
| Testcase: tempest_smoke_serial |
+======================================================================+
| Description: |
| This test case runs the smoke subset of the OpenStack Tempest |
| suite. The list of test cases is generated by Tempest |
| automatically and depends on the parameters of the OpenStack |
| deplopyment. |
| Dependencies: |
| - Installer: |
| - Scenario : |
| |
+----------------------------------------------------------------------+
To execute a Test Tier or Test Case, the 'run' command is used::
root@22e436918db0:~/repos/functest/ci# functest tier run healthcheck
Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t healthcheck'
2016-06-30 11:44:56,933 - run_tests - INFO - Sourcing the OpenStack RC file...
2016-06-30 11:44:56,937 - run_tests - INFO - ############################################
2016-06-30 11:44:56,938 - run_tests - INFO - Running tier 'healthcheck'
2016-06-30 11:44:56,938 - run_tests - INFO - ############################################
2016-06-30 11:44:56,938 - run_tests - INFO - ============================================
2016-06-30 11:44:56,938 - run_tests - INFO - Running test case 'healthcheck'...
2016-06-30 11:44:56,938 - run_tests - INFO - ============================================
2016-06-30 11:44:56,953 - healtcheck - INFO - Testing Keystone API...
2016-06-30 11:45:05,351 - healtcheck - INFO - ...Keystone OK!
2016-06-30 11:45:05,354 - healtcheck - INFO - Testing Glance API...
2016-06-30 11:45:29,746 - healtcheck - INFO - ... Glance OK!
2016-06-30 11:45:29,749 - healtcheck - INFO - Testing Cinder API...
2016-06-30 11:45:37,502 - healtcheck - INFO - ...Cinder OK!
2016-06-30 11:45:37,505 - healtcheck - INFO - Testing Neutron API...
2016-06-30 11:45:39,664 - healtcheck - INFO - External network found. ccd98ad6-d34a-4768-b03c-e28ecfcd51ca
2016-06-30 11:45:39,667 - healtcheck - INFO - 1. Create Networks...
2016-06-30 11:45:44,227 - healtcheck - INFO - 2. Create subnets...
2016-06-30 11:45:46,805 - healtcheck - INFO - 4. Create Routers...
2016-06-30 11:45:54,261 - healtcheck - INFO - ...Neutron OK!
2016-06-30 11:45:54,264 - healtcheck - INFO - Testing Nova API...
2016-06-30 11:47:12,272 - healtcheck - INFO - ...Nova OK!
2016-06-30 11:47:12,274 - healtcheck - INFO - Checking if instances get an IP from DHCP...
:
:
2016-06-30 11:48:17,832 - healtcheck - INFO - ...DHCP OK!
2016-06-30 11:48:17,835 - healtcheck - INFO - Health check passed!
2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
2016-06-30 11:48:17,837 - clean_openstack - INFO - Cleaning OpenStack resources...
2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
Version 1 is deprecated, use alternative version 2 instead.
WARNING:cinderclient.api_versions:Version 1 is deprecated, use alternative version 2 instead.
2016-06-30 11:48:18,272 - clean_openstack - INFO - Removing Nova instances...
2016-06-30 11:48:24,439 - clean_openstack - INFO - -------------------------------------------
2016-06-30 11:48:24,440 - clean_openstack - INFO - Removing Glance images...
2016-06-30 11:48:35,853 - clean_openstack - INFO - -------------------------------------------
2016-06-30 11:48:35,854 - clean_openstack - INFO - Removing Cinder volumes...
2016-06-30 11:48:37,344 - clean_openstack - INFO - -------------------------------------------
2016-06-30 11:48:37,344 - clean_openstack - INFO - Removing floating IPs...
2016-06-30 11:48:37,467 - clean_openstack - INFO - -------------------------------------------
2016-06-30 11:48:37,467 - clean_openstack - INFO - Removing Neutron objects
2016-06-30 11:48:53,633 - clean_openstack - INFO - -------------------------------------------
2016-06-30 11:48:53,633 - clean_openstack - INFO - Removing Security groups...
2016-06-30 11:48:53,689 - clean_openstack - INFO - -------------------------------------------
2016-06-30 11:48:53,689 - clean_openstack - INFO - Removing Users...
2016-06-30 11:48:54,444 - clean_openstack - INFO - -------------------------------------------
2016-06-30 11:48:54,444 - clean_openstack - INFO - Removing Tenants...
2016-06-30 11:48:54,711 - clean_openstack - INFO - -------------------------------------------
and
root@22e436918db0:~/repos/functest/ci# functest testcase run vping_ssh
Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t vping_ssh'
2016-06-30 11:50:31,861 - run_tests - INFO - Sourcing the OpenStack RC file...
2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
2016-06-30 11:50:31,865 - run_tests - INFO - Running test case 'vping_ssh'...
2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
2016-06-30 11:50:32,977 - vping_ssh - INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.4-x86_64-disk.img'...
2016-06-30 11:50:45,470 - vping_ssh - INFO - Creating neutron network vping-net...
2016-06-30 11:50:47,645 - vping_ssh - INFO - Creating security group 'vPing-sg'...
2016-06-30 11:50:48,843 - vping_ssh - INFO - Using existing Flavor 'm1.small'...
2016-06-30 11:50:48,927 - vping_ssh - INFO - vPing Start Time:'2016-06-30 11:50:48'
2016-06-30 11:50:48,927 - vping_ssh - INFO - Creating instance 'opnfv-vping-1'...
2016-06-30 11:51:34,664 - vping_ssh - INFO - Instance 'opnfv-vping-1' is ACTIVE.
2016-06-30 11:51:34,818 - vping_ssh - INFO - Adding 'opnfv-vping-1' to security group 'vPing-sg'...
2016-06-30 11:51:35,209 - vping_ssh - INFO - Creating instance 'opnfv-vping-2'...
2016-06-30 11:52:01,439 - vping_ssh - INFO - Instance 'opnfv-vping-2' is ACTIVE.
2016-06-30 11:52:01,439 - vping_ssh - INFO - Adding 'opnfv-vping-2' to security group 'vPing-sg'...
2016-06-30 11:52:01,754 - vping_ssh - INFO - Creating floating IP for VM 'opnfv-vping-2'...
2016-06-30 11:52:01,969 - vping_ssh - INFO - Floating IP created: '10.17.94.140'
2016-06-30 11:52:01,969 - vping_ssh - INFO - Associating floating ip: '10.17.94.140' to VM 'opnfv-vping-2'
2016-06-30 11:52:02,792 - vping_ssh - INFO - Trying to establish SSH connection to 10.17.94.140...
2016-06-30 11:52:19,915 - vping_ssh - INFO - Waiting for ping...
2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing detected!
2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing duration:'92.2' s.
2016-06-30 11:52:21,109 - vping_ssh - INFO - vPing OK
2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
2016-06-30 11:52:21,153 - clean_openstack - INFO - Cleaning OpenStack resources...
2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
Version 1 is deprecated, use alternative version 2 instead.
:
:
etc.
To list the test cases which are part of a specific Test Tier, the 'get-tests'
command is used with 'functest tier'::
root@22e436918db0:~/repos/functest/ci# functest tier get-tests sdn_suites
Test cases in tier 'sdn_suites':
['odl']
Please note that for some scenarios some test cases might not be launched.
For example, the last example displayed only the 'odl' testcase for the given
environment. In this particular system the deployment does not support the 'onos' SDN
Controller Test Case; for example.
**Important** If you use the command 'functest tier run <tier_name>', then the
Functest CLI utility will call **all valid Test Cases**, which belong to the
specified Test Tier, as relevant to scenarios deployed to the SUT environment.
Thus, the Functest CLI utility calculates automatically which tests can be
executed and which cannot, given the environment variable **DEPLOY_SCENARIO**,
which is passed in to the Functest docker container.
Currently, the Functest CLI command 'functest testcase run <testcase_name>', supports
two possibilities::
* Run a single Test Case, specified by a valid choice of <testcase_name>
* Run ALL test Test Cases (for all Tiers) by specifying <testcase_name> = 'all'
Example::
root@22e436918db0:~/repos/functest/ci# functest testcase run all
Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t all'
2016-06-30 12:03:28,628 - run_tests - INFO - Sourcing the OpenStack RC file...
2016-06-30 12:03:28,634 - run_tests - INFO - Tiers to be executed:
- 0. healthcheck:
['healthcheck']
- 1. smoke:
['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity']
- 2. sdn_suites:
['odl']
- 3. features:
['doctor', 'security_scan']
- 4. openstack:
['tempest_full_parallel', 'rally_full']
- 5. vnf:
['vims']
2016-06-30 12:03:28,634 - run_tests - INFO - ############################################
2016-06-30 12:03:28,635 - run_tests - INFO - Running tier 'healthcheck'
2016-06-30 12:03:28,635 - run_tests - INFO - ############################################
2016-06-30 12:03:28,635 - run_tests - INFO - ============================================
2016-06-30 12:03:28,635 - run_tests - INFO - Running test case 'healthcheck'...
2016-06-30 12:03:28,635 - run_tests - INFO - ============================================
2016-06-30 12:03:28,651 - healtcheck - INFO - Testing Keystone API...
2016-06-30 12:03:36,676 - healtcheck - INFO - ...Keystone OK!
2016-06-30 12:03:36,679 - healtcheck - INFO - Testing Glance API...
:
:
etc.
Functest includes a cleaning mechanism in order to remove all the OpenStack
resources except those present before running any test. The script
*$REPOS_DIR/functest/functest/utils/generate_defaults.py* is called once when setting up
the Functest environment (i.e. CLI command 'functest env prepare') to snapshot
all the OpenStack resources (images, networks, volumes, security groups, tenants,
users) so that an eventual cleanup does not remove any of these defaults.
The script **clean_openstack.py** which is located in
*$REPOS_DIR/functest/functest/utils/* is normally called after a test execution. It is
in charge of cleaning the OpenStack resources that are not specified in the
defaults file generated previously which is stored in
*/home/opnfv/functest/conf/os_defaults.yaml* in the Functest docker container.
It is important to mention that if there are new OpenStack resources created
manually after preparing the Functest environment, they will be removed, unless
you use the special method of invoking the test case with specific suppression
of clean up. (See the `Troubleshooting`_ section).
The reason to include this cleanup meachanism in Functest is because some
test suites such as Tempest or Rally create a lot of resources (users,
tenants, networks, volumes etc.) that are not always properly cleaned, so this
function has been set to keep the system as clean as it was before a
full Functest execution.
Although the Functest CLI provides an easy way to run any test, it is possible to
do a direct call to the desired test script. For example:
python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/vPing/vPing_ssh.py -d
Automated testing
-----------------
As mentioned previously, the Functest Docker container preparation as well as
invocation of Test Cases can be called within the container from the Jenkins CI
system. There are 3 jobs that automate the whole process. The first job runs all
the tests referenced in the daily loop (i.e. that must been run daily), the second
job runs the tests referenced in the weekly loop (usually long duration tests run
once a week maximum) and the third job allows testing test suite by test suite specifying
the test suite name. The user may also use either of these Jenkins jobs to execute
the desired test suites.
One of the most challenging task in the Colorado release consists
in dealing with lots of scenarios and installers. Thus, when the tests are
automatically started from CI, a basic algorithm has been created in order to
detect whether a given test is runnable or not on the given scenario.
Some Functest test suites cannot be systematically run (e.g. ODL suite can not
be run on an ONOS scenario). Moreover since Colorado, we also introduce the
notion of daily/weekly in order to save CI time and avoid running systematically
long duration tests.
CI provides some useful information passed to the container as environment
variables:
* Installer (apex|compass|fuel|joid), stored in INSTALLER_TYPE
* Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP
* The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with
* controller = (odl|onos|ocl|nosdn)
* feature = (ovs(dpdk)|kvm|sfc|bgpvpn|moon|multisites)
* mode = (ha|noha)
The constraints per test case are defined in the Functest configuration file
*/home/opnfv/repos/functest/functest/ci/testcases.yaml*::
tiers:
-
name: healthcheck
order: 0
ci_loop: '(daily)|(weekly)'
description : >-
First tier to be executed to verify the basic
operations in the VIM.
testcases:
-
name: healthcheck
criteria: 'status == "PASS"'
blocking: true
description: >-
This test case verifies the basic OpenStack services like
Keystone, Glance, Cinder, Neutron and Nova.
dependencies:
installer: ''
scenario: ''
-
name: smoke
order: 1
ci_loop: '(daily)|(weekly)'
description : >-
Set of basic Functional tests to validate the OpenStack deployment.
testcases:
-
name: vping_ssh
criteria: 'status == "PASS"'
blocking: true
description: >-
This test case verifies: 1) SSH to an instance using floating
IPs over the public network. 2) Connectivity between 2 instances
over a private network.
dependencies:
installer: ''
scenario: '^((?!bgpvpn|odl_l3).)*$'
....
We may distinguish 2 levels in the test case description:
* Tier level
* Test case level
At the tier level, we define the following parameters:
* ci_loop: indicate if in automated mode, the test case must be run in daily and/or weekly jobs
* description: a high level view of the test case
For a given test case we defined:
* the name of the test case
* the criteria (experimental): a criteria used to declare the test case as PASS or FAIL
* blocking: if set to true, if the test is failed, the execution of the following tests is canceled
* the description of the test case
* the dependencies: a combination of 2 regex on the scenario and the installer name
The order of execution is the one defined in the file if all test cases are selected.
In CI daily job the tests are executed in the following order:
1) healthcheck (blocking)
2) smoke: both vPings are blocking
3) SDN controller suites (blocking)
4) Feature project tests cases
In CI weekly job we add 2 tiers:
5) vIMS suite
6) Rally suite
As explained before, at the end of an automated execution, the OpenStack resources
might be eventually removed.
Please note that a system snapshot is taken before any test case execution.
This testcase.yaml file is used for CI, for the CLI and for the automatic reporting.
|
