summaryrefslogtreecommitdiffstats
path: root/docs/testing/user/userguide/cli_reference.rst
blob: 7dd5c8e4c366a2ab8d927239f12ce486a946093f (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
.. (c) OPNFV

.. _cli-reference:

=========================================
Dovetail Command Line Interface Reference
=========================================

Dovetail command line is to have a simple command line interface in Dovetail to
make easier for users to handle the functions that dovetail framework provides.

Commands List
=============

+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| Commands                                                               | Action                                                                                            |
|                                                                        |                                                                                                   |
+========================================================================+===================================================================================================+
| dovetail --help | -h                                                   | Show usage of command "dovetail"                                                                  |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail --version                                                     | Show version number                                                                               |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| **Dovetail List Commands**                                                                                                                                                 |
|                                                                                                                                                                            |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail list --help | -h                                              | Show usage of command "dovetail list"                                                             |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail list                                                          | List all available test suites and all test cases within each test suite                          |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail list <test_suite_name>                                        | List all available test areas within test suite <test_suite_name>                                 |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| **Dovetail Show Commands**                                                                                                                                                 |
|                                                                                                                                                                            |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail show --help | -h                                              | Show usage of command "dovetail show"                                                             |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail show <test_case_name>                                         | Show the details of one test case                                                                 |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| **Dovetail Run Commands**                                                                                                                                                  |
|                                                                                                                                                                            |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail run --help | -h                                               | Show usage of command "dovetail run"                                                              |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail run                                                           | Run Dovetail with all test cases within default test suite                                        |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail run --testsuite <test_suite_name>                             | Run Dovetail with all test cases within test suite <test_suite_name>                              |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail run --testsuite <test_suite_name> --testarea <test_area_name> | Run Dovetail with test area <test_area_name> within test suite <test_suite_name>.                 |
|                                                                        | Test area can be chosen from (vping, tempest, security, ha, stress, bgpvpn, vnf, snaps).          |
|                                                                        | Repeat option to set multiple test areas.                                                         |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail run --testcase <test_case_name>                               | Run Dovetail with one or more specified test cases.                                               |
|                                                                        | Repeat option to set multiple test cases.                                                         |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail run --mandatory --testsuite <test_suite_name>                 | Run Dovetail with all mandatory test cases within test suite <test_suite_name>                    |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail run --optional --testsuite <test_suite_name>                  | Run Dovetail with all optional test cases within test suite <test_suite_name>                     |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail run --debug | -d                                              | Run Dovetail with debug mode and show all debug logs                                              |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail run --offline                                                 | Run Dovetail offline, use local docker images instead of download online                          |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail run --report | -r <db_url>                                    | Package the results directory which can be used to upload to OVP web portal                       |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail run --deploy-scenario <deploy_scenario_name>                  | Specify the deploy scenario having as project name 'ovs'                                          |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail run --no-api-validation                                       | Disable strict API response validation                                                            |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail run --no-clean | -n                                           | Keep all Containers created for debuging                                                          |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| dovetail run --stop | -s                                               | Stop immediately when one test case failed                                                        |
|                                                                        |                                                                                                   |
+------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+


Commands Examples
=================

Dovetail Commands
-----------------

.. code-block:: bash

   root@1f230e719e44:~/dovetail/dovetail# dovetail --help
   Usage: dovetail [OPTIONS] COMMAND [ARGS]...

   Options:
     --version   Show the version and exit.
     -h, --help  Show this message and exit.

   Commands:
     list  list the testsuite details
     run   run the testcases
     show  show the testcases details

.. code-block:: bash

   root@1f230e719e44:~/dovetail/dovetail# dovetail --version
   dovetail, version 2018.9.0

Dovetail List Commands
----------------------

.. code-block:: bash

   root@1f230e719e44:~/dovetail/dovetail# dovetail list --help
   Usage: dovetail list [OPTIONS] [TESTSUITE]

     list the testsuite details

   Options:
     -h, --help  Show this message and exit.

.. code-block:: bash

   root@1f230e719e44:~/dovetail/dovetail# dovetail list ovp.2019.12
   - mandatory
       functest.vping.userdata
       functest.vping.ssh
       functest.tempest.osinterop
       functest.tempest.compute
       functest.tempest.identity_v3
       functest.tempest.image
       functest.tempest.network_api
       functest.tempest.volume
       functest.tempest.neutron_trunk_ports
       functest.tempest.ipv6_api
       functest.security.patrole
       yardstick.ha.nova_api
       yardstick.ha.neutron_server
       yardstick.ha.keystone
       yardstick.ha.glance_api
       yardstick.ha.cinder_api
       yardstick.ha.cpu_load
       yardstick.ha.disk_load
       yardstick.ha.haproxy
       yardstick.ha.rabbitmq
       yardstick.ha.database
       bottlenecks.stress.ping
   - optional
       functest.tempest.ipv6_scenario
       functest.tempest.multi_node_scheduling
       functest.tempest.network_security
       functest.tempest.vm_lifecycle
       functest.tempest.network_scenario
       functest.tempest.bgpvpn
       functest.security.patrole_vxlan_dependent
       yardstick.ha.neutron_l3_agent
       yardstick.ha.controller_restart
       functest.vnf.vims
       functest.vnf.vepc

Dovetail Show Commands
----------------------

.. code-block:: bash

   root@1f230e719e44:~/dovetail/dovetail# dovetail show --help
   Usage: dovetail show [OPTIONS] TESTCASE

     show the testcases details

   Options:
     -h, --help  Show this message and exit.

.. code-block:: bash

   root@1f230e719e44:~/dovetail/dovetail# dovetail show functest.vping.ssh
   ---
   functest.vping.ssh:
     name: functest.vping.ssh
     objective: testing for vping using ssh
     validate:
       type: functest
       testcase: vping_ssh
       image_name: opnfv/functest-healthcheck
     report:
       source_archive_files:
         - functest.log
       dest_archive_files:
         - vping_logs/functest.vping.ssh.log
       check_results_file:
         - 'functest_results.txt'
       portal_key_file: vping_logs/functest.vping.ssh.log
       sub_testcase_list:

.. code-block:: bash

   root@1f230e719e44:~/dovetail/dovetail# dovetail show functest.tempest.image
   ---
   functest.tempest.image:
     name: functest.tempest.image
     objective: tempest smoke test cases about image
     validate:
       type: functest
       testcase: tempest_custom
       pre_condition:
         - 'cp /home/opnfv/userconfig/pre_config/tempest_conf.yaml /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/tempest_conf.yaml'
         - 'cp /home/opnfv/userconfig/tempest_custom_testcases.yaml /usr/lib/python2.7/site-packages/xtesting/ci/testcases.yaml'
         - 'cp /home/opnfv/functest/results/tempest_custom.txt /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt'
     report:
       source_archive_files:
         - functest.log
         - tempest_custom/rally.log
         - tempest_custom/tempest-report.html
       dest_archive_files:
         - tempest_logs/functest.tempest.image.functest.log
         - tempest_logs/functest.tempest.image.log
         - tempest_logs/functest.tempest.image.html
       check_results_file:
         - 'functest_results.txt'
       portal_key_file: tempest_logs/functest.tempest.image.html
       sub_testcase_list:
         - tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_register_upload_get_image_file[id-139b765e-7f3d-4b3d-8b37-3ca3876ee318,smoke]
         - tempest.api.image.v2.test_versions.VersionsTest.test_list_versions[id-659ea30a-a17c-4317-832c-0f68ed23c31d,smoke]

Dovetail Run Commands
----------------------

.. code-block:: bash

   root@1f230e719e44:~/dovetail/dovetail# dovetail run --help
   Usage: run.py [OPTIONS]

   Dovetail compliance test entry!

   Options:
   --opnfv-ci              Only enabled when running with OPNFV CI jobs and pushing results to TestAPI DB
   --optional              Run all optional test cases.
   --mandatory             Run all mandatory test cases.
   --deploy-scenario TEXT  Specify the DEPLOY_SCENARIO which will be used as input by each testcase respectively
   -n, --no-clean          Keep all Containers created for debuging.
   --no-api-validation     disable strict API response validation
   --offline               run in offline method, which means not to update the docker upstream images, functest, yardstick, etc.
   -r, --report            Create a tarball file to upload to OVP web portal
   -s, --stop              Flag for stopping on test case failure.
   -d, --debug             Flag for showing debug log on screen.
   --testcase TEXT         Compliance testcase. Specify option multiple times to include multiple test cases.
   --testarea TEXT         Compliance testarea within testsuite. Specify option multiple times to include multiple test areas.
   --testsuite TEXT        compliance testsuite.
   -h, --help              Show this message and exit.

.. code-block:: bash

   root@1f230e719e44:~/dovetail/dovetail# dovetail run --testcase functest.vping.ssh --offline -r --deploy-scenario os-nosdn-ovs-ha
   2019-12-06 02:51:52,634 - run - INFO - ================================================
   2019-12-06 02:51:52,634 - run - INFO - Dovetail compliance: ovp.2019.12!
   2019-12-06 02:51:52,634 - run - INFO - ================================================
   2019-12-06 02:51:52,634 - run - INFO - Build tag: daily-master-5b58584a-17d3-11ea-878a-0242ac110002
   2019-12-06 02:51:52,634 - run - INFO - DEPLOY_SCENARIO : os-nosdn-ovs-ha
   2019-12-06 02:51:53,077 - run - INFO - >>[testcase]: functest.vping.ssh
   2019-12-06 02:51:53,078 - dovetail.test_runner.DockerRunner - WARNING - There is no hosts file /home/ovp/pre_config/hosts.yaml. This may cause some issues with domain name resolution.
   2019-12-06 02:51:54,048 - dovetail.test_runner.DockerRunner - INFO - Get hardware info of all nodes list in file /home/ovp/pre_config/pod.yaml ...
   2019-12-06 02:51:54,049 - dovetail.test_runner.DockerRunner - INFO - Hardware info of all nodes are stored in file /home/dovetail/results/all_hosts_info.json.
   2019-12-06 02:51:54,073 - dovetail.container.Container - WARNING - There is no hosts file /home/ovp/pre_config/hosts.yaml. This may cause some issues with domain name resolution.
   2019-12-06 02:52:57,982 - dovetail.report.Report - INFO - Results have been stored with files: ['/home/ovp/results/functest_results.txt'].
   2019-12-06 02:52:57,986 - dovetail.report.Report - INFO -

   Dovetail Report
   Version: 2019.12
   Build Tag: daily-master-5b58584a-17d3-11ea-878a-0242ac110002
   Test Date: 2019-12-06 02:52:57 UTC
   Duration: 64.91 s

   Pass Rate: 100.00% (1/1)
   vping:                     pass rate 100.00%
   -functest.vping.ssh        PASS