summaryrefslogtreecommitdiffstats
path: root/docs/testing/user/userguide/xtesting.rst
blob: 0b863ca1fad530615463e20cdedd25561dd411c7 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. SPDX-License-Identifier: CC-BY-4.0


NFVbench Xtesting test cases and Xtesting CI integration
--------------------------------------------------------

NFVbench can leverages on `Xtesting CI <https://galaxy.ansible.com/collivier/xtesting>`_ and the common test case execution proposed by `Xtesting <https://xtesting.readthedocs.io/en/latest/>`_.
Thanks to a simple test case list, this tool deploys anywhere plug-and-play `CI/CD toolchains in a few commands <https://wiki.opnfv.org/pages/viewpage.action?pageId=32015004>`_.
In addition, it supports multiple components such as Jenkins and Gitlab CI (test schedulers) and multiple deployment models such as all-in-one or centralized services.

NFVbench using Xtesting and Xtesting CI will permit:

- smoothly assemble multiple heterogeneous test cases
- generate the Jenkins jobs
- deploy local CI/CD toolchains everywhere
- dump all test case results and logs for third-party conformance review

Xtesting CI only requires GNU/Linux as Operating System and asks for a few dependencies as described in Deploy your own Xtesting CI/CD toolchains:

- python-virtualenv
- docker.io
- git

Please note the next two points depending on the GNU/Linux distributions and the network settings:

- SELinux: you may have to add --system-site-packages when creating the virtualenv ("Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!")
- Proxy: you may set your proxy in env for Ansible and in systemd for Docker https://docs.docker.com/config/daemon/systemd/#httphttps-proxy

Here is the default NFVbench tree as proposed in ``xtesting/ansible/host_vars/127.0.0.1`` file:

- ``/home/opnfv/nfvbench``

File content:

.. code-block:: bash

    docker_args:
    env: {}
    volumes:
      - /lib/modules/$(uname -r):/lib/modules/$(uname -r)
      - /usr/src/kernels:/usr/src/kernels -v /dev:/dev
      - /home/opnfv/nfvbench:/tmp/nfvbench

Please note: replace ``/home/opnfv/nfvbench`` with appropriate path to permit NFVbench container to access config file

To deploy your own CI toolchain running NFVbench:

.. code-block:: bash

    virtualenv nfvbench
    . nfvbench/bin/activate
    pip install ansible
    ansible-galaxy install collivier.xtesting
    git clone https://gerrit.opnfv.org/gerrit/nfvbench nfvbench
    ansible-playbook nfvbench/xtesting/ansible/site.yml


NFVbench Xtesting test cases and existing CI chain
--------------------------------------------------

For test automation purpose, Xtesting framework can be used as an executor of NFVbench test cases and called by a CI chain (Jenkins, Gitlab CI ...).
Xtesting use a testcases.yaml file to list and run test case. One basic testcases.yaml is provided by NFVbench natively but can be override.

Example of CI scenario:

.. image:: images/nfvbench-xtesting.png

1. Run NFVbench container using Xtesting python library

The NFVbench container can be started using docker run command.

To run NFVbench using docker run:

.. code-block:: bash

    docker run --rm \
        -e S3_ENDPOINT_URL=http://127.0.0.1:9000 \
        -e S3_DST_URL=s3://xtesting/$BUILD_TAG/$JOB_NAME-$BUILD_ID \
        -e HTTP_DST_URL=http://127.0.0.1:8181/$BUILD_TAG/$JOB_NAME-$BUILD_ID \
        -e AWS_ACCESS_KEY_ID=xtesting \
        -e AWS_SECRET_ACCESS_KEY=xtesting \
        -e TEST_DB_URL=http://127.0.0.1:8000/api/v1/results \
        -e NODE_NAME=nfvbench \
        -e BUILD_TAG=$BUILD_TAG \
        --privileged \
        -v /lib/modules/$(uname -r):/lib/modules/$(uname -r) \
        -v /usr/src/kernels:/usr/src/kernels -v /dev:/dev \
        -v $HOME/nfvbench:/tmp/nfvbench \
        -v $HOME/workspace/$JOB_NAME/results:/var/lib/xtesting/results \
        opnfv/nfvbench run_tests -t 10kpps-pvp-run -r -p

+---------------------------------------------------------------+----------------------------------------------------------------------------+
| Docker options                                                | Description                                                                |
+===============================================================+============================================================================+
| --rm                                                          | clean up container after execution                                         |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| -e S3_ENDPOINT_URL                                            | (Xtesting) Environnement variable used to store NFVbench artifacts to Minio|
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| -e S3_DST_URL                                                 | (Xtesting) Environnement variable used for S3 storage destination          |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| -e HTTP_DST_URL                                               | (Xtesting) Environnement variable used for S3www service                   |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| -e AWS_ACCESS_KEY_ID                                          | (Xtesting) Environnement variable used for S3 access key                   |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| -e AWS_SECRET_ACCESS_KEY                                      | (Xtesting) Environnement variable used for S3 access secret                |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| -e TEST_DB_URL                                                | (Xtesting) Environnement variable used to export NFVbench results in DB    |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| -e NODE_NAME                                                  | (Xtesting) Environnement variable used as result key identifier in DB      |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| -e BUILD_TAG                                                  | (Xtesting) Environnement variable used as result key identifier in DB      |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| --privileged                                                  | (optional) required if SELinux is enabled on the host                      |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| -v /lib/modules:/lib/modules                                  | needed by kernel modules in the container                                  |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| -v /usr/src/kernels:/usr/src/kernels                          | needed by TRex to build kernel modules when needed                         |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| -v /dev:/dev                                                  | needed by kernel modules in the container                                  |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| -v $HOME/nfvbench:/tmp/nfvbench                               | folder mapping to pass files between the                                   |
|                                                               | host and the docker space (see examples below)                             |
|                                                               | Here we map the $HOME/nfvbench directory on the host                       |
|                                                               | to the /tmp/nfvbench director in the container.                            |
|                                                               | Any other mapping can work as well                                         |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| -v $HOME/workspace/$JOB_NAME/results:/var/lib/xtesting/results| (Xtesting) folder mapping to pass files between the                        |
|                                                               | CI chain workspace and the docker space to store Xtesting result files     |
|                                                               | in orchestrator (Jenkins, Gitlab ...)                                      |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| opnfv/nfvbench                                                | container image name                                                       |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| run_tests                                                     | (Xtesting) Xtesting command to run test cases                              |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| -t 10kpps-pvp-run                                             | (Xtesting) Xtesting parameter: Test case or tier (group of tests)          |
|                                                               | to be executed. It will run all the test if not specified.                 |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| -r                                                            | (Xtesting) Xtesting parameter: publish result to database                  |
+---------------------------------------------------------------+----------------------------------------------------------------------------+
| -p                                                            | (Xtesting) Xtesting parameter: publish artifacts to a S3 service           |
+---------------------------------------------------------------+----------------------------------------------------------------------------+

2. Run Xtesting test cases

Executed directly by NFVbench docker entrypoint after docker start.

3. Perform NFVbench test

Xtesting call NFVbench python script to execute test case scenario and wait for run to be terminated.

4. Export NFVbench result

If ``-r`` option is used, results are pushed to a DB through Xtesting.
If ``-p`` option is used, results are pushed to a S3 service through Xtesting.


Override testcases.yaml file
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To replace existing testcases.yaml file, using Xtesting CI add the volume mapping in ``xtesting/ansible/host_vars/127.0.0.1`` file:

.. code-block:: bash

    docker_args:
    env: {}
    volumes:
      - /lib/modules/$(uname -r):/lib/modules/$(uname -r)
      - /usr/src/kernels:/usr/src/kernels -v /dev:/dev
      - /home/opnfv/nfvbench:/tmp/nfvbench
      - /home/opnfv/nfvbench/xtesting/testcases.yaml:/usr/local/lib/python3.6/dist-packages/xtesting/ci/testcases.yaml

* ``/home/opnfv/nfvbench/xtesting/testcases.yaml:/usr/local/lib/python3.6/dist-packages/xtesting/ci/testcases.yaml`` : volume mapping to pass testcases.yaml file between the host and the docker space. Host path required testcases.yaml file inside.


To replace existing testcases.yaml file, using NFVbench container:

.. code-block:: bash

    docker run --name nfvbench --detach --privileged -v /lib/modules/$(uname -r):/lib/modules/$(uname -r) -v /usr/src/kernels:/usr/src/kernels -v /dev:/dev -v $HOME/nfvbench:/tmp/nfvbench \
    -v $HOME/nfvbench/xtesting/testcases.yaml:/usr/local/lib/python3.6/dist-packages/xtesting/ci/testcases.yaml \
    opnfv/nfvbench


* ``$HOME/nfvbench/xtesting/testcases.yaml:/usr/local/lib/python3.6/dist-packages/xtesting/ci/testcases.yaml`` : volume mapping to pass testcases.yaml file between the host and the docker space. Host path required testcases.yaml file inside.


Example of Xtesting test case
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: bash

    ---
    tiers:
        -
            name: nfvbench
            order: 1
            description: 'Data Plane Performance Testing'
            testcases:
                -
                    case_name: 10kpps-pvp-run
                    project_name: nfvbench
                    criteria: 100
                    blocking: true
                    clean_flag: false
                    description: ''
                    run:
                        name: 'bashfeature'
                        args:
                            cmd:
                                - nfvbench -c /tmp/nfvbench/nfvbench.cfg --rate 10kpps


Examples of manual run
~~~~~~~~~~~~~~~~~~~~~~

If NFVbench container is already started in CLI mode (see Starting NFVbench in CLI mode dedicated chapter).
To do a single run at 10,000pps bi-directional (or 5kpps in each direction) using the PVP packet path:

.. code-block:: bash

   docker exec -it nfvbench run_tests -t 10kpps-pvp-run

Xtesting option used:

* ``-t 10kpps-pvp-run`` : specify the test case to run

To pass all test cases:

.. code-block:: bash

   docker exec -it nfvbench run_tests -t all

Xtesting option used:

* ``-t all`` : select all test cases existing in testcases.yaml file