diff options
Diffstat (limited to 'docs/testing/user/userguide/xtesting.rst')
| -rw-r--r-- | docs/testing/user/userguide/xtesting.rst | 252 |
1 files changed, 0 insertions, 252 deletions
diff --git a/docs/testing/user/userguide/xtesting.rst b/docs/testing/user/userguide/xtesting.rst deleted file mode 100644 index 0adfdda..0000000 --- a/docs/testing/user/userguide/xtesting.rst +++ /dev/null @@ -1,252 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. SPDX-License-Identifier: CC-BY-4.0 - - -NFVbench Xtesting test cases and Xtesting CI integration --------------------------------------------------------- - -NFVbench can leverages on `Xtesting CI <https://galaxy.ansible.com/collivier/xtesting>`_ and the common test case execution proposed by `Xtesting <https://xtesting.readthedocs.io/en/latest/>`_. -Thanks to a simple test case list, this tool deploys anywhere plug-and-play `CI/CD toolchains in a few commands <https://github.com/collivier/ansible-role-xtesting/blob/master/README.md>`_. -In addition, it supports multiple components such as Jenkins and Gitlab CI (test schedulers) and multiple deployment models such as all-in-one or centralized services. - -NFVbench using Xtesting and Xtesting CI will permit: - -- smoothly assemble multiple heterogeneous test cases -- generate the Jenkins jobs -- deploy local CI/CD toolchains everywhere -- dump all test case results and logs for third-party conformance review - -Xtesting CI only requires GNU/Linux as Operating System and asks for a few dependencies as described in Deploy your own Xtesting CI/CD toolchains: - -- python-virtualenv -- docker.io -- git - -Please note the next two points depending on the GNU/Linux distributions and the network settings: - -- SELinux: you may have to add --system-site-packages when creating the virtualenv ("Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!") -- Proxy: you may set your proxy in env for Ansible and in systemd for Docker https://docs.docker.com/config/daemon/systemd/#httphttps-proxy - -Here is the default NFVbench tree on the host running nfvbench container as -proposed in ``xtesting/ansible/host_vars/127.0.0.1`` file: - -- ``/home/opnfv/nfvbench/config``: contains nfvbench config files including - ``nfvbench.cfg``, the default config file used by xtesting test cases -- ``/home/opnfv/nfvbench/results``: top directory to write nfvbench results and log files - -File content: - -.. code-block:: yaml - - docker_args: - env: {} - params: - net: host - privileged: true - volumes: - - /lib/modules/$(uname -r):/lib/modules/$(uname -r) - - /usr/src/kernels:/usr/src/kernels -v /dev:/dev - - /home/opnfv/nfvbench/config:/etc/nfvbench - - /home/opnfv/nfvbench/results:/var/lib/xtesting/results - -Please note: if you want to use a different directory structure on the host, -replace ``/home/opnfv/nfvbench/*`` paths with appropriate paths to permit -NFVbench container to access config file and results directory. The config and -results paths do not need to share the same root directory. - -To deploy your own CI toolchain running NFVbench: - -.. code-block:: bash - - virtualenv nfvbench - . nfvbench/bin/activate - pip install ansible - ansible-galaxy install collivier.xtesting - git clone https://gerrit.opnfv.org/gerrit/nfvbench nfvbench - ansible-playbook nfvbench/xtesting/ansible/site.yml - - -NFVbench Xtesting test cases and existing CI chain --------------------------------------------------- - -For test automation purpose, Xtesting framework can be used as an executor of NFVbench test cases and called by a CI chain (Jenkins, Gitlab CI ...). -Xtesting use a testcases.yaml file to list and run test case. One basic testcases.yaml is provided by NFVbench natively but can be override. - -Example of CI scenario: - -.. image:: images/nfvbench-xtesting.png - -1. Run NFVbench container using Xtesting python library - -The NFVbench container can be started using docker run command. - -To run NFVbench using docker run: - -.. code-block:: bash - - docker run --rm \ - -e S3_ENDPOINT_URL=http://127.0.0.1:9000 \ - -e S3_DST_URL=s3://xtesting/$BUILD_TAG/$JOB_NAME-$BUILD_ID \ - -e HTTP_DST_URL=http://127.0.0.1:8181/$BUILD_TAG/$JOB_NAME-$BUILD_ID \ - -e AWS_ACCESS_KEY_ID=xtesting \ - -e AWS_SECRET_ACCESS_KEY=xtesting \ - -e TEST_DB_URL=http://127.0.0.1:8000/api/v1/results \ - -e NODE_NAME=nfvbench \ - -e BUILD_TAG=$BUILD_TAG \ - --privileged \ - -v /lib/modules/$(uname -r):/lib/modules/$(uname -r) \ - -v /usr/src/kernels:/usr/src/kernels -v /dev:/dev \ - -v $HOME/nfvbench/config:/etc/nfvbench \ - -v $HOME/workspace/$JOB_NAME/results:/var/lib/xtesting/results \ - opnfv/nfvbench run_tests -t 10kpps-pvp-run -r -p - -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| Docker options | Description | -+===============================================================+============================================================================+ -| --rm | clean up container after execution | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| -e S3_ENDPOINT_URL | (Xtesting) Environnement variable used to store NFVbench artifacts to Minio| -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| -e S3_DST_URL | (Xtesting) Environnement variable used for S3 storage destination | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| -e HTTP_DST_URL | (Xtesting) Environnement variable used for S3www service | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| -e AWS_ACCESS_KEY_ID | (Xtesting) Environnement variable used for S3 access key | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| -e AWS_SECRET_ACCESS_KEY | (Xtesting) Environnement variable used for S3 access secret | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| -e TEST_DB_URL | (Xtesting) Environnement variable used to export NFVbench results in DB | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| -e NODE_NAME | (Xtesting) Environnement variable used as result key identifier in DB | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| -e BUILD_TAG | (Xtesting) Environnement variable used as result key identifier in DB | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| --privileged | (optional) required if SELinux is enabled on the host | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| -v /lib/modules:/lib/modules | needed by kernel modules in the container | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| -v /usr/src/kernels:/usr/src/kernels | needed by TRex to build kernel modules when needed | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| -v /dev:/dev | needed by kernel modules in the container | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| -v $HOME/nfvbench/config:/etc/nfvbench | folder mapping to pass config files between the | -| | host and the docker space (see examples below) | -| | Here we map the $HOME/nfvbench/config directory on the host | -| | to the /etc/nfvbench directory in the container. | -| | Any other mapping can work as well | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| -v $HOME/workspace/$JOB_NAME/results:/var/lib/xtesting/results| (Xtesting) folder mapping to pass files between the | -| | CI chain workspace and the docker space to store Xtesting result files | -| | in orchestrator (Jenkins, Gitlab ...) | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| opnfv/nfvbench | container image name | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| run_tests | (Xtesting) Xtesting command to run test cases | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| -t 10kpps-pvp-run | (Xtesting) Xtesting parameter: Test case or tier (group of tests) | -| | to be executed. It will run all the test if not specified. | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| -r | (Xtesting) Xtesting parameter: publish result to database | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ -| -p | (Xtesting) Xtesting parameter: publish artifacts to a S3 service | -+---------------------------------------------------------------+----------------------------------------------------------------------------+ - -2. Run Xtesting test cases - -Executed directly by NFVbench docker entrypoint after docker start. - -3. Perform NFVbench test - -Xtesting call NFVbench python script to execute test case scenario and wait for run to be terminated. - -4. Export NFVbench result - -If ``-r`` option is used, results are pushed to a DB through Xtesting. -If ``-p`` option is used, results are pushed to a S3 service through Xtesting. - - -Override testcases.yaml file -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To replace existing testcases.yaml file, using Xtesting CI add the volume mapping in ``xtesting/ansible/host_vars/127.0.0.1`` file: - -.. code-block:: yaml - - docker_args: - env: {} - volumes: - - /lib/modules/$(uname -r):/lib/modules/$(uname -r) - - /usr/src/kernels:/usr/src/kernels -v /dev:/dev - - /home/opnfv/nfvbench/config:/etc/nfvbench - - /home/opnfv/nfvbench/results:/var/lib/xtesting/results - - /home/opnfv/nfvbench/xtesting/testcases.yaml:/usr/local/lib/python3.6/dist-packages/xtesting/ci/testcases.yaml - -* ``/home/opnfv/nfvbench/xtesting/testcases.yaml:/usr/local/lib/python3.6/dist-packages/xtesting/ci/testcases.yaml`` : volume mapping to pass testcases.yaml file between the host and the docker space. Host path required testcases.yaml file inside. - - -To replace existing testcases.yaml file, using NFVbench container: - -.. code-block:: bash - - docker run --name nfvbench --detach --privileged \ - -v /lib/modules/$(uname -r):/lib/modules/$(uname -r) \ - -v /usr/src/kernels:/usr/src/kernels \ - -v /dev:/dev \ - -v $HOME/nfvbench/config:/etc/nfvbench \ - -v $HOME/nfvbench/results:/var/lib/xtesting/results \ - -v $HOME/nfvbench/xtesting/testcases.yaml:/usr/local/lib/python3.8/dist-packages/xtesting/ci/testcases.yaml \ - opnfv/nfvbench - -* ``$HOME/nfvbench/xtesting/testcases.yaml:/usr/local/lib/python3.8/dist-packages/xtesting/ci/testcases.yaml`` : volume mapping to pass testcases.yaml file between the host and the docker space. Host path required testcases.yaml file inside. - - -Example of Xtesting test case -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: yaml - - --- - tiers: - - - name: nfvbench - order: 1 - description: 'Data Plane Performance Testing' - testcases: - - - case_name: 10kpps-pvp-run - project_name: nfvbench - criteria: 100 - blocking: true - clean_flag: false - description: '' - run: - name: 'bashfeature' - args: - cmd: - - nfvbench -c /etc/nfvbench/nfvbench.cfg --rate 10kpps - - -Examples of manual run -~~~~~~~~~~~~~~~~~~~~~~ - -If NFVbench container is already started in CLI mode (see Starting NFVbench in CLI mode dedicated chapter). -To do a single run at 10,000pps bi-directional (or 5kpps in each direction) using the PVP packet path: - -.. code-block:: bash - - docker exec -it nfvbench run_tests -t 10kpps-pvp-run - -Xtesting option used: - -* ``-t 10kpps-pvp-run`` : specify the test case to run - -To pass all test cases: - -.. code-block:: bash - - docker exec -it nfvbench run_tests -t all - -Xtesting option used: - -* ``-t all`` : select all test cases existing in testcases.yaml file - |