diff options
Diffstat (limited to 'docs/testing/user')
7 files changed, 498 insertions, 12 deletions
diff --git a/docs/testing/user/userguide/04-installation.rst b/docs/testing/user/userguide/04-installation.rst index dc528db6c..caebecc09 100644 --- a/docs/testing/user/userguide/04-installation.rst +++ b/docs/testing/user/userguide/04-installation.rst @@ -107,6 +107,12 @@ Run the Docker image to get a Yardstick container:: ======================= ==================================================== --name yardstick The name for this container +If the host is restarted +^^^^^^^^^^^^^^^^^^^^^^^^ + +The yardstick container must be started if the host is rebooted:: + + docker start yardstick Configure the Yardstick container environment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -301,12 +307,6 @@ Prerequisite preparation:: sudo -EH pip install appdirs==1.4.0 sudo -EH pip install virtualenv -Create a virtual environment:: - - virtualenv ~/yardstick_venv - export YARDSTICK_VENV=~/yardstick_venv - source ~/yardstick_venv/bin/activate - Download the source code and install Yardstick from it:: git clone https://gerrit.opnfv.org/gerrit/yardstick @@ -314,6 +314,10 @@ Download the source code and install Yardstick from it:: cd ~/yardstick sudo -EH ./install.sh +If the host is ever restarted, nginx and uwsgi need to be restarted:: + + service nginx restart + uwsgi -i /etc/yardstick/yardstick.ini Configure the Yardstick environment (**Todo**) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -331,6 +335,91 @@ For uninstalling Yardstick, just delete the virtual environment:: rm -rf ~/yardstick_venv +Install Yardstick directly in OpenSUSE +-------------------------------------- + +.. _install-framework: + +You can install Yardstick framework directly in OpenSUSE. + + +Install Yardstick +^^^^^^^^^^^^^^^^^ + +Prerequisite preparation:: + + sudo -EH zypper -n install -y gcc \ + wget \ + git \ + sshpass \ + qemu-tools \ + kpartx \ + libffi-devel \ + libopenssl-devel \ + python \ + python-devel \ + python-virtualenv \ + libxml2-devel \ + libxslt-devel \ + python-setuptools-git + +Create a virtual environment:: + + virtualenv ~/yardstick_venv + export YARDSTICK_VENV=~/yardstick_venv + source ~/yardstick_venv/bin/activate + sudo -EH easy_install -U setuptools + +Download the source code and install Yardstick from it:: + + git clone https://gerrit.opnfv.org/gerrit/yardstick + export YARDSTICK_REPO_DIR=~/yardstick + cd yardstick + sudo -EH python setup.py install + sudo -EH pip install -r requirements.txt + +Install missing python modules:: + + sudo -EH pip install pyyaml \ + oslo_utils \ + oslo_serialization \ + oslo_config \ + paramiko \ + python.heatclient \ + python.novaclient \ + python.glanceclient \ + python.neutronclient \ + scp \ + jinja2 + + +Configure the Yardstick environment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Source the OpenStack environment variables:: + + source DEVSTACK_DIRECTORY/openrc + +Export the Openstack external network. The default installation of Devstack +names the external network public:: + + export EXTERNAL_NETWORK=public + export OS_USERNAME=demo + +Change the API version used by Yardstick to v2.0 (the devstack openrc sets it +to v3):: + + export OS_AUTH_URL=http://PUBLIC_IP_ADDRESS:5000/v2.0 + + +Uninstall Yardstick +^^^^^^^^^^^^^^^^^^^ + +For unistalling Yardstick, just delete the virtual environment:: + + rm -rf ~/yardstick_venv + + Verify the installation ----------------------- diff --git a/docs/testing/user/userguide/12-nsb_installation.rst b/docs/testing/user/userguide/12-nsb_installation.rst index 8cc26acd5..a584ca231 100644 --- a/docs/testing/user/userguide/12-nsb_installation.rst +++ b/docs/testing/user/userguide/12-nsb_installation.rst @@ -112,12 +112,52 @@ Download the source code and install Yardstick from it # git checkout <tag or stable branch> git checkout stable/euphrates - # For Bare-Metal or Standalone Virtualization - ./nsb_setup.sh +Configure the network proxy, either using the environment variables or setting +the global environment file: - # For OpenStack - ./nsb_setup.sh <path to admin-openrc.sh> +.. code-block:: ini + cat /etc/environment + http_proxy='http://proxy.company.com:port' + https_proxy='http://proxy.company.com:port' +.. code-block:: console + export http_proxy='http://proxy.company.com:port' + export https_proxy='http://proxy.company.com:port' + +The last step is to modify the Yardstick installation inventory, used by +Ansible: + +.. code-block:: ini + cat ./ansible/yardstick-install-inventory.ini + [jumphost] + localhost ansible_connection=local + + [yardstick-standalone] + yardstick-standalone-node ansible_host=192.168.1.2 + yardstick-standalone-node-2 ansible_host=192.168.1.3 + + # section below is only due backward compatibility. + # it will be removed later + [yardstick:children] + jumphost + + [all:vars] + ansible_user=root + ansible_pass=root + + +To execute an installation for a Bare-Metal or a Standalone context: + +.. code-block:: console + + ./nsb_setup.sh + + +To execute an installation for an OpenStack context: + +.. code-block:: console + + ./nsb_setup.sh <path to admin-openrc.sh> Above command setup docker with latest yardstick code. To execute diff --git a/docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst b/docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst index 43aa3d69a..895837283 100644 --- a/docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst +++ b/docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst @@ -26,3 +26,5 @@ NSB PROX Test Case Descriptions tc_prox_context_mpls_tagging_port tc_prox_context_buffering_port tc_prox_context_load_balancer_port + tc_prox_context_vpe_port + tc_prox_context_lw_after_port diff --git a/docs/testing/user/userguide/nsb/tc_prox_context_lw_aftr_port.rst b/docs/testing/user/userguide/nsb/tc_prox_context_lw_aftr_port.rst new file mode 100644 index 000000000..5a1fada05 --- /dev/null +++ b/docs/testing/user/userguide/nsb/tc_prox_context_lw_aftr_port.rst @@ -0,0 +1,107 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, 2017 Intel Corporation. + +************************************************ +Yardstick Test Case Description: NSB PROX LwAFTR +************************************************ + ++-----------------------------------------------------------------------------+ +|NSB PROX test for NFVI characterization | +| | ++--------------+--------------------------------------------------------------+ +|test case id | tc_prox_{context}_lw_aftr-{port_num} | +| | | +| | * context = baremetal or heat_context; | +| | * port_num = 4; | +| | | ++--------------+--------------------------------------------------------------+ +|metric | * Network Throughput; | +| | * TG Packets Out; | +| | * TG Packets In; | +| | * VNF Packets Out; | +| | * VNF Packets In; | +| | * Dropped packets; | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | The PROX LW_AFTR test will take packets in from one | +| | port and remove the ipv6 encapsulation and forward them to | +| | another port. While forwarded packets in other direction | +| | will be encapsulated in an ipv6 header. | +| | | +| | The lw_aftr test cases are implemented to run in baremetal | +| | and heat context an require 4 port topology to run the | +| | default configuration. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | The LW_AFTR test cases are listed below: | +| | | +| | * tc_prox_baremetal_lw_aftr-4.yaml | +| | * tc_prox_heat_context_lw_aftr-4.yaml | +| | | +| | Test duration is set as 300sec for each test. | +| | The minimum packet size for MPLS test is 68 bytes. This is | +| | set in the traffic profile and can be configured to use | +| | higher packet sizes. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | PROX | +| | PROX is a DPDK application that can simulate VNF workloads | +| | and can generate traffic and used for NFVI characterization | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | The PROX lwAFTR test cases can be configured with | +| | different: | +| | | +| | * packet sizes; | +| | * test durations; | +| | * tolerated loss; | +| | | +| | Default values exist. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | For Openstack test case image (yardstick-samplevnfs) needs | +|conditions | to be installed into Glance with Prox and Dpdk included in | +| | it. | +| | | +| | For Baremetal tests cases Prox and Dpdk must be installed in | +| | the hosts where the test is executed. The pod.yaml file must | +| | have the necessary system and NIC information | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | For Baremetal test: The TG and VNF are started on the hosts | +| | based on the pod file. | +| | | +| | For Heat test: Two host VMs are booted, as Traffic generator | +| | and VNF(LW_AFTR workload) based on the test flavor. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | Yardstick is connected with the TG and VNF by using ssh. | +| | The test will resolve the topology and instantiate the VNF | +| | and TG and collect the KPI's/metrics. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | The TG will send packets to the VNF. If the number of | +| | dropped packets is more than the tolerated loss the line | +| | rate or throughput is halved. This is done until the dropped | +| | packets are within an acceptable tolerated loss. | +| | | +| | The KPI is the number of packets per second for 86 bytes | +| | packet size with an accepted minimal packet loss for the | +| | default configuration. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | In Baremetal test: The test quits the application and unbind | +| | the dpdk ports. | +| | | +| | In Heat test: Two host VMs are deleted on test completion. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | The test case will achieve a Throughput with an accepted | +| | minimal tolerated packet loss. | ++--------------+--------------------------------------------------------------+ + diff --git a/docs/testing/user/userguide/nsb/tc_prox_context_vpe_port.rst b/docs/testing/user/userguide/nsb/tc_prox_context_vpe_port.rst new file mode 100644 index 000000000..6827b0525 --- /dev/null +++ b/docs/testing/user/userguide/nsb/tc_prox_context_vpe_port.rst @@ -0,0 +1,108 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, 2017 Intel Corporation. + +********************************************** +Yardstick Test Case Description: NSB PROXi VPE +********************************************** + ++-----------------------------------------------------------------------------+ +|NSB PROX test for NFVI characterization | +| | ++--------------+--------------------------------------------------------------+ +|test case id | tc_prox_{context}_vpe-{port_num} | +| | | +| | * context = baremetal or heat_context; | +| | * port_num = 4; | +| | | ++--------------+--------------------------------------------------------------+ +|metric | * Network Throughput; | +| | * TG Packets Out; | +| | * TG Packets In; | +| | * VNF Packets Out; | +| | * VNF Packets In; | +| | * Dropped packets; | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | The PROX VPE test handles packet processing, routing, QinQ | +| | encapsulation, flows, ACL rules, adds/removes MPLS tagging | +| | and performs QoS before forwarding packet to another port. | +| | The reverse applies to forwarded packets in the other | +| | direction. | +| | | +| | The VPE test cases are implemented to run in baremetal | +| | and heat context an require 4 port topology to run the | +| | default configuration. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | The VPE test cases are listed below: | +| | | +| | * tc_prox_baremetal_vpe-4.yaml | +| | * tc_prox_heat_context_vpe-4.yaml | +| | | +| | Test duration is set as 300sec for each test. | +| | The minimum packet size for VPE test is 68 bytes. This is | +| | set in the traffic profile and can be configured to use | +| | higher packet sizes. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | PROX | +| | PROX is a DPDK application that can simulate VNF workloads | +| | and can generate traffic and used for NFVI characterization | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | The PROX VPE test cases can be configured with | +| | different: | +| | | +| | * packet sizes; | +| | * test durations; | +| | * tolerated loss; | +| | | +| | Default values exist. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | For Openstack test case image (yardstick-samplevnfs) needs | +|conditions | to be installed into Glance with Prox and Dpdk included in | +| | it. | +| | | +| | For Baremetal tests cases Prox and Dpdk must be installed in | +| | the hosts where the test is executed. The pod.yaml file must | +| | have the necessary system and NIC information | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | For Baremetal test: The TG and VNF are started on the hosts | +| | based on the pod file. | +| | | +| | For Heat test: Two host VMs are booted, as Traffic generator | +| | and VNF(VPE workload) based on the test flavor. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | Yardstick is connected with the TG and VNF by using ssh. | +| | The test will resolve the topology and instantiate the VNF | +| | and TG and collect the KPI's/metrics. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | The TG will send packets to the VNF. If the number of | +| | dropped packets is more than the tolerated loss the line | +| | rate or throughput is halved. This is done until the dropped | +| | packets are within an acceptable tolerated loss. | +| | | +| | The KPI is the number of packets per second for 68 bytes | +| | packet size with an accepted minimal packet loss for the | +| | default configuration. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | In Baremetal test: The test quits the application and unbind | +| | the dpdk ports. | +| | | +| | In Heat test: Two host VMs are deleted on test completion. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | The test case will achieve a Throughput with an accepted | +| | minimal tolerated packet loss. | ++--------------+--------------------------------------------------------------+ + diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc081.rst b/docs/testing/user/userguide/opnfv_yardstick_tc081.rst index 90af8a382..793c3fdd5 100644 --- a/docs/testing/user/userguide/opnfv_yardstick_tc081.rst +++ b/docs/testing/user/userguide/opnfv_yardstick_tc081.rst @@ -4,7 +4,7 @@ .. (c) OPNFV, Huawei Technologies Co.,Ltd and others. ************************************* -Yardstick Test Case Description TC080 +Yardstick Test Case Description TC081 ************************************* .. _cirros-image: https://download.cirros-cloud.net @@ -21,7 +21,7 @@ Yardstick Test Case Description TC080 |metric | RTT (Round Trip Time) | | | | +--------------+--------------------------------------------------------------+ -|test purpose | The purpose of TC080 is to do a basic verification that | +|test purpose | The purpose of TC081 is to do a basic verification that | | | network latency is within acceptable boundaries when packets | | | travel between a containers and a VM. | | | | diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc084.rst b/docs/testing/user/userguide/opnfv_yardstick_tc084.rst new file mode 100644 index 000000000..2e7b28e25 --- /dev/null +++ b/docs/testing/user/userguide/opnfv_yardstick_tc084.rst @@ -0,0 +1,140 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +************************************* +Yardstick Test Case Description TC084 +************************************* + +.. _spec_cpu_2006: https://www.spec.org/cpu2006/ + ++-----------------------------------------------------------------------------+ +|Compute Performance | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC084_SPEC CPU 2006 FOR VM | +| | | ++--------------+--------------------------------------------------------------+ +|metric | compute-intensive performance | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | The purpose of TC084 is to evaluate the IaaS compute | +| | performance by using SPEC CPU 2006 benchmark. The SPEC CPU | +| | 2006 benchmark has several different ways to measure | +| | computer performance. One way is to measure how fast the | +| | computer completes a single task; this is called a speed | +| | measurement. Another way is to measure how many tasks | +| | computer can accomplish in a certain amount of time; this is | +| | called a throughput, capacity or rate measurement. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | SPEC CPU 2006 | +| | | +| | The SPEC CPU 2006 benchmark is SPEC's industry-standardized, | +| | CPU-intensive benchmark suite, stressing a system's | +| | processor, memory subsystem and compiler. This benchmark | +| | suite includes the SPECint benchmarks and the SPECfp | +| | benchmarks. The SPECint 2006 benchmark contains 12 different | +| | benchmark tests and the SPECfp 2006 benchmark contains 19 | +| | different benchmark tests. | +| | | +| | SPEC CPU 2006 is not always part of a Linux distribution. | +| | SPEC requires that users purchase a license and agree with | +| | their terms and conditions. For this test case, users must | +| | manually download cpu2006-1.2.iso from the SPEC website and | +| | save it under the yardstick/resources folder (e.g. /home/ | +| | opnfv/repos/yardstick/yardstick/resources/cpu2006-1.2.iso) | +| | SPEC CPU® 2006 benchmark is available for purchase via the | +| | SPEC order form (https://www.spec.org/order.html). | +| | | ++--------------+--------------------------------------------------------------+ +|test | This test case uses SPEC CPU 2006 benchmark to measure | +|description | compute-intensive performance of VMs. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc084.yaml | +| | | +| | benchmark_subset is set to int. | +| | | +| | SLA is not available in this test case. | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different: | +| | | +| | * benchmark_subset - a subset of SPEC CPU 2006 benchmarks | +| | to run; | +| | * SPECint_benchmark - a SPECint benchmark to run; | +| | * SPECint_benchmark - a SPECfp benchmark to run; | +| | * output_format - desired report format; | +| | * runspec_config - SPEC CPU 2006 config file provided to | +| | the runspec binary; | +| | * runspec_iterations - the number of benchmark iterations | +| | to execute. For a reportable run, must be 3; | +| | * runspec_tune - tuning to use (base, peak, or all). For a | +| | reportable run, must be either base or all. Reportable | +| | runs do base first, then (optionally) peak; | +| | * runspec_size - size of input data to run (test, train, or | +| | ref). Reportable runs ensure that your binaries can | +| | produce correct results with the test and train workloads | +| | | ++--------------+--------------------------------------------------------------+ +|usability | This test case is used for executing SPEC CPU 2006 benchmark | +| | on virtual machines. The SPECint 2006 benchmark takes | +| | approximately 5 hours. (The time may vary due to different | +| | VM cpu configurations) | +| | | ++--------------+--------------------------------------------------------------+ +|references | spec_cpu_2006_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | To run and install SPEC CPU 2006, the following are | +|conditions | required: | +| | * For SPECint 2006: Both C99 and C++98 compilers are | +| | installed in VM images; | +| | * For SPECfp 2006: All three of C99, C++98 and Fortran-95 | +| | compilers installed in VM images; | +| | * At least 4GB of disk space availabile on VM. | +| | | +| | gcc 4.8.* and g++ 4.8.* version have been tested in Ubuntu | +| | 14.04, Ubuntu 16.04 and Redhat Enterprise Linux 7.4 image. | +| | Higher gcc and g++ version may cause compiling error. | +| | | +| | For more SPEC CPU 2006 dependencies please visit | +| | (https://www.spec.org/cpu2006/Docs/techsupport.html) | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | cpu2006-1.2.iso has been saved under the yardstick/resources | +| | folder (e.g. /home/opnfv/repos/yardstick/yardstick/resources | +| | /cpu2006-1.2.iso). Additionally, to use your custom runspec | +| | config file you can save it under the yardstick/resources/ | +| | files folder and specify the config file name in the | +| | runspec_config parameter. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | Upload SPEC CPU 2006 ISO to the target VM using scp and | +| | install SPEC CPU 2006. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | Connect to the target server using SSH. | +| | If custom runspec config file is used, copy this file from | +| | yardstick to the target VM via the SSH tunnel. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | SPEC CPU 2006 benchmark is invoked and SPEC CPU 2006 metrics | +| | are generated. | +| | | ++--------------+--------------------------------------------------------------+ +|step 5 | Text, HTML, CSV, PDF, and Configuration file outputs for the | +| | SPEC CPU 2006 metrics are fetched from the VM and stored | +| | under /tmp/result folder. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | None. SPEC CPU 2006 results are collected and stored. | +| | | ++--------------+--------------------------------------------------------------+ |