diff options
Diffstat (limited to 'docs/templates')
| -rwxr-xr-x | docs/templates/Yardstick_task_templates.rst | 141 | ||||
| -rw-r--r-- | docs/templates/testcase_description_template.rst | 94 | ||||
| -rw-r--r-- | docs/templates/testcase_description_v2_template.rst | 51 |
3 files changed, 286 insertions, 0 deletions
diff --git a/docs/templates/Yardstick_task_templates.rst b/docs/templates/Yardstick_task_templates.rst new file mode 100755 index 000000000..538937fd7 --- /dev/null +++ b/docs/templates/Yardstick_task_templates.rst @@ -0,0 +1,141 @@ +Task Template Syntax +==================== + +Basic template syntax +--------------------- +A nice feature of the input task format used in Yardstick is that it supports the template syntax based on Jinja2. +This turns out to be extremely useful when, say, you have a fixed structure of your task but you want to +parameterize this task in some way. +For example, imagine your input task file (task.yaml) runs a set of Ping scenarios: + +:: + + # Sample benchmark task config file + # measure network latency using ping + schema: "yardstick:task:0.1" + + scenarios: + - + type: Ping + options: + packetsize: 200 + host: athena.demo + target: ares.demo + + runner: + type: Duration + duration: 60 + interval: 1 + + sla: + max_rtt: 10 + action: monitor + + context: + ... + +Let's say you want to run the same set of scenarios with the same runner/context/sla, +but you want to try another packetsize to compare the performance. +The most elegant solution is then to turn the packetsize name into a template variable: + +:: + + # Sample benchmark task config file + # measure network latency using ping + + schema: "yardstick:task:0.1" + scenarios: + - + type: Ping + options: + packetsize: {{packetsize}} + host: athena.demo + target: ares.demo + + runner: + type: Duration + duration: 60 + interval: 1 + + sla: + max_rtt: 10 + action: monitor + + context: + ... + +and then pass the argument value for {{packetsize}} when starting a task with this configuration file. +Yardstick provides you with different ways to do that: + +1.Pass the argument values directly in the command-line interface (with either a JSON or YAML dictionary): + +:: + + yardstick task start samples/ping-template.yaml --task-args '{"packetsize": "200"}' + +2.Refer to a file that specifies the argument values (JSON/YAML): + +:: + + yardstick task start samples/ping-template.yaml --task-args-file args.yaml + +Using the default values +------------------------ +Note that the Jinja2 template syntax allows you to set the default values for your parameters. +With default values set, your task file will work even if you don't parameterize it explicitly while starting a task. +The default values should be set using the {% set ... %} clause (task.yaml).For example: + +:: + + # Sample benchmark task config file + # measure network latency using ping + schema: "yardstick:task:0.1" + {% set packetsize = packetsize or "100" %} + scenarios: + - + type: Ping + options: + packetsize: {{packetsize}} + host: athena.demo + target: ares.demo + + runner: + type: Duration + duration: 60 + interval: 1 + ... + +If you don't pass the value for {{packetsize}} while starting a task, the default one will be used. + +Advanced templates +------------------ +Yardstick makes it possible to use all the power of Jinja2 template syntax, including the mechanism of built-in functions. +As an example, let us make up a task file that will do a block storage performance test. +The input task file (fio-template.yaml) below uses the Jinja2 for-endfor construct to accomplish that: + +:: + + #Test block sizes of 4KB, 8KB, 64KB, 1MB + #Test 5 workloads: read, write, randwrite, randread, rw + schema: "yardstick:task:0.1" + + scenarios: + {% for bs in ['4k', '8k', '64k', '1024k' ] %} + {% for rw in ['read', 'write', 'randwrite', 'randread', 'rw' ] %} + - + type: Fio + options: + filename: /home/ec2-user/data.raw + bs: {{bs}} + rw: {{rw}} + ramp_time: 10 + host: fio.demo + runner: + type: Duration + duration: 60 + interval: 60 + + {% endfor %} + {% endfor %} + context + ... diff --git a/docs/templates/testcase_description_template.rst b/docs/templates/testcase_description_template.rst new file mode 100644 index 000000000..1651d360c --- /dev/null +++ b/docs/templates/testcase_description_template.rst @@ -0,0 +1,94 @@ +.. Template to be used for test case descriptions in Yardstick Project. + Write one .rst per test case. + Upload the .rst for the test case in /docs/source/yardstick directory. + Review in Gerrit. + +.. image:: ../etc/opnfv-logo.png + :height: 40 + :width: 200 + :alt: OPNFV + :align: left + +****************** +Test Case <slogan> +****************** + +.. contents:: Table of Contents + :depth: 3 + +--------------------- +Test Case Description +--------------------- + +Yardstick Test Case ID +---------------------- + +OPNFV_YARDSTICK_TC<abc>_<slogan> + +where: + - <abc>: check Jira issue for the test case + - <slogan>: check Jira issue for the test case + + +Purpose +------- + +Describe what is the purpose of the test case + +Area +---- + +State the area and sub-area covered by the test case. + +Areas: Compute, Networking, Storage + +Sub-areas: Performance, System limit, QoS + +Metrics +------- + +What will be measured, attribute name or collection of attributes, behavior + +References +---------- + +Reference documentation + +-------------- +Pre-requisites +-------------- + +Tools +----- + +What tools are used to perform the measurements (e.g. fio, pktgen) + + +Configuration +------------- + +State the .yaml file to use. + +State default configuration in the tool(s) used to perform the measurements +(e.g. fio, pktgen). + +State what POD-specific configuration is required to enable running the test +case in different PODs. + + +State SLA, if applicable. + +State test duration. + +------- +Results +------- + +Expected outcome +---------------- + +State applicable graphical presentation + +State applicable output details + +State expected Value, behavior, pass/fail criteria diff --git a/docs/templates/testcase_description_v2_template.rst b/docs/templates/testcase_description_v2_template.rst new file mode 100644 index 000000000..0fa2359e9 --- /dev/null +++ b/docs/templates/testcase_description_v2_template.rst @@ -0,0 +1,51 @@ +.. Template to be used for test case descriptions in Yardstick Project. + Write one .rst per test case. + Upload the .rst for the test case in /docs/source/yardstick directory. + Review in Gerrit. + +.. image:: ../etc/opnfv-logo.png + :height: 40 + :width: 200 + :alt: OPNFV + :align: left + +************************************* +Yardstick Test Case Description TCXXX +************************************* + ++-----------------------------------------------------------------------------+ +|test case slogan e.g. Network Latency | ++==============+==============================================================+ +|test case id | e.g. OPNFV_YARDSTICK_TC001_NW Latency | ++--------------+--------------------------------------------------------------+ +|metric | what will be measured, e.g. latency | ++--------------+--------------------------------------------------------------+ +|test purpose | describe what is the purpose of the test case | ++--------------+--------------------------------------------------------------+ +|configuration | what .yaml file to use, state SLA if applicable, state | +| | test duration, list and describe the scenario options used in| +| | this TC and also list the options using default values. | ++--------------+--------------------------------------------------------------+ +|test tool | e.g. ping | ++--------------+--------------------------------------------------------------+ +|references | e.g. RFCxxx, ETSI-NFVyyy | ++--------------+--------------------------------------------------------------+ +|applicability | describe variations of the test case which can be | +| | performend, e.g. run the test for different packet sizes | ++--------------+--------------------------------------------------------------+ +|pre-test | describe configuration in the tool(s) used to perform | +|conditions | the measurements (e.g. fio, pktgen), POD-specific | +| | configuration required to enable running the test | ++--------------+------+----------------------------------+--------------------+ +|test sequence | step | description | result | +| +------+----------------------------------+--------------------+ +| | 1 | use this to describe tests that | what happens in | +| | | require several steps e.g. | this step | +| | | step 1 collect logs | e.g. logs collected| +| +------+----------------------------------+--------------------+ +| | 2 | remove interface | interface down | +| +------+----------------------------------+--------------------+ +| | N | what is done in step N | what happens | ++--------------+------+----------------------------------+--------------------+ +|test verdict | expected behavior, or SLA, pass/fail criteria | ++--------------+--------------------------------------------------------------+ |