| Age | Commit message (Expand) | Author | Files | Lines |
|---|
| 2017-07-24 | Merge "Open storperf testcase to huawei-pod2" |   Jing Lu | 5 | -12/+30 |
| 2017-07-20 | Add multiqueue support |   Jing Zhang | 2 | -0/+543 |
| 2017-07-19 | heat: port_security_enabled, allowed_address_pairs, disable gateway_ip |   Ross Brattain | 1 | -7/+28 |
| 2017-07-17 | Merge "test_base: speedup unittest, wait for queue not empty" | Ross Brattain | 1 | -2/+8 |
| 2017-07-17 | Open storperf testcase to huawei-pod2 | JingLu5 | 5 | -12/+30 |
| 2017-07-17 | Merge "test_nstat: speedup unittests, sleep for 0 seconds" | Ross Brattain | 1 | -3/+3 |
| 2017-07-17 | Merge "pktgen: speedup unittest, mock time.sleep" | Ross Brattain | 1 | -9/+10 |
| 2017-07-17 | Merge "tg: speedup unittests, mock time.sleep" | Ross Brattain | 3 | -8/+12 |
| 2017-07-17 | Merge "test_pktgen_dpdk_throughput: speedup unittest, mock time.sleep()" | Ross Brattain | 1 | -8/+9 |
| 2017-07-17 | test_nstat: speedup unittests, sleep for 0 seconds | Ross Brattain | 1 | -3/+3 |
| 2017-07-17 | test_heat: ipaddress expects unicode inputs | Ross Brattain | 1 | -14/+14 |
| 2017-07-17 | Merge "test_tg_rfc2544_trex: speedup unittest, set time.sleep(0)" | Ross Brattain | 1 | -4/+5 |
| 2017-07-17 | test_tg_rfc2544_trex: speedup unittest, set time.sleep(0) | Ross Brattain | 1 | -4/+5 |
| 2017-07-17 | test_delete_all_calls_delete: fix delete mock | Ross Brattain | 1 | -4/+6 |
| 2017-07-16 | test_heat: don't waste 2 seconds in unittest | Ross Brattain | 1 | -12/+2 |
| 2017-07-17 | Merge "test_attacker_baremetal: don't run local commands" | ![](//seccdn.libravatar.org/avatar/3d7f8560c23e7613e73c4fe0ad2<style>.highlight .hll { background-color: #ffffcc }
.highlight .c { color: #888888 } /* Comment */
.highlight .err { color: #a61717; background-color: #e3d2d2 } /* Error */
.highlight .k { color: #008800; font-weight: bold } /* Keyword */
.highlight .ch { color: #888888 } /* Comment.Hashbang */
.highlight .cm { color: #888888 } /* Comment.Multiline */
.highlight .cp { color: #cc0000; font-weight: bold } /* Comment.Preproc */
.highlight .cpf { color: #888888 } /* Comment.PreprocFile */
.highlight .c1 { color: #888888 } /* Comment.Single */
.highlight .cs { color: #cc0000; font-weight: bold; background-color: #fff0f0 } /* Comment.Special */
.highlight .gd { color: #000000; background-color: #ffdddd } /* Generic.Deleted */
.highlight .ge { font-style: italic } /* Generic.Emph */
.highlight .gr { color: #aa0000 } /* Generic.Error */
.highlight .gh { color: #333333 } /* Generic.Heading */
.highlight .gi { color: #000000; background-color: #ddffdd } /* Generic.Inserted */
.highlight .go { color: #888888 } /* Generic.Output */
.highlight .gp { color: #555555 } /* Generic.Prompt */
.highlight .gs { font-weight: bold } /* Generic.Strong */
.highlight .gu { color: #666666 } /* Generic.Subheading */
.highlight .gt { color: #aa0000 } /* Generic.Traceback */
.highlight .kc { color: #008800; font-weight: bold } /* Keyword.Constant */
.highlight .kd { color: #008800; font-weight: bold } /* Keyword.Declaration */
.highlight .kn { color: #008800; font-weight: bold } /* Keyword.Namespace */
.highlight .kp { color: #008800 } /* Keyword.Pseudo */
.highlight .kr { color: #008800; font-weight: bold } /* Keyword.Reserved */
.highlight .kt { color: #888888; font-weight: bold } /* Keyword.Type */
.highlight .m { color: #0000DD; font-weight: bold } /* Literal.Number */
.highlight .s { color: #dd2200; background-color: #fff0f0 } /* Literal.String */
.highlight .na { color: #336699 } /* Name.Attribute */
.highlight .nb { color: #003388 } /* Name.Builtin */
.highlight .nc { color: #bb0066; font-weight: bold } /* Name.Class */
.highlight .no { color: #003366; font-weight: bold } /* Name.Constant */
.highlight .nd { color: #555555 } /* Name.Decorator */
.highlight .ne { color: #bb0066; font-weight: bold } /* Name.Exception */
.highlight .nf { color: #0066bb; font-weight: bold } /* Name.Function */
.highlight .nl { color: #336699; font-style: italic } /* Name.Label */
.highlight .nn { color: #bb0066; font-weight: bold } /* Name.Namespace */
.highlight .py { color: #336699; font-weight: bold } /* Name.Property */
.highlight .nt { color: #bb0066; font-weight: bold } /* Name.Tag */
.highlight .nv { color: #336699 } /* Name.Variable */
.highlight .ow { color: #008800 } /* Operator.Word */
.highlight .w { color: #bbbbbb } /* Text.Whitespace */
.highlight .mb { color: #0000DD; font-weight: bold } /* Literal.Number.Bin */
.highlight .mf { color: #0000DD; font-weight: bold } /* Literal.Number.Float */
.highlight .mh { color: #0000DD; font-weight: bold } /* Literal.Number.Hex */
.highlight .mi { color: #0000DD; font-weight: bold } /* Literal.Number.Integer */
.highlight .mo { color: #0000DD; font-weight: bold } /* Literal.Number.Oct */
.highlight .sa { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Affix */
.highlight .sb { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Backtick */
.highlight .sc { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Char */
.highlight .dl { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Delimiter */
.highlight .sd { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Doc */
.highlight .s2 { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Double */
.highlight .se { color: #0044dd; background-color: #fff0f0 } /* Literal.String.Escape */
.highlight .sh { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Heredoc */
.highlight .si { color: #3333bb; background-color: #fff0f0 } /* Literal.String.Interpol */
.highlight .sx { color: #22bb22; background-color: #f0fff0 } /* Literal.String.Other */
.highlight .sr { color: #008800; background-color: #fff0ff } /* Literal.String.Regex */
.highlight .s1 { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Single */
.highlight .ss { color: #aa6600; background-color: #fff0f0 } /* Literal.String.Symbol */
.highlight .bp { color: #003388 } /* Name.Builtin.Pseudo */
.highlight .fm { color: #0066bb; font-weight: bold } /* Name.Function.Magic */
.highlight .vc { color: #336699 } /* Name.Variable.Class */
.highlight .vg { color: #dd7700 } /* Name.Variable.Global */
.highlight .vi { color: #3333bb } /* Name.Variable.Instance */
.highlight .vm { color: #336699 } /* Name.Variable.Magic */
.highlight .il { color: #0000DD; font-weight: bold } /* Literal.Number.Integer.Long */</style><div class="highlight"><pre><span></span><span class="cp">.. This work is licensed under a Creative Commons Attribution 4.0 International</span>
<span class="cp">.. License.</span>
<span class="cp">.. http://creativecommons.org/licenses/by/4.0</span>
<span class="cp">.. (c) 2016 Huawei Technologies Co.,Ltd and others</span>
<span class="gh">============</span>
<span class="gh">Architecture</span>
<span class="gh">============</span>
<span class="gh">Abstract</span>
<span class="gh">========</span>
This chapter describes the yardstick framework software architecture. we will introduce it from Use-Case View,
Logical View, Process View and Deployment View. More technical details will be introduced in this chapter.
<span class="gh">Overview</span>
<span class="gh">========</span>
<span class="gh">Architecture overview</span>
<span class="gh">---------------------</span>
Yardstick is mainly written in Python, and test configurations are made
in YAML. Documentation is written in reStructuredText format, i.e. .rst
files. Yardstick is inspired by Rally. Yardstick is intended to run on a
computer with access and credentials to a cloud. The test case is described
in a configuration file given as an argument.
How it works: the benchmark task configuration file is parsed and converted into
an internal model. The context part of the model is converted into a Heat
template and deployed into a stack. Each scenario is run using a runner, either
serially or in parallel. Each runner runs in its own subprocess executing
commands in a VM using SSH. The output of each scenario is written as json
records to a file or influxdb or http server, we use influxdb as the backend,
the test result will be shown with grafana.
<span class="gh">Concept</span>
<span class="gh">-------</span>
<span class="gs">**Benchmark**</span> - assess the relative performance of something
<span class="gs">**Benchmark**</span> configuration file - describes a single test case in yaml format
<span class="gs">**Context**</span> - The set of Cloud resources used by a scenario, such as user
names, image names, affinity rules and network configurations. A context is
converted into a simplified Heat template, which is used to deploy onto the
Openstack environment.
<span class="gs">**Data**</span> - Output produced by running a benchmark, written to a file in json format
<span class="gs">**Runner**</span> - Logic that determines how a test scenario is run and reported, for
example the number of test iterations, input value stepping and test duration.
Predefined runner types exist for re-usage, see <span class="s">`Runner types`_</span>.
<span class="gs">**Scenario**</span> - Type/class of measurement for example Ping, Pktgen, (Iperf, LmBench, ...)
<span class="gs">**SLA**</span> - Relates to what result boundary a test case must meet to pass. For
example a latency limit, amount or ratio of lost packets and so on. Action
based on <span class="na">:term:</span><span class="nv">`SLA`</span> can be configured, either just to log (monitor) or to stop
further testing (assert). The <span class="na">:term:</span><span class="nv">`SLA`</span> criteria is set in the benchmark
configuration file and evaluated by the runner.
<span class="gh">Runner types</span>
<span class="gh">------------</span>
There exists several predefined runner types to choose between when designing
a test scenario:
<span class="gs">**Arithmetic:**</span>
Every test run arithmetically steps the specified input value(s) in the
test scenario, adding a value to the previous input value. It is also possible
to combine several input values for the same test case in different
combinations.
Snippet of an Arithmetic runner configuration:
::
runner:
type: Arithmetic
iterators:
-
name: stride
start: 64
stop: 128
step: 64
<span class="gs">**Duration:**</span>
The test runs for a specific period of time before completed.
Snippet of a Duration runner configuration:
::
runner:
type: Duration
duration: 30
<span class="gs">**Sequence:**</span>
The test changes a specified input value to the scenario. The input values
to the sequence are specified in a list in the benchmark configuration file.
Snippet of a Sequence runner configuration:
::
runner:
type: Sequence
scenario_option_name: packetsize
sequence:
<span class="m">-</span> 100
<span class="m">-</span> 200
<span class="m">-</span> 250
<span class="gs">**Iteration:**</span>
Tests are run a specified number of times before completed.
Snippet of an Iteration runner configuration:
::
runner:
type: Iteration
iterations: 2
<span class="gh">Use-Case View</span>
<span class="gh">=============</span>
Yardstick Use-Case View shows two kinds of users. One is the Tester who will
do testing in cloud, the other is the User who is more concerned with test result
and result analyses.
For testers, they will run a single test case or test case suite to verify
infrastructure compliance or bencnmark their own infrastructure performance.
Test result will be stored by dispatcher module, three kinds of store method
(file, influxdb and http) can be configured. The detail information of
scenarios and runners can be queried with CLI by testers.
For users, they would check test result with four ways.
If dispatcher module is configured as file(default), there are two ways to
check test result. One is to get result from yardstick.out ( default path:
/tmp/yardstick.out), the other is to get plot of test result, it will be shown
if users execute command "yardstick-plot".
If dispatcher module is configured as influxdb, users will check test
result on Grafana which is most commonly used for visualizing time series data.
If dispatcher module is configured as http, users will check test result
on OPNFV testing dashboard which use MongoDB as backend.
<span class="p">..</span> <span class="ow">image</span><span class="p">::</span> images/Use_case.png
<span class="nc">:width:</span> <span class="nf">800px</span>
<span class="nc">:alt:</span> <span class="nf">Yardstick Use-Case View</span>
<span class="gh">Logical View</span>
<span class="gh">============</span>
Yardstick Logical View describes the most important classes, their
organization, and the most important use-case realizations.
Main classes:
<span class="gs">**TaskCommands**</span> - "yardstick task" subcommand handler.
<span class="gs">**HeatContext**</span> - Do test yaml file context section model convert to HOT,
deploy and undeploy Openstack heat stack.
<span class="gs">**Runner**</span> - Logic that determines how a test scenario is run and reported.
<span class="gs">**TestScenario**</span> - Type/class of measurement for example Ping, Pktgen, (Iperf,
LmBench, ...)
<span class="gs">**Dispatcher**</span> - Choose user defined way to store test results.
TaskCommands is the "yardstick task" subcommand's main entry. It takes yaml
file (e.g. test.yaml) as input, and uses HeatContext to convert the yaml
file's context section to HOT. After Openstack heat stack is deployed by
HeatContext with the converted HOT, TaskCommands use Runner to run specified
TestScenario. During first runner initialization, it will create output
process. The output process use Dispatcher to push test results. The Runner
will also create a process to execute TestScenario. And there is a
multiprocessing queue between each runner process and output process, so the
runner process can push the real-time test results to the storage media.
TestScenario is commonly connected with VMs by using ssh. It sets up VMs and
run test measurement scripts through the ssh tunnel. After all TestScenaio
is finished, TaskCommands will undeploy the heat stack. Then the whole test is
finished.
<span class="p">..</span> <span class="ow">image</span><span class="p">::</span> images/Yardstick_framework_architecture_in_D.png
<span class="nc">:width:</span> <span class="nf">800px</span>
<span class="nc">:alt:</span> <span class="nf">Yardstick framework architecture in Danube</span>
<span class="gh">Process View (Test execution flow)</span>
<span class="gh">==================================</span>
Yardstick process view shows how yardstick runs a test case. Below is the
sequence graph about the test execution flow using heat context, and each
object represents one module in yardstick:
<span class="p">..</span> <span class="ow">image</span><span class="p">::</span> images/test_execution_flow.png
<span class="nc">:width:</span> <span class="nf">800px</span>
<span class="nc">:alt:</span> <span class="nf">Yardstick Process View</span>
A user wants to do a test with yardstick. He can use the CLI to input the
command to start a task. "TaskCommands" will receive the command and ask
"HeatContext" to parse the context. "HeatContext" will then ask "Model" to
convert the model. After the model is generated, "HeatContext" will inform
"Openstack" to deploy the heat stack by heat template. After "Openstack"
deploys the stack, "HeatContext" will inform "Runner" to run the specific test
case.
Firstly, "Runner" would ask "TestScenario" to process the specific scenario.
Then "TestScenario" will start to log on the openstack by ssh protocal and
execute the test case on the specified VMs. After the script execution
finishes, "TestScenario" will send a message to inform "Runner". When the
testing job is done, "Runner" will inform "Dispatcher" to output the test
result via file, influxdb or http. After the result is output, "HeatContext"
will call "Openstack" to undeploy the heat stack. Once the stack is
undepoyed, the whole test ends.
<span class="gh">Deployment View</span>
<span class="gh">===============</span>
Yardstick deployment view shows how the yardstick tool can be deployed into the
underlying platform. Generally, yardstick tool is installed on JumpServer(see
<span class="nv">`07-installation`</span> for detail installation steps), and JumpServer is
connected with other control/compute servers by networking. Based on this
deployment, yardstick can run the test cases on these hosts, and get the test
result for better showing.
<span class="p">..</span> <span class="ow">image</span><span class="p">::</span> images/Deployment.png
<span class="nc">:width:</span> <span class="nf">800px</span>
<span class="nc">:alt:</span> <span class="nf">Yardstick Deployment View</span>
<span class="gh">Yardstick Directory structure</span>
<span class="gh">=============================</span>
<span class="gs">**yardstick/**</span> - Yardstick main directory.
<span class="ge">*tests/ci/*</span> - Used for continuous integration of Yardstick at different PODs and
with support for different installers.
<span class="ge">*docs/*</span> - All documentation is stored here, such as configuration guides,
user guides and Yardstick descriptions.
<span class="ge">*etc/*</span> - Used for test cases requiring specific POD configurations.
<span class="ge">*samples/*</span> - test case samples are stored here, most of all scenario and
feature's samples are shown in this directory.
<span class="ge">*tests/*</span> - Here both Yardstick internal tests (<span class="ge">*functional/*</span> and <span class="ge">*unit/*</span>) as
well as the test cases run to verify the NFVI (<span class="ge">*opnfv/*</span>) are stored.
Also configurations of what to run daily and weekly at the different
PODs is located here.
<span class="ge">*tools/*</span> - Currently contains tools to build image for VMs which are deployed
by Heat. Currently contains how to build the yardstick-trusty-server
image with the different tools that are needed from within the image.
<span class="ge">*plugin/*</span> - Plug-in configuration files are stored here.
<span class="ge">*vTC/*</span> - Contains the files for running the virtual Traffic Classifier tests.
<span class="ge">*yardstick/*</span> - Contains the internals of Yardstick: Runners, Scenario, Contexts,
CLI parsing, keys, plotting tools, dispatcher, plugin
install/remove scripts and so on.
</pre></div>
insertions) +51 |
| 2017-07-06 | Merge "Blacklist TC011 from suites with ODL" | Jing Lu | 5 | -0/+10 |
| 2017-07-05 | Merge "unify pod keywork so api can easily used" |   Kubi | 52 | -126/+172 |
| 2017-07-05 | Merge "Use "OS_INSECURE" variable as the insecure mode indicator" | Kubi | 2 | -2/+2 |
| 2017-07-03 | Yardstick: Aarch64 jenkins slave support |   Alexandru Nemes | 2 | -4/+12 |
| 2017-07-03 | Blacklist TC011 from suites with ODL | Alexandru Nemes | 5 | -0/+10 |
| 2017-07-03 | Merge "Set TEST_DB_URL for storperf" | Jing Lu | 1 | -0/+1 |
