vsperf: Performance Matrix functionality support

The patch expands the vsperf --test-params argument with list
functionality, which enables running multiple tests with different
parameters. If more tests are run then parameters provided, the
last parameters will be reused. Example:
./vsperf --test-params "['','TRAFFICGEN_PKTSIZE = (64,)']"
phy2phy_cont phy2phy_cont
CUMULATIVE_PARAMS if true, instead of using the default settings,
each test will take the parameters of the previous test before
applying it's own.
The patch also adds the vsperf --matrix argument which analyzes and
compares the results of all the tests run, printing it as a table,
as well as saving it into a file in the results directory.
MATRIX_METRIC metric used by Performance Matrix to compare tests.

JIRA: VSPERF-554

Change-Id: I71530ddf110890612236a7e57039f249609c835b
Signed-off-by: Richard Elias <richardx.elias@intel.com>
Reviewed-by: Martin Klozik <martinx.klozik@intel.com>
Reviewed-by: Al Morton <acmorton@att.com>
Reviewed-by: Christian Trautman <ctrautma@redhat.com>
Reviewed-by: Sridhar Rao <sridhar.rao@spirent.com>

1 files changed, 24 insertions, 3 deletions

diff --git a/conf/00_common.conf b/conf/00_common.conf
index 279c67b4..d54c8a5a 100644
--- a/conf/00_common.conf
+++ b/conf/00_common.conf
@@ -90,9 +90,6 @@ PATHS = {}
 # shell command to use when running commands through Pexpect
 SHELL_CMD = ['/bin/bash', '-c']
 
-# internal list to keep track of PIDs of jobs executed by vsperf
-_EXECUTED_PIDS = []
-
 # ############################
 # Logging configuration
 # ############################
@@ -122,6 +119,15 @@ TEST_PARAMS = {}
 # delay enforced after every step to allow system to process changes
 TEST_STEP_DELAY = 5
 
+# parameter used, when running mupltiple tests, to accumulate _PARAMS_LIST
+# parameters for multiple tests running in a series
+CUMULATIVE_PARAMS = False
+
+# metric used by the performance matrix for comparision and analysis
+# of tests run in a series. Must always refer to a numeric value.
+# For example: 'throughput_rx_mbps', 'throughput_rx_fps', 'avg_latency_ns'
+MATRIX_METRIC = 'throughput_rx_fps'
+
 # ############################
 # Modules
 # ############################
@@ -131,3 +137,18 @@ TEST_STEP_DELAY = 5
 # Example:
 #   EXCLUDE_MODULES = ['ovs_vanilla', 'qemu_virtio_net', 'pidstat']
 EXCLUDE_MODULES = ["testcenter-rfc2544-throughput"]
+
+# ############################
+# Vsperf Internal Options
+# ############################
+# following options should not be changed by the user
+
+# internal list to keep track of PIDs of jobs executed by vsperf
+_EXECUTED_PIDS = []
+
+# dictionary containing the test-specific parameters of all tests being run
+# for the purposes of cummulative parameter assignment using performance matrix
+_PARAMS_LIST = {}
+
+# index number of the current test, used for naming of result files
+_TEST_INDEX = 0