cli: Modify configuration via CLI

In the past, only a few configuration parameters could be
modified via --test-params CLI argument and it was not easy
to find out their complete list.
This patch adds support for generic modification of any
configuration parameter via CLI argument --test-params
or by "Parameters" section of testcase definition.
Thus it is possible to customize a vsperf configuration
environment per testcase or for each vsperf execution.
Old CLI parameters duration, pkt_sizes, rfc2544_tests
and rfc2889_trials were renamed to TRAFFICGEN_DURATION,
TRAFFICGEN_PKT_SIZES, TRAFFICGEN_RFC2544_TESTS and
TRAFFICGEN_RFC2889_TRIALS to be consistent with
other configuration parameters.

JIRA: VSPERF-375

Change-Id: I50a1f4ff7250d754aa8af0295a9c7c1be8151175
Signed-off-by: Martin Klozik <martinx.klozik@intel.com>
Reviewed-by: Al Morton <acmorton@att.com>
Reviewed-by: Christian Trautman <ctrautma@redhat.com>
Reviewed-by: Bill Michalowski <bmichalo@redhat.com>
Reviewed-by: Antonio Fischetti <antonio.fischetti@intel.com>
Reviewed-by: <sridhar.rao@spirent.com>

6 files changed, 83 insertions, 41 deletions

diff --git a/docs/configguide/trafficgen.rst b/docs/configguide/trafficgen.rst
index 85fc35b8..efcc4d83 100644
--- a/docs/configguide/trafficgen.rst
+++ b/docs/configguide/trafficgen.rst
@@ -69,7 +69,7 @@ OR from the commandline:
 
 .. code-block:: console
 
-    $ ./vsperf --test-params "pkt_sizes=x,y" $TESTNAME
+    $ ./vsperf --test-params "TRAFFICGEN_PKT_SIZES=(x,y)" $TESTNAME
 
 You can also modify the traffic transmission duration and the number
 of tests run by the traffic generator by extending the example
@@ -77,7 +77,8 @@ commandline above to:
 
 .. code-block:: console
 
-    $ ./vsperf --test-params "pkt_sizes=x,y;duration=10;rfc2544_tests=1" $TESTNAME
+    $ ./vsperf --test-params "TRAFFICGEN_PKT_SIZES=(x,y);TRAFFICGEN_DURATION=10;" \
+                             "TRAFFICGEN_RFC2544_TESTS=1" $TESTNAME
 
 Dummy Setup
 ------------
diff --git a/docs/design/vswitchperf_design.rst b/docs/design/vswitchperf_design.rst
index 375fa12e..88b6ba88 100755
--- a/docs/design/vswitchperf_design.rst
+++ b/docs/design/vswitchperf_design.rst
@@ -53,7 +53,8 @@ for development purposes:
 
 .. code-block:: console
 
-   $ ./vsperf --test-params 'duration=10;rfc2544_tests=1;pkt_sizes=64' --tests 'pvp_tput'
+   $ ./vsperf --test-params 'TRAFFICGEN_DURATION=10;TRAFFICGEN_RFC2544_TESTS=1;' \
+                            'TRAFFICGEN_PKT_SIZES=(64,)' pvp_tput
 
 Typical Test Sequence
 =====================
diff --git a/docs/requirements/vswitchperf_ltp.rst b/docs/requirements/vswitchperf_ltp.rst
index 6b2ee9bc..14303de5 100644
--- a/docs/requirements/vswitchperf_ltp.rst
+++ b/docs/requirements/vswitchperf_ltp.rst
@@ -1304,7 +1304,7 @@ vsperf CI jobs are broken down into:
     * Runs everyday takes about 10 hours to complete.
     * TESTCASES_DAILY='phy2phy_tput back2back phy2phy_tput_mod_vlan
       phy2phy_scalability pvp_tput pvp_back2back pvvp_tput pvvp_back2back'.
-    * TESTPARAM_DAILY='--test-params pkt_sizes=64,128,512,1024,1518'.
+    * TESTPARAM_DAILY='--test-params TRAFFICGEN_PKT_SIZES=(64,128,512,1024,1518)'.
 
   * Merge job:
 
diff --git a/docs/userguide/integration.rst b/docs/userguide/integration.rst
index 5b26a716..003e8adb 100755
--- a/docs/userguide/integration.rst
+++ b/docs/userguide/integration.rst
@@ -81,21 +81,21 @@ To run VXLAN encapsulation tests:
 
   .. code-block:: console
 
-    ./vsperf --conf-file user_settings.py --integration
+    ./vsperf --conf-file user_settings.py --integration \
              --test-params 'tunnel_type=vxlan' overlay_p2p_tput
 
 To run GRE encapsulation tests:
 
   .. code-block:: console
 
-    ./vsperf --conf-file user_settings.py --integration
+    ./vsperf --conf-file user_settings.py --integration \
              --test-params 'tunnel_type=gre' overlay_p2p_tput
 
 To run GENEVE encapsulation tests:
 
   .. code-block:: console
 
-    ./vsperf --conf-file user_settings.py --integration
+    ./vsperf --conf-file user_settings.py --integration \
              --test-params 'tunnel_type=geneve' overlay_p2p_tput
 
 To run OVS NATIVE tunnel tests (VXLAN/GRE/GENEVE):
@@ -127,7 +127,7 @@ To run OVS NATIVE tunnel tests (VXLAN/GRE/GENEVE):
 
   .. code-block:: console
 
-    ./vsperf --conf-file user_settings.py --integration
+    ./vsperf --conf-file user_settings.py --integration \
              --test-params 'tunnel_type=vxlan' overlay_p2p_tput
 
 
@@ -189,7 +189,7 @@ To run GRE decapsulation tests:
 
   .. code-block:: console
 
-    ./vsperf --conf-file user_settings.py --test-params 'tunnel_type=gre'
+    ./vsperf --conf-file user_settings.py --test-params 'tunnel_type=gre' \
              --integration overlay_p2p_decap_cont
 
 
@@ -246,7 +246,7 @@ To run GENEVE decapsulation tests:
 
   .. code-block:: console
 
-    ./vsperf --conf-file user_settings.py --test-params 'tunnel_type=geneve'
+    ./vsperf --conf-file user_settings.py --test-params 'tunnel_type=geneve' \
              --integration overlay_p2p_decap_cont
 
 
@@ -330,7 +330,7 @@ To run VXLAN decapsulation tests:
 
   .. code-block:: console
 
-    ./vsperf --conf-file user_settings.py --integration
+    ./vsperf --conf-file user_settings.py --integration \
              --test-params 'tunnel_type=vxlan' overlay_p2p_decap_cont
 
 Executing Native/Vanilla OVS GRE decapsulation tests
@@ -389,7 +389,7 @@ To run GRE decapsulation tests:
 
   .. code-block:: console
 
-    ./vsperf --conf-file user_settings.py --integration
+    ./vsperf --conf-file user_settings.py --integration \
              --test-params 'tunnel_type=gre' overlay_p2p_decap_cont
 
 Executing Native/Vanilla OVS GENEVE decapsulation tests
@@ -448,7 +448,7 @@ To run GENEVE decapsulation tests:
 
   .. code-block:: console
 
-    ./vsperf --conf-file user_settings.py --integration
+    ./vsperf --conf-file user_settings.py --integration \
              --test-params 'tunnel_type=geneve' overlay_p2p_decap_cont
 
 
@@ -498,5 +498,5 @@ To run VXLAN encapsulation+decapsulation tests:
 
   .. code-block:: console
 
-    ./vsperf --conf-file user_settings.py --integration
+    ./vsperf --conf-file user_settings.py --integration \
              overlay_p2p_mod_tput
diff --git a/docs/userguide/teststeps.rst b/docs/userguide/teststeps.rst
index 51e62020..65a25b0a 100644
--- a/docs/userguide/teststeps.rst
+++ b/docs/userguide/teststeps.rst
@@ -523,8 +523,8 @@ To run the test:
 
   .. code-block:: console
 
-    ./vsperf --conf-file user_settings.py --test-params
-            "guest_loopback=linux_bridge" --integration ex_pvp_rule_l3da
+    ./vsperf --conf-file user_settings.py --test-params \
+            "GUEST_LOOPBACK=['linux_bridge']" --integration ex_pvp_rule_l3da
 
 Forward packets based on UDP port
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
diff --git a/docs/userguide/testusage.rst b/docs/userguide/testusage.rst
index 97c74721..f446f261 100755
--- a/docs/userguide/testusage.rst
+++ b/docs/userguide/testusage.rst
@@ -105,14 +105,61 @@ or via another command line argument will override both the default and
 your custom configuration files. This "priority hierarchy" can be
 described like so (1 = max priority):
 
-1. Command line arguments
-2. Environment variables
-3. Configuration file(s)
+1. Testcase definition section ``Parameters``
+2. Command line arguments
+3. Environment variables
+4. Configuration file(s)
 
 Further details about configuration files evaluation and special behaviour
 of options with ``GUEST_`` prefix could be found at `design document
 <http://artifacts.opnfv.org/vswitchperf/docs/design/vswitchperf_design.html#configuration>`__.
 
+Overriding values defined in configuration files
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The configuration items can be overridden by command line argument
+``--test-params``. In this case, the configuration items and
+their values should be passed in form of ``item=value`` and separated
+by semicolon.
+
+Example:
+
+.. code:: console
+
+    $ ./vsperf --test-params "TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,);" \
+                             "GUEST_LOOPBACK=['testpmd','l2fwd']" pvvp_tput
+
+The second option is to override configuration items by ``Parameters`` section
+of the test case definition. The configuration items can be added into ``Parameters``
+dictionary with their new values. These values will override values defined in
+configuration files or specified by ``--test-params`` command line argument.
+
+Example:
+
+.. code:: python
+
+    "Parameters" : {'TRAFFICGEN_PKT_SIZES' : (128,),
+                    'TRAFFICGEN_DURATION' : 10,
+                    'GUEST_LOOPBACK' : ['testpmd','l2fwd'],
+                   }
+
+**NOTE:** In both cases, configuration item names and their values must be specified
+in the same form as they are defined inside configuration files. Parameter names
+must be specified in uppercase and data types of original and new value must match.
+Python syntax rules related to data types and structures must be followed.
+For example, parameter ``TRAFFICGEN_PKT_SIZES`` above is defined as a tuple
+with a single value ``128``. In this case trailing comma is mandatory, otherwise
+value can be wrongly interpreted as a number instead of a tuple and vsperf
+execution would fail. Please check configuration files for default values and their
+types and use them as a basis for any customized values. In case of any doubt, please
+check official python documentation related to data structures like tuples, lists
+and dictionaries.
+
+**NOTE:** Vsperf execution will terminate with runtime error in case, that unknown
+parameter name is passed via ``--test-params`` CLI argument or defined in ``Parameters``
+section of test case definition. It is also forbidden to redefine a value of
+``TEST_PARAMS`` configuration item via CLI or ``Parameters`` section.
+
 vloop_vnf
 ^^^^^^^^^
 
@@ -192,9 +239,9 @@ Some tests allow for configurable parameters, including test duration
 
 .. code:: bash
 
-    $ ./vsperf --conf-file user_settings.py
-        --tests RFC2544Tput
-        --test-params "duration=10;pkt_sizes=128"
+    $ ./vsperf --conf-file user_settings.py \
+        --tests RFC2544Tput \
+        --test-params "TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)"
 
 For all available options, check out the help dialog:
 
@@ -283,9 +330,11 @@ To run tests using Vanilla OVS:
 
    .. code-block:: console
 
-       $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
-                  --test-params "vanilla_tgen_tx_ip=n.n.n.n;
-                                 vanilla_tgen_tx_mac=nn:nn:nn:nn:nn:nn"
+       $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
+                  --test-params "VANILLA_TGEN_PORT1_IP=n.n.n.n;" \
+                                "VANILLA_TGEN_PORT1_MAC=nn:nn:nn:nn:nn:nn;" \
+                                "VANILLA_TGEN_PORT2_IP=n.n.n.n;" \
+                                "VANILLA_TGEN_PORT2_MAC=nn:nn:nn:nn:nn:nn"
 
 2. If needed, recompile src for all OVS variants
 
@@ -389,7 +438,7 @@ Execution of test with PCI passthrough with vswitch disabled:
 
 .. code-block:: console
 
-    $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
+    $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
                --vswitch none --vnf QemuPciPassthrough pvp_tput
 
 Any of supported guest-loopback-application_ can be used inside VM with
@@ -414,8 +463,8 @@ or use ``--test-params`` CLI argument:
 
 .. code-block:: console
 
-        $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
-              --test-params "guest_loopback=testpmd"
+        $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
+              --test-params "GUEST_LOOPBACK=['testpmd']"
 
 Supported loopback applications are:
 
@@ -549,15 +598,6 @@ environment.
 on the same numa as the NIC in use if possible/applicable. Testpmd should be
 assigned at least (nb_cores +1) total cores with the cpu mask.
 
-The following CLI parameters override the corresponding configuration settings:
-
-1. ``guest_nic_queues``, which overrides all ``GUEST_NIC_QUEUES`` values
-2. ``guest_testpmd_params``, which overrides all ``GUEST_TESTPMD_PARAMS``
-   values
-3. ``vswitch_dpdk_multi_queues``, which overrides ``VSWITCH_DPDK_MULTI_QUEUES``
-4. ``guest_smp``, which overrides all ``GUEST_SMP`` values
-5. ``guest_core_binding``, which overrides all ``GUEST_CORE_BINDING`` values
-
 Executing Packet Forwarding tests
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -573,8 +613,8 @@ or use ``--vswitch`` and ``--fwdapp`` CLI arguments:
 
 .. code-block:: console
 
-    $ ./vsperf --conf-file user_settings.py
-               --vswitch none
+    $ ./vsperf --conf-file user_settings.py \
+               --vswitch none \
                --fwdapp TestPMD
 
 Supported Packet Forwarding applications are:
@@ -629,7 +669,7 @@ Mode of operation is driven by configuration parameter -m or --mode
             "trafficgen-pause" - execute vSwitch and VNF but wait before traffic transmission
 
 In case, that VSPERF is executed in "trafficgen" mode, then configuration
-of traffic generator should be configured through --test-params option.
+of traffic generator should be configured through ``--test-params`` option.
 Supported CLI options useful for traffic generator configuration are:
 
 .. code-block:: console
@@ -656,7 +696,7 @@ Example of execution of VSPERF in "trafficgen" mode:
 
 .. code-block:: console
 
-    $ ./vsperf -m trafficgen --trafficgen IxNet --conf-file vsperf.conf
+    $ ./vsperf -m trafficgen --trafficgen IxNet --conf-file vsperf.conf \
         --test-params "traffic_type=continuous;bidirectional=True;iload=60"
 
 Code change verification by pylint