Merge "Document for Fraser test case results" into stable/fraser

1 files changed, 602 insertions, 0 deletions

diff --git a/docs/release/results/euphrates_fraser_comparison.rst b/docs/release/results/euphrates_fraser_comparison.rst
new file mode 100644
index 000000000..53dfb994f
--- /dev/null
+++ b/docs/release/results/euphrates_fraser_comparison.rst
@@ -0,0 +1,602 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+
+=======================================================
+Test results analysis for Euphrates and Fraser releases
+=======================================================
+
+TC002
+-----
+
+The round-trip-time (RTT) between 2 VMs on different blades is measured using
+ping.
+
+Most test run measurements result on average between 0.39 and 4.00 ms.
+Compared with Euphrates release, the average RTT result of the same pod experiences
+a slight decline in Fraser release. For example, the average RTT of arm-pod5 is
+1.518 in Ehphrates and 1.714 in Fraser. The average RTT of intel-pod18 is 1.6575
+ms in Ehphrates and 1.856 ms in Fraser.
+
+{
+
+    "huawei-pod2:stable/euphrates": [0.3925],
+
+    "lf-pod2:stable/euphrates": [0.5315],
+
+    "lf-pod1:stable/euphrates": [0.62],
+
+    "huawei-pod2:stable/fraser": [0.677],
+
+    "lf-pod1:stable/fraser": [0.725],
+
+    "flex-pod2:stable/euphrates": [0.795],
+
+    "huawei-pod12:stable/euphrates": [0.87],
+
+    "ericsson-pod1:stable/fraser": [0.9165],
+
+    "huawei-pod12:stable/fraser": [1.0465],
+
+    "lf-pod2:stable/fraser": [1.2325],
+
+    "intel-pod5:stable/euphrates": [1.25],
+
+    "ericsson-virtual3:stable/euphrates": [1.2655],
+
+    "ericsson-pod1:stable/euphrates": [1.372],
+
+    "zte-pod2:stable/fraser": [1.395],
+
+    "arm-pod5:stable/euphrates": [1.518],
+
+    "huawei-virtual4:stable/euphrates": [1.5355],
+
+    "ericsson-virtual4:stable/fraser": [1.582],
+
+    "huawei-virtual3:stable/euphrates": [1.606],
+
+    "intel-pod18:stable/euphrates": [1.6575],
+
+    "huawei-virtual4:stable/fraser": [1.697],
+
+    "huawei-virtual8:stable/euphrates": [1.709],
+
+    "arm-pod5:stable/fraser": [1.714],
+
+    "huawei-virtual3:stable/fraser": [1.716],
+
+    "intel-pod18:stable/fraser": [1.856],
+
+    "huawei-virtual2:stable/euphrates": [1.872],
+
+    "arm-pod6:stable/euphrates": [1.895],
+
+    "huawei-virtual2:stable/fraser": [1.964],
+
+    "huawei-virtual1:stable/fraser": [1.9765],
+
+    "huawei-virtual9:stable/euphrates": [2.0745],
+
+    "arm-pod6:stable/fraser": [2.209],
+
+    "huawei-virtual1:stable/euphrates": [2.495],
+
+    "ericsson-virtual2:stable/euphrates": [2.7895],
+
+    "ericsson-virtual4:stable/euphrates": [3.768],
+
+    "ericsson-virtual1:stable/euphrates": [3.8035],
+
+    "ericsson-virtual3:stable/fraser": [3.9175],
+
+    "ericsson-virtual2:stable/fraser": [4.004]
+
+}
+
+TC010
+-----
+
+The tool we use to measure memory read latency is lmbench, which is a series of
+micro benchmarks intended to measure basic operating system and hardware system
+metrics. Compared with Euphrates release, the memory read latency of the same pod
+also experience a slight decline. Virtual pods seem to have a higher memory read
+latency than physical pods. Compared with X86 pods, the memory read latency of
+arm pods is significant higher.
+
+{
+
+    "ericsson-pod1:stable/euphrates": [5.7785],
+
+    "flex-pod2:stable/euphrates": [5.908],
+
+    "ericsson-virtual1:stable/euphrates": [6.412],
+
+    "intel-pod18:stable/euphrates": [6.5905],
+
+    "intel-pod5:stable/euphrates": [6.6975],
+
+    "ericsson-pod1:stable/fraser": [7.0645],
+
+    "ericsson-virtual4:stable/euphrates": [7.183],
+
+    "intel-pod18:stable/fraser": [7.4465],
+
+    "zte-pod2:stable/fraser": [8.1865],
+
+    "ericsson-virtual2:stable/euphrates": [8.4985],
+
+    "huawei-pod2:stable/euphrates": [8.877],
+
+    "huawei-pod12:stable/euphrates": [9.091],
+
+    "huawei-pod2:stable/fraser": [9.236],
+
+    "huawei-pod12:stable/fraser": [9.615],
+
+    "ericsson-virtual3:stable/euphrates": [9.719],
+
+    "ericsson-virtual2:stable/fraser": [9.8925],
+
+    "huawei-virtual4:stable/euphrates": [10.1195],
+
+    "huawei-virtual3:stable/euphrates": [10.19],
+
+    "huawei-virtual2:stable/fraser": [10.22],
+
+    "huawei-virtual1:stable/euphrates": [10.3045],
+
+    "huawei-virtual9:stable/euphrates": [10.318],
+
+    "ericsson-virtual4:stable/fraser": [10.5465],
+
+    "ericsson-virtual3:stable/fraser": [10.9355],
+
+    "huawei-virtual3:stable/fraser": [10.95],
+
+    "huawei-virtual2:stable/euphrates": [11.274],
+
+    "huawei-virtual4:stable/fraser": [11.557],
+
+    "lf-pod1:stable/euphrates": [15.7025],
+
+    "lf-pod2:stable/euphrates": [15.8495],
+
+    "lf-pod2:stable/fraser": [16.5595],
+
+    "lf-pod1:stable/fraser": [16.8395],
+
+    "arm-pod5:stable/euphrates": [18.092],
+
+    "arm-pod5:stable/fraser": [18.744],
+
+    "huawei-virtual1:stable/fraser": [19.8235],
+
+    "huawei-virtual8:stable/euphrates": [33.999],
+
+    "arm-pod6:stable/euphrates": [41.5605],
+
+    "arm-pod6:stable/fraser": [55.804]
+
+}
+
+TC011
+-----
+
+Iperf3 is a tool for evaluating the packet delay variation between 2 VMs on
+different blades. In general, the packet delay variation of the two releases
+look similar.
+
+{
+
+    "arm-pod6:stable/fraser": [1],
+
+    "ericsson-pod1:stable/fraser": [1],
+
+    "ericsson-virtual2:stable/fraser": [1],
+
+    "ericsson-virtual3:stable/fraser": [1],
+
+    "lf-pod2:stable/fraser": [1],
+
+    "huawei-virtual1:stable/fraser": [2997],
+
+    "huawei-virtual2:stable/euphrates": [2997],
+
+    "flex-pod2:stable/euphrates": [2997.5],
+
+    "huawei-virtual3:stable/euphrates": [2998],
+
+    "huawei-virtual3:stable/fraser": [2999],
+
+    "huawei-virtual9:stable/euphrates": [3000],
+
+    "huawei-virtual8:stable/euphrates": [3001],
+
+    "huawei-virtual4:stable/euphrates": [3002],
+
+    "huawei-virtual4:stable/fraser": [3002],
+
+    "ericsson-virtual3:stable/euphrates": [3006],
+
+    "huawei-virtual1:stable/euphrates": [3007],
+
+    "ericsson-virtual2:stable/euphrates": [3009],
+
+    "intel-pod18:stable/euphrates": [3010],
+
+    "ericsson-virtual4:stable/euphrates": [3017],
+
+    "lf-pod2:stable/euphrates": [3021],
+
+    "arm-pod5:stable/euphrates": [3022],
+
+    "arm-pod6:stable/euphrates": [3022],
+
+    "ericsson-pod1:stable/euphrates": [3022],
+
+    "huawei-pod12:stable/euphrates": [3022],
+
+    "huawei-pod12:stable/fraser": [3022],
+
+    "huawei-pod2:stable/euphrates": [3022],
+
+    "huawei-pod2:stable/fraser": [3022],
+
+    "intel-pod18:stable/fraser": [3022],
+
+    "intel-pod5:stable/euphrates": [3022],
+
+    "lf-pod1:stable/euphrates": [3022],
+
+    "lf-pod1:stable/fraser": [3022],
+
+    "zte-pod2:stable/fraser": [3022],
+
+    "huawei-virtual2:stable/fraser": [3025]
+
+}
+
+TC012
+-----
+
+Lmbench is also used to measure the memory read and write bandwidth.
+Like TC010, compared with Euphrates release, the memory read and write bandwidth
+of the same pod also experience a slight decline. And compared with X86 pods, the memory
+read and write bandwidth of arm pods is significant lower.
+
+{
+
+    "lf-pod1:stable/euphrates": [22912.39],
+
+    "lf-pod2:stable/euphrates": [22637.67],
+
+    "lf-pod1:stable/fraser": [20552.9],
+
+    "flex-pod2:stable/euphrates": [20229.99],
+
+    "lf-pod2:stable/fraser": [20058.925],
+
+    "ericsson-pod1:stable/fraser": [18930.78],
+
+    "intel-pod18:stable/fraser": [18757.545],
+
+    "ericsson-virtual1:stable/euphrates": [17474.965],
+
+    "ericsson-pod1:stable/euphrates": [17127.38],
+
+    "ericsson-virtual4:stable/euphrates": [16219.97],
+
+    "ericsson-virtual2:stable/euphrates": [15652.28],
+
+    "ericsson-virtual3:stable/euphrates": [15551.26],
+
+    "ericsson-virtual4:stable/fraser": [15389.465],
+
+    "ericsson-virtual2:stable/fraser": [15343.79],
+
+    "huawei-pod2:stable/euphrates": [15017.2],
+
+    "huawei-pod2:stable/fraser": [14870.78],
+
+    "huawei-virtual4:stable/euphrates": [14266.34],
+
+    "huawei-virtual1:stable/euphrates": [14233.035],
+
+    "huawei-virtual3:stable/euphrates": [14227.63],
+
+    "zte-pod2:stable/fraser": [14157.99],
+
+    "huawei-pod12:stable/euphrates": [14147.245],
+
+    "huawei-pod12:stable/fraser": [14126.99],
+
+    "intel-pod18:stable/euphrates": [14058.33],
+
+    "huawei-virtual3:stable/fraser": [13929.67],
+
+    "huawei-virtual2:stable/euphrates": [13862.85],
+
+    "huawei-virtual4:stable/fraser": [13847.155],
+
+    "huawei-virtual2:stable/fraser": [13702.92],
+
+    "huawei-virtual1:stable/fraser": [13496.45],
+
+    "intel-pod5:stable/euphrates": [13280.32],
+
+    "ericsson-virtual3:stable/fraser": [12733.19],
+
+    "huawei-virtual9:stable/euphrates": [12559.445],
+
+    "huawei-virtual8:stable/euphrates": [8998.02],
+
+    "arm-pod5:stable/euphrates": [4388.875],
+
+    "arm-pod5:stable/fraser": [4326.11],
+
+    "arm-pod6:stable/euphrates": [4260.2],
+
+    "arm-pod6:stable/fraser": [3809.885]
+
+}
+
+TC014
+-----
+
+The Unixbench is used to evaluate the IaaS processing speed with regards to
+score of single CPU running and parallel running. Below are the single CPU running
+scores. It can be seen that the processing test results vary from scores 715 to 3737.
+In general, the single CPU score of the two releases look similar.
+
+{
+
+    "lf-pod2:stable/fraser": [3737.6],
+
+    "lf-pod2:stable/euphrates": [3723.95],
+
+    "lf-pod1:stable/fraser": [3702.7],
+
+    "lf-pod1:stable/euphrates": [3669],
+
+    "intel-pod5:stable/euphrates": [3388.6],
+
+    "intel-pod18:stable/euphrates": [3298.4],
+
+    "flex-pod2:stable/euphrates": [3208.6],
+
+    "ericsson-pod1:stable/fraser": [3131.6],
+
+    "intel-pod18:stable/fraser": [3098.1],
+
+    "ericsson-virtual1:stable/euphrates": [2988.9],
+
+    "zte-pod2:stable/fraser": [2831.4],
+
+    "ericsson-pod1:stable/euphrates": [2669.1],
+
+    "ericsson-virtual4:stable/euphrates": [2598.5],
+
+    "ericsson-virtual2:stable/fraser": [2559.7],
+
+    "ericsson-virtual3:stable/euphrates": [2553.15],
+
+    "huawei-pod2:stable/euphrates": [2531.2],
+
+    "huawei-pod2:stable/fraser": [2528.9],
+
+    "ericsson-virtual4:stable/fraser": [2527.8],
+
+    "ericsson-virtual2:stable/euphrates": [2526.9],
+
+    "huawei-virtual4:stable/euphrates": [2407.4],
+
+    "huawei-virtual3:stable/fraser": [2379.1],
+
+    "huawei-virtual3:stable/euphrates": [2374.6],
+
+    "huawei-virtual4:stable/fraser": [2362.1],
+
+    "huawei-virtual2:stable/euphrates": [2326.4],
+
+    "huawei-virtual9:stable/euphrates": [2324.95],
+
+    "huawei-virtual1:stable/euphrates": [2302.6],
+
+    "huawei-virtual2:stable/fraser": [2299.3],
+
+    "huawei-pod12:stable/euphrates": [2232.2],
+
+    "huawei-pod12:stable/fraser": [2229],
+
+    "huawei-virtual1:stable/fraser": [2171.3],
+
+    "ericsson-virtual3:stable/fraser": [2104.8],
+
+    "huawei-virtual8:stable/euphrates": [2085.3],
+
+    "arm-pod5:stable/fraser": [1764.2],
+
+    "arm-pod5:stable/euphrates": [1754.4],
+
+    "arm-pod6:stable/euphrates": [716.15],
+
+    "arm-pod6:stable/fraser": [715.4]
+
+}
+
+TC069
+-----
+
+With the block size changing from 1 kb to 512 kb, the memory write bandwidth
+tends to become larger first and then smaller within every run test. Below are
+the scores for 32mb block array.
+
+{
+
+    "intel-pod18:stable/euphrates": [18871.79],
+
+    "intel-pod18:stable/fraser": [16939.24],
+
+    "intel-pod5:stable/euphrates": [16055.79],
+
+    "arm-pod6:stable/euphrates": [13327.02],
+
+    "arm-pod6:stable/fraser": [11895.71],
+
+    "flex-pod2:stable/euphrates": [9384.585],
+
+    "zte-pod2:stable/fraser": [9375.33],
+
+    "ericsson-pod1:stable/euphrates": [9331.535],
+
+    "huawei-pod12:stable/euphrates": [9164.88],
+
+    "ericsson-pod1:stable/fraser": [9140.42],
+
+    "huawei-pod2:stable/euphrates": [9026.52],
+
+    "huawei-pod12:stable/fraser": [8993.37],
+
+    "huawei-virtual9:stable/euphrates": [8825.805],
+
+    "huawei-pod2:stable/fraser": [8794.01],
+
+    "huawei-virtual2:stable/fraser": [7670.21],
+
+    "ericsson-virtual1:stable/euphrates": [7615.97],
+
+    "ericsson-virtual4:stable/euphrates": [7539.23],
+
+    "arm-pod5:stable/fraser": [7479.32],
+
+    "arm-pod5:stable/euphrates": [7403.38],
+
+    "huawei-virtual3:stable/euphrates": [7247.89],
+
+    "ericsson-virtual2:stable/fraser": [7219.21],
+
+    "huawei-virtual2:stable/euphrates": [7205.35],
+
+    "huawei-virtual1:stable/euphrates": [7196.405],
+
+    "ericsson-virtual3:stable/euphrates": [7173.72],
+
+    "huawei-virtual4:stable/euphrates": [7131.47],
+
+    "ericsson-virtual2:stable/euphrates": [7129.08],
+
+    "huawei-virtual4:stable/fraser": [7059.045],
+
+    "huawei-virtual3:stable/fraser": [7023.57],
+
+    "lf-pod1:stable/euphrates": [6928.18],
+
+    "lf-pod2:stable/euphrates": [6875.88],
+
+    "lf-pod2:stable/fraser": [6834.7],
+
+    "lf-pod1:stable/fraser": [6775.27],
+
+    "ericsson-virtual4:stable/fraser": [6522.86],
+
+    "ericsson-virtual3:stable/fraser": [5835.59],
+
+    "huawei-virtual8:stable/euphrates": [5729.705],
+
+    "huawei-virtual1:stable/fraser": [5617.12]
+
+}
+
+TC082
+-----
+
+For this test case, we use perf to measure context-switches under load.
+High context switch rates are not themselves an issue, but they may point the
+way to a more significant problem.
+
+{
+
+    "zte-pod2:stable/fraser": [306.5],
+
+    "huawei-pod12:stable/euphrates": [316],
+
+    "lf-pod2:stable/fraser": [337.5],
+
+    "intel-pod18:stable/euphrates": [340],
+
+    "intel-pod18:stable/fraser": [343.5],
+
+    "intel-pod5:stable/euphrates": [357.5],
+
+    "ericsson-pod1:stable/euphrates": [384],
+
+    "lf-pod2:stable/euphrates": [394.5],
+
+    "huawei-pod12:stable/fraser": [399],
+
+    "lf-pod1:stable/euphrates": [435],
+
+    "lf-pod1:stable/fraser": [454],
+
+    "flex-pod2:stable/euphrates": [476],
+
+    "huawei-pod2:stable/euphrates": [518],
+
+    "huawei-pod2:stable/fraser": [544.5],
+
+    "arm-pod5:stable/euphrates": [869.5],
+
+    "huawei-virtual9:stable/euphrates": [1002],
+
+    "huawei-virtual4:stable/fraser": [1138],
+
+    "huawei-virtual4:stable/euphrates": [1174],
+
+    "huawei-virtual3:stable/euphrates": [1239],
+
+    "ericsson-pod1:stable/fraser": [1305],
+
+    "huawei-virtual2:stable/euphrates": [1430],
+
+    "huawei-virtual3:stable/fraser": [1433],
+
+    "huawei-virtual1:stable/fraser": [1470],
+
+    "huawei-virtual1:stable/euphrates": [1489],
+
+    "arm-pod6:stable/fraser": [1738.5],
+
+    "arm-pod6:stable/euphrates": [1883.5]
+
+}
+
+TC083
+-----
+
+TC083 measures network latency and throughput between VMs using netperf.
+The test results shown below are for UDP throughout.
+
+{
+
+    "lf-pod1:stable/euphrates": [2204.42],
+
+    "lf-pod2:stable/fraser": [1893.39],
+
+    "intel-pod18:stable/euphrates": [1835.55],
+
+    "lf-pod2:stable/euphrates": [1676.705],
+
+    "intel-pod5:stable/euphrates": [1612.555],
+
+    "zte-pod2:stable/fraser": [1543.995],
+
+    "lf-pod1:stable/fraser": [1480.86],
+
+    "intel-pod18:stable/fraser": [1417.015],
+
+    "flex-pod2:stable/euphrates": [1370.23],
+
+    "huawei-pod12:stable/euphrates": [1300.12]
+
+}