CSIT-901: Throughput Speedup Analysis - Static content

 - Add robustnes to the function which generates the plots
 - Add "Model for TSA" to LLD.

Change-Id: I78dffe3622b4566b30ecac411f8ff285689d1427
Signed-off-by: Tibor Frank <tifrank@cisco.com>
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
(cherry picked from commit b4b7fac76d29cc780b3ccd865379ad2ed38c0586)

5 files changed, 394 insertions, 0 deletions

diff --git a/docs/report/vpp_performance_tests/index.rst b/docs/report/vpp_performance_tests/index.rst
index 4c11957f34..2d234ce921 100644
--- a/docs/report/vpp_performance_tests/index.rst
+++ b/docs/report/vpp_performance_tests/index.rst
@@ -7,5 +7,6 @@ VPP Performance Tests
     csit_release_notes
     packet_throughput_graphs/index
     packet_latency_graphs/index
+    throughput_speedup_multi_core
     test_environment
     documentation/index
diff --git a/docs/report/vpp_performance_tests/throughput_speedup_multi_core/index.rst b/docs/report/vpp_performance_tests/throughput_speedup_multi_core/index.rst
new file mode 100644
index 0000000000..ddce548b6c
--- /dev/null
+++ b/docs/report/vpp_performance_tests/throughput_speedup_multi_core/index.rst
@@ -0,0 +1,20 @@
+Throughput Speedup Multi-Core
+=============================
+
+Throughput Speedup Multi-Core graphs are generated based on results from
+multiple executions of the same CSIT performance tests across the three
+physical testbeds within LF FD.io labs. Plotted grouped bars illustrate
+the 64B/78B packet throughput speedup ratio for 2- and 4-core multi-
+threaded VPP configurations relative to 1-core configurations.
+
+.. note::
+
+    Test results have been generated by
+    `FD.io test executor vpp performance jobs`_ with Robot Framework result
+    files csit-vpp-perf-\*.zip `archived here <../../_static/archive/>`_.
+
+.. toctree::
+
+    l2
+    ip4
+    ip6
diff --git a/docs/report/vpp_performance_tests/throughput_speedup_multi_core/ip4.rst b/docs/report/vpp_performance_tests/throughput_speedup_multi_core/ip4.rst
new file mode 100644
index 0000000000..af9bd255be
--- /dev/null
+++ b/docs/report/vpp_performance_tests/throughput_speedup_multi_core/ip4.rst
@@ -0,0 +1,125 @@
+IPv4 Routed-Forwarding
+======================
+
+Following sections include Throughput Speedup Analysis for VPP multi-
+core multi-thread configurations with no Hyper-Threading, specifically
+for tested 2t2c (2threads, 2cores) and 4t4c scenarios. 1t1c throughput
+results are used as a reference for reported speedup ratio. Input data
+used for the graphs comes from Phy-to-Phy 64B performance tests with VPP
+IPv4 Routed-Forwarding, including NDR throughput (zero packet loss) and
+PDR throughput (<0.5% packet loss).
+
+NDR Throughput
+--------------
+
+VPP NDR 64B packet throughput speedup ratio is presented in the graphs
+below for 10ge2p1x520 and 40ge2p1xl710 network interface cards.
+
+NIC 10ge2p1x520
+~~~~~~~~~~~~~~~
+
+.. raw:: html
+
+    <iframe width="700" height="1000" frameborder="0" scrolling="no" src="../../_static/vpp/10ge2p1x520-64B-ip4-tsa-ndrdisc.html"></iframe>
+
+.. raw:: latex
+
+    \begin{figure}[H]
+        \centering
+            \graphicspath{{../_build/_static/vpp/}}
+            \includegraphics[clip, trim=0cm 8cm 5cm 0cm, width=0.70\textwidth]{10ge2p1x520-64B-ip4-tsa-ndrdisc}
+            \label{fig:10ge2p1x520-64B-ip4-tsa-ndrdisc}
+    \end{figure}
+
+CSIT source code for the test cases used for above plots can be found in CSIT
+git repository:
+
+.. only:: html
+
+   .. program-output:: cd ../../../../../ && set +x && cd tests/vpp/perf/ip4 && grep -E '64B-(1t1c|2t2c|4t4c)-ethip4-ip4(base|scale[a-z0-9]*)*-ndrdisc' 10ge2p1x520*
+      :shell:
+
+.. only:: latex
+
+   .. code-block:: bash
+
+      $ cd tests/vpp/perf/ip4
+      $ grep -E '64B-(1t1c|2t2c|4t4c)-ethip4-ip4(base|scale[a-z0-9]*)*-ndrdisc' 10ge2p1x520*
+
+*Figure 1. Throughput Speedup Analysis - Multi-Core Speedup Ratio - Normalized
+NDR Throughput for Phy-to-Phy IPv4 Routed-Forwarding.*
+
+NIC 40ge2p1xl710
+~~~~~~~~~~~~~~~~
+
+.. raw:: html
+
+    <iframe width="700" height="1000" frameborder="0" scrolling="no" src="../../_static/vpp/40ge2p1xl710-64B-ip4-tsa-ndrdisc.html"></iframe>
+
+.. raw:: latex
+
+    \begin{figure}[H]
+        \centering
+            \graphicspath{{../_build/_static/vpp/}}
+            \includegraphics[clip, trim=0cm 8cm 5cm 0cm, width=0.70\textwidth]{40ge2p1xl710-64B-ip4-tsa-ndrdisc}
+            \label{fig:40ge2p1xl710-64B-ip4-tsa-ndrdisc}
+    \end{figure}
+
+CSIT source code for the test cases used for above plots can be found in CSIT
+git repository:
+
+.. only:: html
+
+   .. program-output:: cd ../../../../../ && set +x && cd tests/vpp/perf/ip4 && grep -P '64B-(1t1c|2t2c|4t4c)-ethip4-ip4(base|scale[a-z0-9]*)*-ndrdisc' 40ge2p1xl710*
+      :shell:
+
+.. only:: latex
+
+   .. code-block:: bash
+
+      $ cd tests/vpp/perf/ip4
+      $ grep -P '64B-(1t1c|2t2c|4t4c)-ethip4-ip4(base|scale[a-z0-9]*)*-ndrdisc' 40ge2p1xl710*
+
+*Figure 2. Throughput Speedup Analysis - Multi-Core Speedup Ratio - Normalized
+NDR Throughput for Phy-to-Phy IPv4 Routed-Forwarding.*
+
+PDR Throughput
+--------------
+
+VPP PDR 64B packet throughput speedup ratio is presented in the graphs
+below for 10ge2p1x520 and 40ge2p1xl710 network interface cards. PDR
+measured for 0.5% packet loss ratio.
+
+NIC 10ge2p1x520
+~~~~~~~~~~~~~~~
+
+.. raw:: html
+
+    <iframe width="700" height="1000" frameborder="0" scrolling="no" src="../../_static/vpp/10ge2p1x520-64B-ip4-tsa-pdrdisc.html"></iframe>
+
+.. raw:: latex
+
+    \begin{figure}[H]
+        \centering
+            \graphicspath{{../_build/_static/vpp/}}
+            \includegraphics[clip, trim=0cm 8cm 5cm 0cm, width=0.70\textwidth]{10ge2p1x520-64B-ip4-tsa-pdrdisc}
+            \label{fig:10ge2p1x520-64B-ip4-tsa-pdrdisc}
+    \end{figure}
+
+CSIT source code for the test cases used for above plots can be found in CSIT
+git repository:
+
+.. only:: html
+
+   .. program-output:: cd ../../../../../ && set +x && cd tests/vpp/perf/ip4 && grep -E '64B-(1t1c|2t2c|4t4c)-ethip4-ip4(base|scale[a-z0-9]*)*-ndrdisc' 10ge2p1x520*
+      :shell:
+
+.. only:: latex
+
+   .. code-block:: bash
+
+      $ cd tests/vpp/perf/ip4
+      $ grep -E '64B-(1t1c|2t2c|4t4c)-ethip4-ip4(base|scale[a-z0-9]*)*-ndrdisc' 10ge2p1x520*
+
+*Figure 3. Throughput Speedup Analysis - Multi-Core Speedup Ratio - Normalized
+PDR Throughput for Phy-to-Phy IPv4 Routed-Forwarding.*
diff --git a/docs/report/vpp_performance_tests/throughput_speedup_multi_core/ip6.rst b/docs/report/vpp_performance_tests/throughput_speedup_multi_core/ip6.rst
new file mode 100644
index 0000000000..077b330652
--- /dev/null
+++ b/docs/report/vpp_performance_tests/throughput_speedup_multi_core/ip6.rst
@@ -0,0 +1,126 @@
+IPv6 Routed-Forwarding
+======================
+
+Following sections include Throughput Speedup Analysis for VPP multi-
+core multi-thread configurations with no Hyper-Threading, specifically
+for tested 2t2c (2threads, 2cores) and 4t4c scenarios. 1t1c throughput
+results are used as a reference for reported speedup ratio. Input data
+used for the graphs comes from Phy-to-Phy 78B performance tests with VPP
+IPv6 Routed-Forwarding, including NDR throughput (zero packet loss) and
+PDR throughput (<0.5% packet loss).
+
+NDR Throughput
+--------------
+
+VPP NDR 78B packet throughput speedup ratio is presented in the graphs
+below for 10ge2p1x520 and 40ge2p1xl710 network interface cards.
+
+
+NIC 10ge2p1x520
+~~~~~~~~~~~~~~~
+
+.. raw:: html
+
+    <iframe width="700" height="1000" frameborder="0" scrolling="no" src="../../_static/vpp/10ge2p1x520-78B-ip6-tsa-ndrdisc.html"></iframe>
+
+.. raw:: latex
+
+    \begin{figure}[H]
+        \centering
+            \graphicspath{{../_build/_static/vpp/}}
+            \includegraphics[clip, trim=0cm 8cm 5cm 0cm, width=0.70\textwidth]{10ge2p1x520-78B-ip6-tsa-ndrdisc}
+            \label{fig:10ge2p1x520-78B-ip6-tsa-ndrdisc}
+    \end{figure}
+
+CSIT source code for the test cases used for above plots can be found in CSIT
+git repository:
+
+.. only:: html
+
+   .. program-output:: cd ../../../../../ && set +x && cd tests/vpp/perf/ip6 && grep -E '78B-(1t1c|2t2c|4t4c)-ethip6-ip6(base|scale[a-z0-9]*)*-ndrdisc' 10ge2p1x520*
+      :shell:
+
+.. only:: latex
+
+   .. code-block:: bash
+
+      $ cd tests/vpp/perf/ip6
+      $ grep -E '78B-(1t1c|2t2c|4t4c)-ethip6-ip6(base|scale[a-z0-9]*)*-ndrdisc' 10ge2p1x520*
+
+*Figure 1. Throughput Speedup Analysis - Multi-Core Speedup Ratio - Normalized
+NDR Throughput for Phy-to-Phy IPv6 Routed-Forwarding.*
+
+NIC 40ge2p1xl710
+~~~~~~~~~~~~~~~~
+
+.. raw:: html
+
+    <iframe width="700" height="1000" frameborder="0" scrolling="no" src="../../_static/vpp/40ge2p1xl710-78B-ip6-tsa-ndrdisc.html"></iframe>
+
+.. raw:: latex
+
+    \begin{figure}[H]
+        \centering
+            \graphicspath{{../_build/_static/vpp/}}
+            \includegraphics[clip, trim=0cm 8cm 5cm 0cm, width=0.70\textwidth]{40ge2p1xl710-78B-ip6-tsa-ndrdisc}
+            \label{fig:40ge2p1xl710-78B-ip6-tsa-ndrdisc}
+    \end{figure}
+
+CSIT source code for the test cases used for above plots can be found in CSIT
+git repository:
+
+.. only:: html
+
+   .. program-output:: cd ../../../../../ && set +x && cd tests/vpp/perf/ip6 && grep -E '78B-(1t1c|2t2c|4t4c)-ethip6-ip6(base|scale[a-z0-9]*)*-ndrdisc' 40ge2p1xl710*
+      :shell:
+
+.. only:: latex
+
+   .. code-block:: bash
+
+      $ cd tests/vpp/perf/ip6
+      $ grep -E '78B-(1t1c|2t2c|4t4c)-ethip6-ip6(base|scale[a-z0-9]*)*-ndrdisc' 40ge2p1xl710*
+
+*Figure 2. Throughput Speedup Analysis - Multi-Core Speedup Ratio - Normalized
+NDR Throughput for Phy-to-Phy IPv6 Routed-Forwarding.*
+
+PDR Throughput
+--------------
+
+VPP PDR 78B packet throughput speedup ratio is presented in the graphs
+below for 10ge2p1x520 and 40ge2p1xl710 network interface cards. PDR
+measured for 0.5% packet loss ratio.
+
+NIC 10ge2p1x520
+~~~~~~~~~~~~~~~
+
+.. raw:: html
+
+    <iframe width="700" height="1000" frameborder="0" scrolling="no" src="../../_static/vpp/10ge2p1x520-78B-ip6-tsa-pdrdisc.html"></iframe>
+
+.. raw:: latex
+
+    \begin{figure}[H]
+        \centering
+            \graphicspath{{../_build/_static/vpp/}}
+            \includegraphics[clip, trim=0cm 8cm 5cm 0cm, width=0.70\textwidth]{10ge2p1x520-78B-ip6-tsa-pdrdisc}
+            \label{fig:10ge2p1x520-78B-ip6-tsa-pdrdisc}
+    \end{figure}
+
+CSIT source code for the test cases used for above plots can be found in CSIT
+git repository:
+
+.. only:: html
+
+   .. program-output:: cd ../../../../../ && set +x && cd tests/vpp/perf/ip6 && grep -E '78B-(1t1c|2t2c|4t4c)-ethip6-ip6(base|scale[a-z0-9]*)*-pdrdisc' 10ge2p1x520*
+      :shell:
+
+.. only:: latex
+
+   .. code-block:: bash
+
+      $ cd tests/vpp/perf/ip6
+      $ grep -E '78B-(1t1c|2t2c|4t4c)-ethip6-ip6(base|scale[a-z0-9]*)*-pdrdisc' 10ge2p1x520*
+
+*Figure 3. Throughput Speedup Analysis - Multi-Core Speedup Ratio - Normalized
+PDR Throughput for Phy-to-Phy IPv6 Routed-Forwarding.*
diff --git a/docs/report/vpp_performance_tests/throughput_speedup_multi_core/l2.rst b/docs/report/vpp_performance_tests/throughput_speedup_multi_core/l2.rst
new file mode 100644
index 0000000000..2df3fb6229
--- /dev/null
+++ b/docs/report/vpp_performance_tests/throughput_speedup_multi_core/l2.rst
@@ -0,0 +1,122 @@
+L2 Ethernet Switching
+=====================
+
+Following sections include Throughput Speedup Analysis for VPP multi-
+core multi-thread configurations with no Hyper-Threading, specifically
+for tested 2t2c (2threads, 2cores) and 4t4c scenarios. 1t1c throughput
+results are used as a reference for reported speedup ratio. Input data
+used for the graphs comes from Phy-to-Phy 64B performance tests with VPP
+L2 Ethernet switching, including NDR throughput (zero packet loss) and
+PDR throughput (<0.5% packet loss).
+
+NDR Throughput
+--------------
+
+VPP NDR 64B packet throughput speedup ratio is presented in the graphs
+below for 10ge2p1x520 and 40ge2p1xl710 network interface cards.
+
+NIC 10ge2p1x520
+~~~~~~~~~~~~~~~
+
+.. raw:: html
+
+    <iframe width="700" height="1000" frameborder="0" scrolling="no" src="../../_static/vpp/10ge2p1x520-64B-l2-tsa-ndrdisc.html"></iframe>
+
+.. raw:: latex
+
+    \begin{figure}[H]
+        \centering
+            \graphicspath{{../_build/_static/vpp/}}
+            \includegraphics[clip, trim=0cm 8cm 5cm 0cm, width=0.70\textwidth]{10ge2p1x520-64B-l2-tsa-ndrdisc}
+            \label{fig:10ge2p1x520-64B-l2-tsa-ndrdisc}
+    \end{figure}
+
+CSIT source code for the test cases used for above plots can be found in CSIT
+git repository:
+
+.. only:: html
+
+   .. program-output:: cd ../../../../../ && set +x && cd tests/vpp/perf/l2 && grep -E "64B-(1t1c|2t2c|4t4c)-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn|l2bdscale.*|l2dbscale.*)-(eth.*)*ndrdisc" 10ge2p1x520*
+      :shell:
+
+.. only:: latex
+
+   .. code-block:: bash
+
+      $ grep -E "64B-(1t1c|2t2c|4t4c)-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn|l2bdscale.*|l2dbscale.*)-(eth.*)*ndrdisc" tests/vpp/perf/l2/10ge2p1x520*
+
+*Figure 1. Throughput Speedup Analysis - Multi-Core Speedup Ratio - Normalized
+NDR Throughput for Phy-to-Phy L2 Ethernet Switching.*
+
+NIC 40ge2p1xl710
+~~~~~~~~~~~~~~~~
+
+.. raw:: html
+
+    <iframe width="700" height="1000" frameborder="0" scrolling="no" src="../../_static/vpp/40ge2p1xl710-64B-l2-tsa-ndrdisc.html"></iframe>
+
+.. raw:: latex
+
+    \begin{figure}[H]
+        \centering
+            \graphicspath{{../_build/_static/vpp/}}
+            \includegraphics[clip, trim=0cm 8cm 5cm 0cm, width=0.70\textwidth]{40ge2p1xl710-64B-l2-tsa-ndrdisc}
+            \label{fig:40ge2p1xl710-64B-l2-tsa-ndrdisc}
+    \end{figure}
+
+CSIT source code for the test cases used for above plots can be found in CSIT
+git repository:
+
+.. only:: html
+
+   .. program-output:: cd ../../../../../ && set +x && cd tests/vpp/perf/l2 && grep -E "64B-(1t1c|2t2c|4t4c)-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn|l2bdscale.*|l2dbscale.*)-(eth.*)*ndrdisc" 40ge2p1xl710*
+      :shell:
+
+.. only:: latex
+
+   .. code-block:: bash
+
+      $ grep -E "64B-(1t1c|2t2c|4t4c)-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn|l2bdscale.*|l2dbscale.*)-(eth.*)*ndrdisc" tests/vpp/perf/l2/40ge2p1xl710*
+
+*Figure 2. Throughput Speedup Analysis - Multi-Core Speedup Ratio - Normalized
+NDR Throughput for Phy-to-Phy L2 Ethernet Switching.*
+
+PDR Throughput
+--------------
+
+VPP PDR 64B packet throughput speedup ratio is presented in the graphs
+below for 10ge2p1x520 and 40ge2p1xl710 network interface cards. PDR
+measured for 0.5% packet loss ratio.
+
+NIC 10ge2p1x520
+~~~~~~~~~~~~~~~
+
+.. raw:: html
+
+    <iframe width="700" height="1000" frameborder="0" scrolling="no" src="../../_static/vpp/10ge2p1x520-64B-l2-tsa-pdrdisc.html"></iframe>
+
+.. raw:: latex
+
+    \begin{figure}[H]
+        \centering
+            \graphicspath{{../_build/_static/vpp/}}
+            \includegraphics[clip, trim=0cm 8cm 5cm 0cm, width=0.70\textwidth]{10ge2p1x520-64B-l2-tsa-pdrdisc}
+            \label{fig:10ge2p1x520-64B-l2-tsa-pdrdisc}
+    \end{figure}
+
+CSIT source code for the test cases used for above plots can be found in CSIT
+git repository:
+
+.. only:: html
+
+   .. program-output:: cd ../../../../../ && set +x && cd tests/vpp/perf/l2 && grep -E "64B-(1t1c|2t2c|4t4c)-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn|l2bdscale.*|l2dbscale.*)-(eth.*)*pdrdisc" 10ge2p1x520*
+      :shell:
+
+.. only:: latex
+
+   .. code-block:: bash
+
+      $ grep -E "64B-(1t1c|2t2c|4t4c)-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn|l2bdscale.*|l2dbscale.*)-(eth.*)*pdrdisc" tests/vpp/perf/l2/10ge2p1x520*
+
+*Figure 3. Throughput Speedup Analysis - Multi-Core Speedup Ratio - Normalized
+PDR Throughput for Phy-to-Phy L2 Ethernet Switching.*