From 6aeb3f1bbbda53855f9b0c9148c7e60201502e05 Mon Sep 17 00:00:00 2001 From: Tibor Frank Date: Wed, 18 Sep 2019 10:40:17 +0200 Subject: Report: Add disclaimer for reconf tests + Improve description for reconf tests Change-Id: Ib1f397eda88a6c1db79b7a3a518584420770fd5c Signed-off-by: Tibor Frank (cherry picked from commit 2fc48e8d0a4712ec99dd2bb3a814ec9cb2bc12cf) --- docs/report/index.html.template | 1 + docs/report/introduction/methodology.rst | 1 + .../methodology_data_plane_throughput/index.rst | 1 - .../methodology_reconf.rst | 72 ---------- docs/report/introduction/methodology_reconf.rst | 78 +++++++++++ docs/report/vpp_performance_tests/index.rst | 1 + .../nf_service_density/index.rst | 1 - .../vnf_service_chains_reconf.rst | 138 ------------------- .../vpp_performance_tests/reconf_tests/index.rst | 15 ++ .../reconf_tests/vnf_service_chains_reconf.rst | 151 +++++++++++++++++++++ 10 files changed, 247 insertions(+), 212 deletions(-) delete mode 100644 docs/report/introduction/methodology_data_plane_throughput/methodology_reconf.rst create mode 100644 docs/report/introduction/methodology_reconf.rst delete mode 100644 docs/report/vpp_performance_tests/nf_service_density/vnf_service_chains_reconf.rst create mode 100644 docs/report/vpp_performance_tests/reconf_tests/index.rst create mode 100644 docs/report/vpp_performance_tests/reconf_tests/vnf_service_chains_reconf.rst diff --git a/docs/report/index.html.template b/docs/report/index.html.template index 554dd9bd76..8a01129982 100644 --- a/docs/report/index.html.template +++ b/docs/report/index.html.template @@ -21,6 +21,7 @@ CSIT-1908 vpp_performance_tests/throughput_speedup_multi_core/index vpp_performance_tests/packet_latency_graphs/index vpp_performance_tests/soak_tests/index + vpp_performance_tests/reconf_tests/index vpp_performance_tests/nf_service_density/index vpp_performance_tests/http_server_performance/index vpp_performance_tests/comparisons/index diff --git a/docs/report/introduction/methodology.rst b/docs/report/introduction/methodology.rst index b111fe51d3..af9bfc661a 100644 --- a/docs/report/introduction/methodology.rst +++ b/docs/report/introduction/methodology.rst @@ -16,6 +16,7 @@ Test Methodology methodology_plrsearch methodology_packet_latency methodology_multi_core_speedup + methodology_reconf methodology_vpp_startup_settings methodology_kvm_vms_vhost_user methodology_lxc_drc_container_memif diff --git a/docs/report/introduction/methodology_data_plane_throughput/index.rst b/docs/report/introduction/methodology_data_plane_throughput/index.rst index 71af4a9add..487d300b5b 100644 --- a/docs/report/introduction/methodology_data_plane_throughput/index.rst +++ b/docs/report/introduction/methodology_data_plane_throughput/index.rst @@ -7,4 +7,3 @@ Data Plane Throughput methodology_mlrsearch_tests methodology_mrr_throughput methodology_plrsearch - methodology_reconf diff --git a/docs/report/introduction/methodology_data_plane_throughput/methodology_reconf.rst b/docs/report/introduction/methodology_data_plane_throughput/methodology_reconf.rst deleted file mode 100644 index 8922b32b22..0000000000 --- a/docs/report/introduction/methodology_data_plane_throughput/methodology_reconf.rst +++ /dev/null @@ -1,72 +0,0 @@ -.. _reconf_tests: - -Reconf Tests -^^^^^^^^^^^^ - -Overview -~~~~~~~~ - -Reconf tests are designed to measure the impact of VPP re-configuration -on data plane traffic. -While VPP takes some measures against the traffic being -entirely stopped for a prolonged time, -the immediate forwarding rate varies during the re-configuration, -as some configurations steps need the active dataplane worker threads -to be stopped temporarily. - -As the usual methods of measuring throughput need multiple trial measurements -with somewhat long durations, and the re-configuration process can also be long, -finding an offered load which would result in zero loss -during the re-configuration process would be time-consuming. - -Instead, reconf tests find a througput value (lower bound for NDR) -without re-configuration, and then maintain that ofered load -during re-configuration. The measured loss count is then assumed to be caused -by the re-configuration process. The result published by reconf tests -is the effective blocked time, that is -the loss count divided by the offered load. - -Current Implementation -~~~~~~~~~~~~~~~~~~~~~~ - -Each reconf suite is based on a similar MLRsearch performance suite. - -MLRsearch parameters are changed to speed up the throughput discovery. -For example, PDR is not searched for, and final trial duration is shorter. - -The MLRsearch suite has to contain a configuration parameter -that can be scaled up, e.g. number of routes or number of service chains. -Currently, only increasing the scale is supported -as the re-configuration operation. In future, scale decrease -or other operations can be implemented. - -The traffic profile is not changed, so the traffic present is processed -only by the smaller scale configuration. The added routes / chains -are not targetted by the traffic. - -For the re-configuration, the same Robot Framework and Python libraries -are used, as were used in the initial configuration, with the exception -of the final calls that do not interact with VPP (e.g. starting -virtual machines) being skipped to reduce the test overall duration. - -Discussion -~~~~~~~~~~ - -Robot Framework introduces a certain overhead, which may affect timing -of individual VPP API calls, which in turn may affect -the number of packets lost. - -The exact calls executed may contain unnecessary info dumps, repeated commands, -or commands which change a value that do not need to be changed (e.g. MTU). -Thus, implementation details are affecting the results, even if their effect -on the corresponding MLRsearch suite is negligible. - -The lower bound for NDR is the only value safe to be used when zero packets lost -are expected without re-configuration. But different suites show different -"jitter" in that value. For some suites, the lower bound is not tight, -allowing full NIC buffers to drain quickly between worker pauses. -For other suites, lower bound for NDR still has quite a large probability -of non-zero packet loss even without re-configuration. - -But the results show very high effective blocked time, -so the two objections related to NDR lower bound are negligible in comparison. diff --git a/docs/report/introduction/methodology_reconf.rst b/docs/report/introduction/methodology_reconf.rst new file mode 100644 index 0000000000..32e0fd7561 --- /dev/null +++ b/docs/report/introduction/methodology_reconf.rst @@ -0,0 +1,78 @@ +.. _reconf_tests: + +Reconfiguration Tests +--------------------- + +.. important:: + + **DISCLAIMER**: Described reconf test methodology is experimental, and + subject to change following consultation within csit-dev, vpp-dev + and user communities. Current test results should be treated as indicative. + +Overview +~~~~~~~~ + +Reconf tests are designed to measure the impact of VPP re-configuration +on data plane traffic. +While VPP takes some measures against the traffic being +entirely stopped for a prolonged time, +the immediate forwarding rate varies during the re-configuration, +as some configurations steps need the active dataplane worker threads +to be stopped temporarily. + +As the usual methods of measuring throughput need multiple trial measurements +with somewhat long durations, and the re-configuration process can also be long, +finding an offered load which would result in zero loss +during the re-configuration process would be time-consuming. + +Instead, reconf tests find a througput value (lower bound for NDR) +without re-configuration, and then maintain that ofered load +during re-configuration. The measured loss count is then assumed to be caused +by the re-configuration process. The result published by reconf tests +is the effective blocked time, that is +the loss count divided by the offered load. + +Current Implementation +~~~~~~~~~~~~~~~~~~~~~~ + +Each reconf suite is based on a similar MLRsearch performance suite. + +MLRsearch parameters are changed to speed up the throughput discovery. +For example, PDR is not searched for, and final trial duration is shorter. + +The MLRsearch suite has to contain a configuration parameter +that can be scaled up, e.g. number of routes or number of service chains. +Currently, only increasing the scale is supported +as the re-configuration operation. In future, scale decrease +or other operations can be implemented. + +The traffic profile is not changed, so the traffic present is processed +only by the smaller scale configuration. The added routes / chains +are not targetted by the traffic. + +For the re-configuration, the same Robot Framework and Python libraries +are used, as were used in the initial configuration, with the exception +of the final calls that do not interact with VPP (e.g. starting +virtual machines) being skipped to reduce the test overall duration. + +Discussion +~~~~~~~~~~ + +Robot Framework introduces a certain overhead, which may affect timing +of individual VPP API calls, which in turn may affect +the number of packets lost. + +The exact calls executed may contain unnecessary info dumps, repeated commands, +or commands which change a value that do not need to be changed (e.g. MTU). +Thus, implementation details are affecting the results, even if their effect +on the corresponding MLRsearch suite is negligible. + +The lower bound for NDR is the only value safe to be used when zero packets lost +are expected without re-configuration. But different suites show different +"jitter" in that value. For some suites, the lower bound is not tight, +allowing full NIC buffers to drain quickly between worker pauses. +For other suites, lower bound for NDR still has quite a large probability +of non-zero packet loss even without re-configuration. + +But the results show very high effective blocked time, +so the two objections related to NDR lower bound are negligible in comparison. diff --git a/docs/report/vpp_performance_tests/index.rst b/docs/report/vpp_performance_tests/index.rst index c5c3e6679d..0f09377732 100644 --- a/docs/report/vpp_performance_tests/index.rst +++ b/docs/report/vpp_performance_tests/index.rst @@ -9,6 +9,7 @@ VPP Performance throughput_speedup_multi_core/index packet_latency_graphs/index soak_tests/index + reconf_tests/index nf_service_density/index http_server_performance/index comparisons/index diff --git a/docs/report/vpp_performance_tests/nf_service_density/index.rst b/docs/report/vpp_performance_tests/nf_service_density/index.rst index 8c827ebcc3..4c3143b36d 100644 --- a/docs/report/vpp_performance_tests/nf_service_density/index.rst +++ b/docs/report/vpp_performance_tests/nf_service_density/index.rst @@ -25,4 +25,3 @@ resources. vnf_service_chains cnf_service_chains cnf_service_pipelines - vnf_service_chains_reconf diff --git a/docs/report/vpp_performance_tests/nf_service_density/vnf_service_chains_reconf.rst b/docs/report/vpp_performance_tests/nf_service_density/vnf_service_chains_reconf.rst deleted file mode 100644 index 66b1af15d3..0000000000 --- a/docs/report/vpp_performance_tests/nf_service_density/vnf_service_chains_reconf.rst +++ /dev/null @@ -1,138 +0,0 @@ - -.. raw:: latex - - \clearpage - -.. raw:: html - - - -.. _vnf_service_chains_reconf: - -Reconfiguration of VNF Service Chains -===================================== - -See :ref:`reconf_tests` for methodology description of this test type. - -In each test, a single service chain is added, the re-configuration -contains all the steps the initial chains got, except the last step -(starting VMs) is skipped. - -Additional information about graph data: - -#. **Graph Title**: describes tested packet path including VNF workload - running in each VM. - -#. **X-axis Labels**: indices of individual test suites as listed in - Graph Legend. - -#. **Y-axis Labels**: measured Implied time loss [s] values. - -#. **Graph Legend**: lists X-axis indices with associated CSIT test - suites executed to generate graphed test results and the average value and - standard deviation of measured packet loss in millions of packets. - -#. **Hover Information**: lists minimum, first quartile, median, - third quartile, and maximum. If either type of outlier is present the - whisker on the appropriate side is taken to 1.5×IQR from the quartile - (the "inner fence") rather than the max or min, and individual outlying - data points are displayed as unfilled circles (for suspected outliers) - or filled circles (for outliers). (The "outer fence" is 3×IQR from the - quartile.) - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 2n-skx`_ with RF - result files csit-vpp-perf-|srelease|-\*.zip - `archived here <../../_static/archive/>`_. - -.. raw:: latex - - \clearpage - -2n-skx-xxv710 -~~~~~~~~~~~~~ - -imix-2t1c-dot1qip4vxlan-l2bd ----------------------------- - -.. raw:: html - -
- -
- -.. raw:: latex - - \begin{figure}[H] - \centering - \graphicspath{{../_build/_static/vpp/}} - \includegraphics[clip, trim=0cm 0cm 5cm 0cm, width=0.70\textwidth]{2n-skx-xxv710-imix-2t1c-dot1qip4vxlan-l2bd-reconf} - \label{fig:2n-skx-xxv710-imix-2t1c-dot1qip4vxlan-l2bd-reconf} - \end{figure} - -.. raw:: latex - - \clearpage - -imix-4t2c-dot1qip4vxlan-l2bd ----------------------------- - -.. raw:: html - -
- -
- -.. raw:: latex - - \begin{figure}[H] - \centering - \graphicspath{{../_build/_static/vpp/}} - \includegraphics[clip, trim=0cm 0cm 5cm 0cm, width=0.70\textwidth]{2n-skx-xxv710-imix-4t2c-dot1qip4vxlan-l2bd-reconf} - \label{fig:2n-skx-xxv710-imix-4t2c-dot1qip4vxlan-l2bd-reconf} - \end{figure} - -.. raw:: latex - - \clearpage - -imix-8t4c-dot1qip4vxlan-l2bd ----------------------------- - -.. raw:: html - -
- -
- -.. raw:: latex - - \begin{figure}[H] - \centering - \graphicspath{{../_build/_static/vpp/}} - \includegraphics[clip, trim=0cm 0cm 5cm 0cm, width=0.70\textwidth]{2n-skx-xxv710-imix-8t4c-dot1qip4vxlan-l2bd-reconf} - \label{fig:2n-skx-xxv710-imix-8t4c-dot1qip4vxlan-l2bd-reconf} - \end{figure} diff --git a/docs/report/vpp_performance_tests/reconf_tests/index.rst b/docs/report/vpp_performance_tests/reconf_tests/index.rst new file mode 100644 index 0000000000..c5da438b29 --- /dev/null +++ b/docs/report/vpp_performance_tests/reconf_tests/index.rst @@ -0,0 +1,15 @@ + +.. raw:: latex + + \clearpage + +.. _reconfiguration_tests: + +Reconfiguration Tests +===================== + +See :ref:`reconf_tests` for methodology description of this test type. + +.. toctree:: + + vnf_service_chains_reconf diff --git a/docs/report/vpp_performance_tests/reconf_tests/vnf_service_chains_reconf.rst b/docs/report/vpp_performance_tests/reconf_tests/vnf_service_chains_reconf.rst new file mode 100644 index 0000000000..47934651bc --- /dev/null +++ b/docs/report/vpp_performance_tests/reconf_tests/vnf_service_chains_reconf.rst @@ -0,0 +1,151 @@ + +.. raw:: latex + + \clearpage + +.. raw:: html + + + +.. _vnf_service_chains_reconf: + +VNF Service Chains +================== + +.. important:: + + **DISCLAIMER**: Presented test results should be taken with a pinch of + salt as they may not represent correctly the system behaviour due + to experimental nature of used test methodology (see :ref:`reconf_tests`). + +In each test, a single service chain is added, the re-configuration +contains all the steps the initial chains got, except the last step +(starting VMs) is skipped. + +Additional information about graph data: + +#. **Graph Title**: describes tested VPP packet path. Format: + + - wire encapsulation ``dot1qip4vlxan``, + - VPP forwarding mode ``l2bd``, + - total number {Y} of service chains ``{Y}ch``, + - total number of chains being reconfigured ``1ach``, + - total number of vhost-user interfaces forwarding packets on VPP with {Y} + chains and {X} VMs per chain ``{2XY}vh`` (2 interfaces per {X} VMs per {Y} + chains), + total number {XY} of VNF VMs forwarding packets ``{XY}vm`` and finally + - VNF workload in VM ``testpmd``. + +#. **X-axis Labels**: indices of individual test suites as listed in + Graph Legend. + +#. **Y-axis Labels**: measured Implied time loss [s] values. + +#. **Graph Legend**: lists X-axis indices with associated CSIT test + suites executed to generate graphed test results and the average value and + standard deviation of measured packet loss in millions of packets. + +#. **Hover Information**: lists minimum, first quartile, median, + third quartile, and maximum. If either type of outlier is present the + whisker on the appropriate side is taken to 1.5×IQR from the quartile + (the "inner fence") rather than the max or min, and individual outlying + data points are displayed as unfilled circles (for suspected outliers) + or filled circles (for outliers). (The "outer fence" is 3×IQR from the + quartile.) + +.. note:: + + Test results have been generated by + `FD.io test executor vpp performance job 2n-skx`_ with RF + result files csit-vpp-perf-|srelease|-\*.zip + `archived here <../../_static/archive/>`_. + +.. raw:: latex + + \clearpage + +2n-skx-xxv710 +~~~~~~~~~~~~~ + +imix-2t1c-dot1qip4vxlan-l2bd +---------------------------- + +.. raw:: html + +
+ +
+ +.. raw:: latex + + \begin{figure}[H] + \centering + \graphicspath{{../_build/_static/vpp/}} + \includegraphics[clip, trim=0cm 0cm 5cm 0cm, width=0.70\textwidth]{2n-skx-xxv710-imix-2t1c-dot1qip4vxlan-l2bd-reconf} + \label{fig:2n-skx-xxv710-imix-2t1c-dot1qip4vxlan-l2bd-reconf} + \end{figure} + +.. raw:: latex + + \clearpage + +imix-4t2c-dot1qip4vxlan-l2bd +---------------------------- + +.. raw:: html + +
+ +
+ +.. raw:: latex + + \begin{figure}[H] + \centering + \graphicspath{{../_build/_static/vpp/}} + \includegraphics[clip, trim=0cm 0cm 5cm 0cm, width=0.70\textwidth]{2n-skx-xxv710-imix-4t2c-dot1qip4vxlan-l2bd-reconf} + \label{fig:2n-skx-xxv710-imix-4t2c-dot1qip4vxlan-l2bd-reconf} + \end{figure} + +.. raw:: latex + + \clearpage + +imix-8t4c-dot1qip4vxlan-l2bd +---------------------------- + +.. raw:: html + +
+ +
+ +.. raw:: latex + + \begin{figure}[H] + \centering + \graphicspath{{../_build/_static/vpp/}} + \includegraphics[clip, trim=0cm 0cm 5cm 0cm, width=0.70\textwidth]{2n-skx-xxv710-imix-8t4c-dot1qip4vxlan-l2bd-reconf} + \label{fig:2n-skx-xxv710-imix-8t4c-dot1qip4vxlan-l2bd-reconf} + \end{figure} -- cgit 1.2.3-korg