diff options
| -rw-r--r-- | docs/report/dpdk_performance_tests/csit_release_notes.rst | 96 | ||||
| -rw-r--r-- | docs/report/vpp_performance_tests/csit_release_notes.rst | 114 |
2 files changed, 110 insertions, 100 deletions
diff --git a/docs/report/dpdk_performance_tests/csit_release_notes.rst b/docs/report/dpdk_performance_tests/csit_release_notes.rst index bd08c57bba..d84b4e259a 100644 --- a/docs/report/dpdk_performance_tests/csit_release_notes.rst +++ b/docs/report/dpdk_performance_tests/csit_release_notes.rst @@ -37,129 +37,133 @@ Changes in |csit-release| Performance Changes ------------------- -Relative performance changes in measured packet throughput in |csit-release| -are calculated against the results from |csit-release-1| -report. Listed mean and standard deviation values are computed based on -a series of the same tests executed against respective DPDK releases to -verify test results repeatability, with percentage change calculated for -mean values. +Relative performance changes in measured NDR, PDR and MRR packet +throughput in |csit-release| are calculated against the test results +from |csit-release-1| report, for tests running on 3-Node Intel Xeon +Haswell testbeds (3n-hsw) in 1-core, 2-core and 4-core (MRR only) +configurations. + +Listed mean and standard deviation values are computed based on a series +of the same tests executed against respective VPP releases to verify +test results repeatability, with percentage change calculated for mean +values. NDR Changes ~~~~~~~~~~~ -NDR throughput changes between releases are available in a CSV and pretty ASCII -formats: +NDR throughput changes between releases are available in CSV and pretty +ASCII formats: - - `csv format for 1t1c <../_static/dpdk/performance-changes-1t1c-ndr.csv>`_, - - `csv format for 2t2c <../_static/dpdk/performance-changes-2t2c-ndr.csv>`_, - - `pretty ASCII format for 1t1c <../_static/dpdk/performance-changes-1t1c-ndr.txt>`_, - - `pretty ASCII format for 2t2c <../_static/dpdk/performance-changes-2t2c-ndr.txt>`_. + - `CSV 1t1c NDR changes <../_static/dpdk/performance-changes-1t1c-ndr.csv>`_, + - `CSV 2t2c NDR changes <../_static/dpdk/performance-changes-2t2c-ndr.csv>`_, + - `ASCII 1t1c NDR changes <../_static/dpdk/performance-changes-1t1c-ndr.txt>`_, + - `ASCII 2t2c NDR changes <../_static/dpdk/performance-changes-2t2c-ndr.txt>`_. .. note:: Test results have been generated by `FD.io test executor dpdk performance job 3n-hsw`_ - with Robot Framework result + with RF result files csit-dpdk-perf-|srelease|-\*.zip `archived here <../_static/archive/>`_. PDR Changes ~~~~~~~~~~~ -PDR throughput changes between releases are available in a CSV and pretty ASCII -formats: +PDR throughput changes between releases are available in CSV and pretty +ASCII formats: - - `csv format for 1t1c <../_static/dpdk/performance-changes-1t1c-pdr.csv>`_, - - `csv format for 2t2c <../_static/dpdk/performance-changes-2t2c-pdr.csv>`_, - - `pretty ASCII format for 1t1c <../_static/dpdk/performance-changes-1t1c-pdr.txt>`_, - - `pretty ASCII format for 2t2c <../_static/dpdk/performance-changes-2t2c-pdr.txt>`_. + - `CSV 1t1c PDR changes <../_static/dpdk/performance-changes-1t1c-pdr.csv>`_, + - `CSV 2t2c PDR changes <../_static/dpdk/performance-changes-2t2c-pdr.csv>`_, + - `ASCII 1t1c PDR changes <../_static/dpdk/performance-changes-1t1c-pdr.txt>`_, + - `ASCII 2t2c PDR changes <../_static/dpdk/performance-changes-2t2c-pdr.txt>`_. .. note:: Test results have been generated by `FD.io test executor dpdk performance job 3n-hsw`_ - with Robot Framework result + with RF result files csit-dpdk-perf-|srelease|-\*.zip `archived here <../_static/archive/>`_. MRR Changes ~~~~~~~~~~~ -MRR small packet throughput changes between releases are available in a -CSV and pretty ASCII formats: +MRR throughput changes between releases are available in CSV and pretty +ASCII formats: - - `csv format for 1t1c <../_static/dpdk/performance-changes-1t1c-mrr.csv>`_, - - `csv format for 2t2c <../_static/dpdk/performance-changes-2t2c-mrr.csv>`_, - - `csv format for 4t4c <../_static/dpdk/performance-changes-4t4c-mrr.csv>`_, - - `pretty ASCII format for 1t1c <../_static/dpdk/performance-changes-1t1c-mrr.txt>`_, - - `pretty ASCII format for 2t2c <../_static/dpdk/performance-changes-2t2c-mrr.txt>`_, - - `pretty ASCII format for 4t4c <../_static/dpdk/performance-changes-4t4c-mrr.txt>`_. + - `CSV 1t1c MRR changes <../_static/dpdk/performance-changes-1t1c-mrr.csv>`_, + - `CSV 2t2c MRR changes <../_static/dpdk/performance-changes-2t2c-mrr.csv>`_, + - `CSV 4t4c MRR changes <../_static/dpdk/performance-changes-4t4c-mrr.csv>`_, + - `ASCII 1t1c MRR changes <../_static/dpdk/performance-changes-1t1c-mrr.txt>`_, + - `ASCII 2t2c MRR changes <../_static/dpdk/performance-changes-2t2c-mrr.txt>`_, + - `ASCII 4t4c MRR changes <../_static/dpdk/performance-changes-4t4c-mrr.txt>`_. .. note:: Test results have been generated by `FD.io test executor dpdk performance job 3n-hsw`_, - with Robot Framework result + with RF result files csit-vpp-perf-|srelease|-\*.zip `archived here <../_static/archive/>`_. -Comparison Across Testbeds --------------------------- +Skx vs. Hsw Comparison +---------------------- Relative performance changes in measured packet throughput on 3-Node Skx testbed are calculated against the results measured on 3-Node Hsw testbed. -NDR Changes -~~~~~~~~~~~ +NDR Comparison +~~~~~~~~~~~~~~ NDR throughput changes between testbeds are available in a CSV and pretty ASCII formats: - - `csv format for ndr <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_, - - `pretty ASCII format for ndr <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_. + - `CSV 1c NDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_, + - `ASCII 1c NDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_. .. note:: Test results have been generated by `FD.io test executor dpdk performance job 3n-hsw`_ and `FD.io test executor dpdk performance job 3n-skx`_ - with Robot Framework result + with RF result files csit-dpdk-perf-|srelease|-\*.zip `archived here <../_static/archive/>`_. -PDR Changes -~~~~~~~~~~~ +PDR Comparison +~~~~~~~~~~~~~~ PDR throughput changes between testbeds are available in a CSV and pretty ASCII formats: - - `csv format for pdr <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_, - - `pretty ASCII format for pdr <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_. + - `CSV 1c PDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_, + - `ASCII 1c PDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_. .. note:: Test results have been generated by `FD.io test executor dpdk performance job 3n-hsw`_ and `FD.io test executor dpdk performance job 3n-skx`_ - with Robot Framework result + with RF result files csit-dpdk-perf-|srelease|-\*.zip `archived here <../_static/archive/>`_. -MRR Changes -~~~~~~~~~~~ +MRR Comparison +~~~~~~~~~~~~~~ MRR throughput changes between testbeds are available in a CSV and pretty ASCII formats: - - `csv format for mrr <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_, - - `pretty ASCII format for mrr <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_. + - `CSV 1c MRR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_, + - `ASCII 1c MRR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_. .. note:: Test results have been generated by `FD.io test executor dpdk performance job 3n-hsw`_ and `FD.io test executor dpdk performance job 3n-skx`_ - with Robot Framework result + with RF result files csit-vpp-perf-|srelease|-\*.zip `archived here <../_static/archive/>`_. diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst index 580ef4812a..03fca8f9a0 100644 --- a/docs/report/vpp_performance_tests/csit_release_notes.rst +++ b/docs/report/vpp_performance_tests/csit_release_notes.rst @@ -57,12 +57,16 @@ Changes in |csit-release| Performance Changes ------------------- -Relative performance changes in measured packet throughput in |csit-release| -are calculated against the results from |csit-release-1| -report. Listed mean and standard deviation values are computed based on -a series of the same tests executed against respective VPP releases to -verify test results repeatability, with percentage change calculated for -mean values. Note that the standard deviation is quite high for a small +Relative performance changes in measured NDR, PDR and MRR packet +throughput in |csit-release| are calculated against the test results +from |csit-release-1| report, for tests running on 3-Node Intel Xeon +Haswell testbeds (3n-hsw) in 1-core, 2-core and 4-core (MRR only) +configurations. + +Listed mean and standard deviation values are computed based on a series +of the same tests executed against respective VPP releases to verify +test results repeatability, with percentage change calculated for mean +values. Note that the standard deviation is quite high for a small number of packet throughput tests, what indicates poor test results repeatability and makes the relative change of mean throughput value not fully representative for these tests. The root causes behind poor @@ -71,119 +75,121 @@ results repeatability vary between the test cases. NDR Changes ~~~~~~~~~~~ -NDR throughput changes between releases are available in a -CSV and pretty ASCII formats: +NDR throughput changes between releases are available in CSV and pretty +ASCII formats: - - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-ndr.csv>`_, - - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-ndr.csv>`_, - - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-ndr.txt>`_, - - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-ndr.txt>`_. + - `CSV 1t1c NDR changes <../_static/vpp/performance-changes-1t1c-ndr.csv>`_, + - `CSV 2t2c NDR changes <../_static/vpp/performance-changes-2t2c-ndr.csv>`_, + - `ASCII 1t1c NDR changes <../_static/vpp/performance-changes-1t1c-ndr.txt>`_, + - `ASCII 2t2c NDR changes <../_static/vpp/performance-changes-2t2c-ndr.txt>`_. .. note:: Test results have been generated by `FD.io test executor vpp performance job 3n-hsw`_, - with Robot Framework result + with RF result files csit-vpp-perf-|srelease|-\*.zip `archived here <../_static/archive/>`_. PDR Changes ~~~~~~~~~~~ -PDR throughput changes between releases are available in a -CSV and pretty ASCII formats: +PDR throughput changes between releases are available in CSV and pretty +ASCII formats: - - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-pdr.csv>`_, - - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-pdr.csv>`_, - - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-pdr.txt>`_, - - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-pdr.txt>`_. + - `CSV 1t1c PDR changes <../_static/vpp/performance-changes-1t1c-pdr.csv>`_, + - `CSV 2t2c PDR changes <../_static/vpp/performance-changes-2t2c-pdr.csv>`_, + - `ASCII 1t1c PDR changes <../_static/vpp/performance-changes-1t1c-pdr.txt>`_, + - `ASCII 2t2c PDR changes <../_static/vpp/performance-changes-2t2c-pdr.txt>`_. .. note:: Test results have been generated by `FD.io test executor vpp performance job 3n-hsw`_, - with Robot Framework result + with RF result files csit-vpp-perf-|srelease|-\*.zip `archived here <../_static/archive/>`_. MRR Changes ~~~~~~~~~~~ -MRR small packet throughput changes between releases are available in a -CSV and pretty ASCII formats: +MRR throughput changes between releases are available in CSV and pretty +ASCII formats: - - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-mrr.csv>`_, - - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-mrr.csv>`_, - - `csv format for 4t4c <../_static/vpp/performance-changes-4t4c-mrr.csv>`_, - - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-mrr.txt>`_, - - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-mrr.txt>`_, - - `pretty ASCII format for 4t4c <../_static/vpp/performance-changes-4t4c-mrr.txt>`_. + - `CSV 1t1c MRR changes <../_static/vpp/performance-changes-1t1c-mrr.csv>`_, + - `CSV 2t2c MRR changes <../_static/vpp/performance-changes-2t2c-mrr.csv>`_, + - `CSV 4t4c MRR changes <../_static/vpp/performance-changes-4t4c-mrr.csv>`_, + - `ASCII 1t1c MRR changes <../_static/vpp/performance-changes-1t1c-mrr.txt>`_, + - `ASCII 2t2c MRR changes <../_static/vpp/performance-changes-2t2c-mrr.txt>`_, + - `ASCII 4t4c MRR changes <../_static/vpp/performance-changes-4t4c-mrr.txt>`_. .. note:: Test results have been generated by `FD.io test executor vpp performance job 3n-hsw`_, - with Robot Framework result + with RF result files csit-vpp-perf-|srelease|-\*.zip `archived here <../_static/archive/>`_. -Comparison Across Testbeds --------------------------- +Skx vs. Hsw Comparison +---------------------- -Relative performance changes in measured packet throughputon 3-Node Skx testbed -are calculated against the results measured on 3-Node Hsw testbed. +Relative performance comparison in measured NDR, PDR and MRR packet +throughput is calculated for tests executed on 3-Node Skylake (3n-skx) +and 3-Node Haswell (3n-hsw) physical testbed types in 1-core +configurations. -NDR Changes -~~~~~~~~~~~ +NDR Comparison +~~~~~~~~~~~~~~ -NDR changes between testbeds are available in a -CSV and pretty ASCII formats: +NDR comparison between testbed types is available in CSV and pretty +ASCII formats: - - `csv format for ndr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_, - - `pretty ASCII format for ndr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_. + - `CSV 1c NDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_, + - `ASCII 1c NDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_. .. note:: Test results have been generated by `FD.io test executor vpp performance job 3n-hsw`_ and `FD.io test executor vpp performance job 3n-skx`_ - with Robot Framework result + with RF result files csit-vpp-perf-|srelease|-\*.zip `archived here <../_static/archive/>`_. -PDR Changes -~~~~~~~~~~~ +PDR Comparison +~~~~~~~~~~~~~~ -PDR throughput changes between testbeds are available in a -CSV and pretty ASCII formats: +PDR comparison between testbed types is available in CSV and pretty +ASCII formats: - - `csv format for pdr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_, - - `pretty ASCII format for pdr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_. + - `CSV 1c PDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_, + - `ASCII 1c PDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_. .. note:: Test results have been generated by `FD.io test executor vpp performance job 3n-hsw`_ and `FD.io test executor vpp performance job 3n-skx`_ - with Robot Framework result + with RF result files csit-vpp-perf-|srelease|-\*.zip `archived here <../_static/archive/>`_. -MRR Changes -~~~~~~~~~~~ +MRR Comparison +~~~~~~~~~~~~~~ -MRR throughput changes between testbeds are available in a -CSV and pretty ASCII formats: +MRR comparison between testbed types is available in CSV and pretty +ASCII formats: - - `csv format for mrr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_, - - `pretty ASCII format for mrr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_. + - `CSV 1c MRR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_, + - `ASCII 1c MRR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_. .. note:: Test results have been generated by `FD.io test executor vpp performance job 3n-hsw`_ and `FD.io test executor vpp performance job 3n-skx`_ - with Robot Framework result + with RF result files csit-vpp-perf-|srelease|-\*.zip `archived here <../_static/archive/>`_. |