diff options
14 files changed, 369 insertions, 315 deletions
diff --git a/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst b/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst new file mode 100644 index 0000000000..c148b0619c --- /dev/null +++ b/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst @@ -0,0 +1,61 @@ +3n-Skx vs. 3n-Hsw Testbeds +-------------------------- + +Relative comparison of |dpdk-release| Testpmd and L3fwd packet +throughput (NDR, PDR and MRR) is calculated for the same tests executed +on 3-Node Skylake (3n-skx) and 3-Node Haswell (3n-hsw) physical testbed +types, in 1-core, 2-core and 4-core configurations. + +NDR Comparison +~~~~~~~~~~~~~~ + +NDR throughput comparison between testbeds is available in CSV and +pretty ASCII formats: + + - `CSV NDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_, + - `ASCII NDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_. + +.. note:: + + Test results have been generated by + `FD.io test executor dpdk performance job 3n-hsw`_ and + `FD.io test executor dpdk performance job 3n-skx`_ + with RF result + files csit-dpdk-perf-|srelease|-\*.zip + `archived here <../_static/archive/>`_. + +PDR Comparison +~~~~~~~~~~~~~~ + +PDR throughput comparison between testbeds is available in CSV and +pretty ASCII formats: + + - `CSV PDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_, + - `ASCII PDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_. + +.. note:: + + Test results have been generated by + `FD.io test executor dpdk performance job 3n-hsw`_ and + `FD.io test executor dpdk performance job 3n-skx`_ + with RF result + files csit-dpdk-perf-|srelease|-\*.zip + `archived here <../_static/archive/>`_. + +MRR Comparison +~~~~~~~~~~~~~~ + +MRR throughput comparison between testbeds is available in CSV and +pretty ASCII formats: + + - `CSV MRR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_, + - `ASCII MRR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_. + +.. note:: + + Test results have been generated by + `FD.io test executor dpdk performance job 3n-hsw`_ and + `FD.io test executor dpdk performance job 3n-skx`_ + with RF result + files csit-vpp-perf-|srelease|-\*.zip + `archived here <../_static/archive/>`_.
\ No newline at end of file diff --git a/docs/report/dpdk_performance_tests/comparisons/current_vs_previous_release.rst b/docs/report/dpdk_performance_tests/comparisons/current_vs_previous_release.rst new file mode 100644 index 0000000000..6a7829ac8a --- /dev/null +++ b/docs/report/dpdk_performance_tests/comparisons/current_vs_previous_release.rst @@ -0,0 +1,51 @@ +Current vs. Previous Release +---------------------------- + +Relative comparison of DPDK Testpmd and L3fwd packet throughput (NDR, +PDR and MRR) between |dpdk-release| and DPDK-18.02 (measured for |csit- +release| and |csit-release-1| respectively) is calculated from results +of tests running on 3-Node Intel Xeon Haswell testbeds (3n-hsw) in +1-core, 2-core and 4-core (MRR only) configurations. + +Listed mean and standard deviation values are computed based on a series +of the same tests executed against respective DPDK releases to verify +test results repeatability, with percentage change calculated for mean +values. + +NDR Comparison +~~~~~~~~~~~~~~ + +NDR throughput comparison between releases is available in CSV and pretty +ASCII formats: + + - `CSV 1t1c NDR changes <../_static/dpdk/performance-changes-1t1c-ndr.csv>`_, + - `CSV 2t2c NDR changes <../_static/dpdk/performance-changes-2t2c-ndr.csv>`_, + - `ASCII 1t1c NDR changes <../_static/dpdk/performance-changes-1t1c-ndr.txt>`_, + - `ASCII 2t2c NDR changes <../_static/dpdk/performance-changes-2t2c-ndr.txt>`_. + +.. note:: + + Test results have been generated by + `FD.io test executor dpdk performance job 3n-hsw`_ + with RF result + files csit-dpdk-perf-|srelease|-\*.zip + `archived here <../_static/archive/>`_. + +PDR Comparison +~~~~~~~~~~~~~~ + +PDR throughput comparison between releases is available in CSV and pretty +ASCII formats: + + - `CSV 1t1c PDR changes <../_static/dpdk/performance-changes-1t1c-pdr.csv>`_, + - `CSV 2t2c PDR changes <../_static/dpdk/performance-changes-2t2c-pdr.csv>`_, + - `ASCII 1t1c PDR changes <../_static/dpdk/performance-changes-1t1c-pdr.txt>`_, + - `ASCII 2t2c PDR changes <../_static/dpdk/performance-changes-2t2c-pdr.txt>`_. + +.. note:: + + Test results have been generated by + `FD.io test executor dpdk performance job 3n-hsw`_ + with RF result + files csit-dpdk-perf-|srelease|-\*.zip + `archived here <../_static/archive/>`_. diff --git a/docs/report/dpdk_performance_tests/comparisons/index.rst b/docs/report/dpdk_performance_tests/comparisons/index.rst new file mode 100644 index 0000000000..ace3baf091 --- /dev/null +++ b/docs/report/dpdk_performance_tests/comparisons/index.rst @@ -0,0 +1,13 @@ + +.. raw:: latex + + \clearpage + +Comparisons +============= + +.. toctree:: + + current_vs_previous_release + 3n-skx_vs_3n-hsw_testbeds + diff --git a/docs/report/dpdk_performance_tests/csit_release_notes.rst b/docs/report/dpdk_performance_tests/csit_release_notes.rst index 9fe1ce3c50..e0e20289b0 100644 --- a/docs/report/dpdk_performance_tests/csit_release_notes.rst +++ b/docs/report/dpdk_performance_tests/csit_release_notes.rst @@ -13,8 +13,7 @@ Changes in |csit-release| - **MRR Throughput**: MRR (Maximum Receive Rate) test code has now configurable trial duration and number of consecutive executions. Coverage of MRR tests has been extended across more test - scenarios. MRR tests are used for continuous performance trending - and for comparison between DPDK releases. + scenarios. MRR tests are used for continuous performance trending. - **MLRsearch Throughput**: MLRsearch algorithm has been introduced for all NDR and PDR throughput tests. All tests that previously @@ -38,118 +37,6 @@ Changes in |csit-release| - **Performance trending**: Added continuous performance trending with anomaly detection and analysis for DPDK. -Performance Changes -------------------- - -Relative performance changes in measured NDR, PDR and MRR packet -throughput in |csit-release| are calculated against the test results -from |csit-release-1| report, for tests running on 3-Node Intel Xeon -Haswell testbeds (3n-hsw) in 1-core, 2-core and 4-core (MRR only) -configurations. - -Listed mean and standard deviation values are computed based on a series -of the same tests executed against respective VPP releases to verify -test results repeatability, with percentage change calculated for mean -values. - -NDR Changes -~~~~~~~~~~~ - -NDR throughput changes between releases are available in CSV and pretty -ASCII formats: - - - `CSV 1t1c NDR changes <../_static/dpdk/performance-changes-1t1c-ndr.csv>`_, - - `CSV 2t2c NDR changes <../_static/dpdk/performance-changes-2t2c-ndr.csv>`_, - - `ASCII 1t1c NDR changes <../_static/dpdk/performance-changes-1t1c-ndr.txt>`_, - - `ASCII 2t2c NDR changes <../_static/dpdk/performance-changes-2t2c-ndr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor dpdk performance job 3n-hsw`_ - with RF result - files csit-dpdk-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -PDR Changes -~~~~~~~~~~~ - -PDR throughput changes between releases are available in CSV and pretty -ASCII formats: - - - `CSV 1t1c PDR changes <../_static/dpdk/performance-changes-1t1c-pdr.csv>`_, - - `CSV 2t2c PDR changes <../_static/dpdk/performance-changes-2t2c-pdr.csv>`_, - - `ASCII 1t1c PDR changes <../_static/dpdk/performance-changes-1t1c-pdr.txt>`_, - - `ASCII 2t2c PDR changes <../_static/dpdk/performance-changes-2t2c-pdr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor dpdk performance job 3n-hsw`_ - with RF result - files csit-dpdk-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -Skx vs. Hsw Comparison ----------------------- - -Relative performance changes in measured packet throughput on 3-Node Skx testbed -are calculated against the results measured on 3-Node Hsw testbed. - -NDR Comparison -~~~~~~~~~~~~~~ - -NDR throughput changes between testbeds are available in a CSV and pretty ASCII -formats: - - - `CSV 1c NDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_, - - `ASCII 1c NDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor dpdk performance job 3n-hsw`_ and - `FD.io test executor dpdk performance job 3n-skx`_ - with RF result - files csit-dpdk-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -PDR Comparison -~~~~~~~~~~~~~~ - -PDR throughput changes between testbeds are available in a CSV and pretty ASCII -formats: - - - `CSV 1c PDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_, - - `ASCII 1c PDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor dpdk performance job 3n-hsw`_ and - `FD.io test executor dpdk performance job 3n-skx`_ - with RF result - files csit-dpdk-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -MRR Comparison -~~~~~~~~~~~~~~ - -MRR throughput changes between testbeds are available in a -CSV and pretty ASCII formats: - - - `CSV 1c MRR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_, - - `ASCII 1c MRR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor dpdk performance job 3n-hsw`_ and - `FD.io test executor dpdk performance job 3n-skx`_ - with RF result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - Known Issues ------------ diff --git a/docs/report/dpdk_performance_tests/index.rst b/docs/report/dpdk_performance_tests/index.rst index 3aa32435f0..07632b6368 100644 --- a/docs/report/dpdk_performance_tests/index.rst +++ b/docs/report/dpdk_performance_tests/index.rst @@ -7,5 +7,7 @@ DPDK Performance csit_release_notes packet_throughput_graphs/index packet_latency_graphs/index + comparisons/index + throughput_trending test_environment documentation diff --git a/docs/report/dpdk_performance_tests/throughput_trending.rst b/docs/report/dpdk_performance_tests/throughput_trending.rst new file mode 100644 index 0000000000..2b45f4bdaf --- /dev/null +++ b/docs/report/dpdk_performance_tests/throughput_trending.rst @@ -0,0 +1,17 @@ +Throughput Trending +------------------- + +In addition to reporting throughput comparison between DPDK releases, +CSIT provides regular performance trending for DPDK release branches: + +#. `Performance Dashboard <https://docs.fd.io/csit/master/trending/introduction/index.html>`_ + - per DPDK test case throughput trend, trend compliance and summary of + detected anomalies. + +#. `Trending Methodology <https://docs.fd.io/csit/master/trending/methodology/index.html>`_ + - throughput test metrics, trend calculations and anomaly + classification (progression, regression). + +#. `DPDK Apps Trendline Graphs <https://docs.fd.io/csit/master/trending/trending/dpdk.html>`_ + - weekly DPDK Testpmd and L3fwd MRR throughput measurements against + the trendline with anomaly highlights and associated CSIT test jobs.
\ No newline at end of file diff --git a/docs/report/index.html.template b/docs/report/index.html.template index 66aa54d752..ad61073b09 100644 --- a/docs/report/index.html.template +++ b/docs/report/index.html.template @@ -20,6 +20,8 @@ CSIT 18.07 vpp_performance_tests/throughput_speedup_multi_core/index vpp_performance_tests/packet_latency_graphs/index vpp_performance_tests/http_server_performance/index + vpp_performance_tests/comparisons/index + vpp_performance_tests/throughput_trending vpp_performance_tests/test_environment vpp_performance_tests/documentation/index @@ -31,6 +33,8 @@ CSIT 18.07 dpdk_performance_tests/csit_release_notes dpdk_performance_tests/packet_throughput_graphs/index dpdk_performance_tests/packet_latency_graphs/index + dpdk_performance_tests/comparisons/index + dpdk_performance_tests/throughput_trending dpdk_performance_tests/test_environment dpdk_performance_tests/documentation diff --git a/docs/report/introduction/introduction.rst b/docs/report/introduction/introduction.rst index c88aeb4e30..e5fa19b08d 100644 --- a/docs/report/introduction/introduction.rst +++ b/docs/report/introduction/introduction.rst @@ -20,22 +20,21 @@ files are also provided. |csit-release| report contains following main sections and sub-sections: -#. **Introduction**: general introduction to CSIT project. +#. **Introduction**: General introduction to CSIT project. - - **Introduction**: this section. - - **Test Scenarios Overview**: a brief overview of test scenarios + - **Introduction**: This section. + - **Test Scenarios Overview**: A brief overview of test scenarios covered in this report. - - **Physical Testbeds**: description of physical testbeds. - - **Performance Test Methodology**: benchmarking methodologies. + - **Physical Testbeds**: Description of physical testbeds. + - **Performance Test Methodology**: Benchmarking methodologies. #. **VPP Performance**: VPP performance tests executed in physical FD.io testbeds. - - **Overview**: tested logical topologies, test coverage and naming + - **Overview**: Tested logical topologies, test coverage and naming specifics. - - **Release Notes**: changes in |csit-release|, added tests, - performance changes, environment or methodology changes, known - issues. + - **Release Notes**: Changes in |csit-release|, added tests, + environment or methodology changes, known issues. - **Packet Throughput**: NDR, PDR throughput graphs based on results from repeated same test job executions to verify repeatibility of measurements. @@ -45,67 +44,75 @@ files are also provided. graphs based on results from test job executions. - **HTTP/TCP Performance**: HTTP/TCP VPP test server performance graphs. - - **Test Environment**: performance test environment configuration. - - **Documentation**: documentation of K8s Pod/Container orchestration + - **Comparisons**: Performance comparisons between VPP releases and + between different testbed types. + - **Throughput Trending**: References to continuous VPP performance + trending. + - **Test Environment**: Performance test environment configuration. + - **Documentation**: Documentation of K8s Pod/Container orchestration in CSIT and pointers to CSIT source code documentation for VPP performance tests. #. **DPDK Performance**: DPDK performance tests executed in physical FD.io testbeds. - - **Overview**: tested logical topologies, test coverage. - - **Release Notes**: changes in |csit-release|, known issues. + - **Overview**: Tested logical topologies, test coverage. + - **Release Notes**: Changes in |csit-release|, known issues. - **Packet Throughput**: NDR, PDR throughput graphs based on results from repeated same test job executions to verify repeatibility of measurements. - **Packet Latency**: Latency graphs based on results from test job executions. - - **Test Environment**: performance test environment configuration. - - **Documentation**: pointers to CSIT source code documentation for + - **Comparisons**: Performance comparisons between DPDK releases and + between different testbed types. + - **Throughput Trending**: References to regular DPDK performance + trending. + - **Test Environment**: Performance test environment configuration. + - **Documentation**: Pointers to CSIT source code documentation for DPDK performance tests. #. **VPP Functional**: VPP functional tests executed in virtual FD.io testbeds. - - **Overview**: tested virtual topologies, test coverage and naming + - **Overview**: Tested virtual topologies, test coverage and naming specifics; - - **Release Notes** - changes in |csit-release|, added tests, + - **Release Notes**: Changes in |csit-release|, added tests, environment or methodology changes, known issues. - - **Test Environment**: functional test environment configuration. - - **Documentation**: pointers to CSIT source code documentation for + - **Test Environment**: Functional test environment configuration. + - **Documentation**: Pointers to CSIT source code documentation for VPP functional tests. #. **Honeycomb Functional**: Honeycomb functional tests executed in virtual FD.io testbeds. - - **Overview**: tested virtual topologies, test coverage and naming + - **Overview**: Tested virtual topologies, test coverage and naming specifics; - - **Release Notes** - changes in |csit-release|, known issues. - - **Test Environment**: functional test environment configuration. - - **Documentation**: pointers to CSIT source code documentation for + - **Release Notes**: Changes in |csit-release|, known issues. + - **Test Environment**: Functional test environment configuration. + - **Documentation**: Pointers to CSIT source code documentation for Honeycomb functional tests. #. **NSH_SFC Functional**: NSH_SFC functional tests executed in virtual FD.io testbeds. - - **Overview**: tested virtual topologies, test coverage and naming + - **Overview**: Tested virtual topologies, test coverage and naming specifics; - - **Release Notes** - changes in |csit-release|, known issues. - - **Test Environment**: functional test environment configuration. - - **Documentation**: pointers to CSIT source code documentation for + - **Release Notes**: Changes in |csit-release|, known issues. + - **Test Environment**: Functional test environment configuration. + - **Documentation**: Pointers to CSIT source code documentation for NSH_SFC functional tests. #. **DMM Functional**: DMM functional tests executed in virtual FD.io testbeds. - - **Overview**: tested virtual topologies, test coverage and naming + - **Overview**: Tested virtual topologies, test coverage and naming specifics; - - **Release Notes** - changes in |csit-release|, known issues. - - **Test Environment**: functional test environment configuration. - - **Documentation**: pointers to CSIT source code documentation for + - **Release Notes**: Changes in |csit-release|, known issues. + - **Test Environment**: Functional test environment configuration. + - **Documentation**: Pointers to CSIT source code documentation for DMM functional tests. -#. **Detailed Results**: detailed result tables auto-generated from CSIT +#. **Detailed Results**: Detailed result tables auto-generated from CSIT test job executions using RF (Robot Framework) output files as sources. @@ -115,19 +122,19 @@ files are also provided. NDR/PDR throughput. - **DPDK Performance**: DPDK Testpmd and L3fwd NDR/PDR throughput and latency. - - **VPP Functional**: detailed VPP functional results. - - **Honeycomb Functional**: detailed HoneyComb functional results. - - **NSH_SFC Functional**: detailed nsh-plugin functional results. - - **DMM Functional**: detailed DMM functional results. + - **VPP Functional**: Detailed VPP functional results. + - **Honeycomb Functional**: Detailed HoneyComb functional results. + - **NSH_SFC Functional**: Detailed nsh-plugin functional results. + - **DMM Functional**: Detailed DMM functional results. #. **Test Configuration**: VPP DUT configuration data based on VPP API Test (VAT) Commands History auto-generated from CSIT test job executions using RF output files as sources. - - **VPP Performance NDR/PDR**: configuration data. - - **VPP Performance MRR**: configuration data. - - **VPP K8s Container Memif**: configuration data. - - **VPP Functional**: configuration data. + - **VPP Performance NDR/PDR**: Configuration data. + - **VPP Performance MRR**: Configuration data. + - **VPP K8s Container Memif**: Configuration data. + - **VPP Functional**: Configuration data. #. **Test Operational Data**: VPP DUT operational data auto-generated from CSIT test job executions using RFoutput files as sources. @@ -135,12 +142,12 @@ files are also provided. - **VPP Performance NDR/PDR**: VPP `show run` outputs under test load. -#. **CSIT Framework Documentation**: description of the overall FD.io +#. **CSIT Framework Documentation**: Description of the overall FD.io CSIT framework. - - **Design**: framework modular design hierarchy. - - **Test naming**: test naming convention. - - **Presentation and Analytics Layer**: description of PAL CSIT + - **Design**: Framework modular design hierarchy. + - **Test naming**: Test naming convention. + - **Presentation and Analytics Layer**: Description of PAL CSIT analytics module. - **CSIT RF Tags Descriptions**: CSIT RF Tags used for test suite and test case grouping and selection. diff --git a/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst b/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst new file mode 100644 index 0000000000..c9e0bbbb99 --- /dev/null +++ b/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst @@ -0,0 +1,61 @@ +3n-Skx vs. 3n-Hsw Testbeds +-------------------------- + +Relative comparison of |vpp-release| packet throughput (NDR, PDR and +MRR) is calculated for the same tests executed on 3-Node Skylake (3n- +skx) and 3-Node Haswell (3n-hsw) physical testbed types, in 1-core, +2-core and 4-core configurations. + +NDR Comparison +~~~~~~~~~~~~~~ + +NDR comparison between testbed types is available in CSV and pretty +ASCII formats: + + - `CSV NDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_, + - `ASCII NDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_. + +.. note:: + + Test results have been generated by + `FD.io test executor vpp performance job 3n-hsw`_ and + `FD.io test executor vpp performance job 3n-skx`_ + with RF result + files csit-vpp-perf-|srelease|-\*.zip + `archived here <../_static/archive/>`_. + +PDR Comparison +~~~~~~~~~~~~~~ + +PDR comparison between testbed types is available in CSV and pretty +ASCII formats: + + - `CSV PDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_, + - `ASCII PDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_. + +.. note:: + + Test results have been generated by + `FD.io test executor vpp performance job 3n-hsw`_ and + `FD.io test executor vpp performance job 3n-skx`_ + with RF result + files csit-vpp-perf-|srelease|-\*.zip + `archived here <../_static/archive/>`_. + +MRR Comparison +~~~~~~~~~~~~~~ + +MRR comparison between testbed types is available in CSV and pretty +ASCII formats: + + - `CSV MRR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_, + - `ASCII MRR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_. + +.. note:: + + Test results have been generated by + `FD.io test executor vpp performance job 3n-hsw`_ and + `FD.io test executor vpp performance job 3n-skx`_ + with RF result + files csit-vpp-perf-|srelease|-\*.zip + `archived here <../_static/archive/>`_. diff --git a/docs/report/vpp_performance_tests/comparisons/current_vs_previous_release.rst b/docs/report/vpp_performance_tests/comparisons/current_vs_previous_release.rst new file mode 100644 index 0000000000..c2a87dc2ab --- /dev/null +++ b/docs/report/vpp_performance_tests/comparisons/current_vs_previous_release.rst @@ -0,0 +1,76 @@ +Current vs. Previous Release +---------------------------- + +Relative comparison of VPP packet throughput (NDR, PDR and MRR) between +|vpp-release| and |vpp-release-1| (measured for |csit-release| and +|csit-release-1| respectively) is calculated from results of tests +running on 3-Node Intel Xeon Haswell testbeds (3n-hsw) in 1-core, 2-core +and 4-core (MRR only) configurations. + +Listed mean and standard deviation values are computed based on a series +of the same tests executed against respective VPP releases to verify +test results repeatability, with percentage change calculated for mean +values. Note that the standard deviation is quite high for a small +number of packet throughput tests, what indicates poor test results +repeatability and makes the relative change of mean throughput value not +fully representative for these tests. The root causes behind poor +results repeatability vary between the test cases. + +NDR Comparison +~~~~~~~~~~~~~~ + +NDR throughput comparison between releases is available in CSV and pretty +ASCII formats: + + - `CSV 1t1c NDR changes <../_static/vpp/performance-changes-1t1c-ndr.csv>`_, + - `CSV 2t2c NDR changes <../_static/vpp/performance-changes-2t2c-ndr.csv>`_, + - `ASCII 1t1c NDR changes <../_static/vpp/performance-changes-1t1c-ndr.txt>`_, + - `ASCII 2t2c NDR changes <../_static/vpp/performance-changes-2t2c-ndr.txt>`_. + +.. note:: + + Test results have been generated by + `FD.io test executor vpp performance job 3n-hsw`_, + with RF result + files csit-vpp-perf-|srelease|-\*.zip + `archived here <../_static/archive/>`_. + +PDR Comparison +~~~~~~~~~~~~~~ + +PDR throughput comparison between releases is available in CSV and pretty +ASCII formats: + + - `CSV 1t1c PDR changes <../_static/vpp/performance-changes-1t1c-pdr.csv>`_, + - `CSV 2t2c PDR changes <../_static/vpp/performance-changes-2t2c-pdr.csv>`_, + - `ASCII 1t1c PDR changes <../_static/vpp/performance-changes-1t1c-pdr.txt>`_, + - `ASCII 2t2c PDR changes <../_static/vpp/performance-changes-2t2c-pdr.txt>`_. + +.. note:: + + Test results have been generated by + `FD.io test executor vpp performance job 3n-hsw`_, + with RF result + files csit-vpp-perf-|srelease|-\*.zip + `archived here <../_static/archive/>`_. + +MRR Comparison +~~~~~~~~~~~~~~ + +MRR throughput comparison between releases is available in CSV and pretty +ASCII formats: + + - `CSV 1t1c MRR changes <../_static/vpp/performance-changes-1t1c-mrr.csv>`_, + - `CSV 2t2c MRR changes <../_static/vpp/performance-changes-2t2c-mrr.csv>`_, + - `CSV 4t4c MRR changes <../_static/vpp/performance-changes-4t4c-mrr.csv>`_, + - `ASCII 1t1c MRR changes <../_static/vpp/performance-changes-1t1c-mrr.txt>`_, + - `ASCII 2t2c MRR changes <../_static/vpp/performance-changes-2t2c-mrr.txt>`_, + - `ASCII 4t4c MRR changes <../_static/vpp/performance-changes-4t4c-mrr.txt>`_. + +.. note:: + + Test results have been generated by + `FD.io test executor vpp performance job 3n-hsw`_, + with RF result + files csit-vpp-perf-|srelease|-\*.zip + `archived here <../_static/archive/>`_. diff --git a/docs/report/vpp_performance_tests/comparisons/index.rst b/docs/report/vpp_performance_tests/comparisons/index.rst new file mode 100644 index 0000000000..a9179c6900 --- /dev/null +++ b/docs/report/vpp_performance_tests/comparisons/index.rst @@ -0,0 +1,13 @@ + +.. raw:: latex + + \clearpage + +Comparisons +=========== + +.. toctree:: + + current_vs_previous_release + 3n-skx_vs_3n-hsw_testbeds + diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst index c9df5478bc..0318e7a01e 100644 --- a/docs/report/vpp_performance_tests/csit_release_notes.rst +++ b/docs/report/vpp_performance_tests/csit_release_notes.rst @@ -54,163 +54,6 @@ Changes in |csit-release| - **General Code Housekeeping**: Ongoing RF keywords optimizations, removal of redundant RF keywords. -Performance Changes -------------------- - -Relative performance changes in measured NDR, PDR and MRR packet -throughput in |csit-release| are calculated against the test results -from |csit-release-1| report, for tests running on 3-Node Intel Xeon -Haswell testbeds (3n-hsw) in 1-core, 2-core and 4-core (MRR only) -configurations. - -Listed mean and standard deviation values are computed based on a series -of the same tests executed against respective VPP releases to verify -test results repeatability, with percentage change calculated for mean -values. Note that the standard deviation is quite high for a small -number of packet throughput tests, what indicates poor test results -repeatability and makes the relative change of mean throughput value not -fully representative for these tests. The root causes behind poor -results repeatability vary between the test cases. - -NDR Changes -~~~~~~~~~~~ - -NDR throughput changes between releases are available in CSV and pretty -ASCII formats: - - - `CSV 1t1c NDR changes <../_static/vpp/performance-changes-1t1c-ndr.csv>`_, - - `CSV 2t2c NDR changes <../_static/vpp/performance-changes-2t2c-ndr.csv>`_, - - `ASCII 1t1c NDR changes <../_static/vpp/performance-changes-1t1c-ndr.txt>`_, - - `ASCII 2t2c NDR changes <../_static/vpp/performance-changes-2t2c-ndr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_, - with RF result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -PDR Changes -~~~~~~~~~~~ - -PDR throughput changes between releases are available in CSV and pretty -ASCII formats: - - - `CSV 1t1c PDR changes <../_static/vpp/performance-changes-1t1c-pdr.csv>`_, - - `CSV 2t2c PDR changes <../_static/vpp/performance-changes-2t2c-pdr.csv>`_, - - `ASCII 1t1c PDR changes <../_static/vpp/performance-changes-1t1c-pdr.txt>`_, - - `ASCII 2t2c PDR changes <../_static/vpp/performance-changes-2t2c-pdr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_, - with RF result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -MRR Changes -~~~~~~~~~~~ - -MRR throughput changes between releases are available in CSV and pretty -ASCII formats: - - - `CSV 1t1c MRR changes <../_static/vpp/performance-changes-1t1c-mrr.csv>`_, - - `CSV 2t2c MRR changes <../_static/vpp/performance-changes-2t2c-mrr.csv>`_, - - `CSV 4t4c MRR changes <../_static/vpp/performance-changes-4t4c-mrr.csv>`_, - - `ASCII 1t1c MRR changes <../_static/vpp/performance-changes-1t1c-mrr.txt>`_, - - `ASCII 2t2c MRR changes <../_static/vpp/performance-changes-2t2c-mrr.txt>`_, - - `ASCII 4t4c MRR changes <../_static/vpp/performance-changes-4t4c-mrr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_, - with RF result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -Skx vs. Hsw Comparison ----------------------- - -Relative performance comparison in measured NDR, PDR and MRR packet -throughput is calculated for tests executed on 3-Node Skylake (3n-skx) -and 3-Node Haswell (3n-hsw) physical testbed types in 1-core -configurations. - -NDR Comparison -~~~~~~~~~~~~~~ - -NDR comparison between testbed types is available in CSV and pretty -ASCII formats: - - - `CSV 1c NDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_, - - `ASCII 1c NDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_ and - `FD.io test executor vpp performance job 3n-skx`_ - with RF result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -PDR Comparison -~~~~~~~~~~~~~~ - -PDR comparison between testbed types is available in CSV and pretty -ASCII formats: - - - `CSV 1c PDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_, - - `ASCII 1c PDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_ and - `FD.io test executor vpp performance job 3n-skx`_ - with RF result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -MRR Comparison -~~~~~~~~~~~~~~ - -MRR comparison between testbed types is available in CSV and pretty -ASCII formats: - - - `CSV 1c MRR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_, - - `ASCII 1c MRR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_ and - `FD.io test executor vpp performance job 3n-skx`_ - with RF result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -Throughput Trending -------------------- - -In addition to reporting throughput changes between VPP releases, CSIT -provides continuous performance trending for VPP master branch: - -#. `VPP Performance Dashboard <https://docs.fd.io/csit/master/trending/introduction/index.html>`_ - - per VPP test case throughput trend, trend compliance and summary of - detected anomalies. - -#. `Trending Methodology <https://docs.fd.io/csit/master/trending/methodology/index.html>`_ - - throughput test metrics, trend calculations and anomaly - classification (progression, regression, outlier). - -#. `Trendline Graphs <https://docs.fd.io/csit/master/trending/trending/index.html>`_ - - per VPP build MRR throughput measurements against the trendline - with anomaly highlights, with associated CSIT test jobs. - Known Issues ------------ diff --git a/docs/report/vpp_performance_tests/index.rst b/docs/report/vpp_performance_tests/index.rst index abef2c8667..888be08ac2 100644 --- a/docs/report/vpp_performance_tests/index.rst +++ b/docs/report/vpp_performance_tests/index.rst @@ -9,5 +9,7 @@ VPP Performance throughput_speedup_multi_core/index packet_latency_graphs/index http_server_performance/index + comparisons/index + throughput_trending test_environment documentation/index diff --git a/docs/report/vpp_performance_tests/throughput_trending.rst b/docs/report/vpp_performance_tests/throughput_trending.rst new file mode 100644 index 0000000000..c2ca421d57 --- /dev/null +++ b/docs/report/vpp_performance_tests/throughput_trending.rst @@ -0,0 +1,17 @@ +Throughput Trending +------------------- + +In addition to reporting throughput comparison between VPP releases, +CSIT provides continuous performance trending for VPP master branch: + +#. `VPP Performance Dashboard <https://docs.fd.io/csit/master/trending/introduction/index.html>`_ + - per VPP test case throughput trend, trend compliance and summary of + detected anomalies. + +#. `Trending Methodology <https://docs.fd.io/csit/master/trending/methodology/index.html>`_ + - throughput test metrics, trend calculations and anomaly + classification (progression, regression). + +#. `Trendline Graphs <https://docs.fd.io/csit/master/trending/trending/index.html>`_ + - per VPP build MRR throughput measurements against the trendline + with anomaly highlights and associated CSIT test jobs.
\ No newline at end of file |