diff options
12 files changed, 268 insertions, 230 deletions
diff --git a/docs/report/dpdk_performance_tests/comparisons/2n-skx_vs_2n-clx_testbeds.rst b/docs/report/dpdk_performance_tests/comparisons/2n-skx_vs_2n-clx_testbeds.rst index 6e14312c36..ec7a9f3609 100644 --- a/docs/report/dpdk_performance_tests/comparisons/2n-skx_vs_2n-clx_testbeds.rst +++ b/docs/report/dpdk_performance_tests/comparisons/2n-skx_vs_2n-clx_testbeds.rst @@ -1,5 +1,5 @@ -2n-Skx vs. 2n-Clx Testbeds --------------------------- +2n-Skx vs 2n-Clx Testbeds +------------------------- Relative comparison of |dpdk-release| Testpmd and L3fwd packet throughput (NDR, PDR and MRR) is calculated for the same tests executed diff --git a/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_2n-skx_testbeds.rst b/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_2n-skx_testbeds.rst index bafbb38e20..2eb1e7b961 100644 --- a/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_2n-skx_testbeds.rst +++ b/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_2n-skx_testbeds.rst @@ -1,8 +1,8 @@ .. _dpdk_compare_topologies_3n-Skx_vs_2n-Skx: -3n-Skx vs. 2n-Skx Testbeds --------------------------- +3n-Skx vs 2n-Skx Testbeds +------------------------- Relative comparison of |dpdk-release| Testpmd and L3fwd packet throughput (NDR, PDR and MRR) is calculated for the same tests executed diff --git a/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst b/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst index e60606a6ab..8b581f91d2 100644 --- a/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst +++ b/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst @@ -1,5 +1,5 @@ -3n-Skx vs. 3n-Hsw Testbeds --------------------------- +3n-Skx vs 3n-Hsw Testbeds +------------------------- Relative comparison of |dpdk-release| Testpmd and L3fwd packet throughput (NDR, PDR and MRR) is calculated for the same tests executed diff --git a/docs/report/dpdk_performance_tests/comparisons/current_vs_previous_release.rst b/docs/report/dpdk_performance_tests/comparisons/current_vs_previous_release.rst index 4557c43600..6358185c30 100644 --- a/docs/report/dpdk_performance_tests/comparisons/current_vs_previous_release.rst +++ b/docs/report/dpdk_performance_tests/comparisons/current_vs_previous_release.rst @@ -1,8 +1,8 @@ .. _dpdk_compare_current_vs_previous_release: -Current vs. Previous Release ----------------------------- +Current vs Previous Release +--------------------------- Relative comparison of DPDK Testpmd and L3fwd packet throughput (NDR, PDR and MRR) between |dpdk-release| and |dpdk-release-1| (measured for diff --git a/docs/report/introduction/methodology_data_plane_throughput/methodology_data_plane_throughput.rst b/docs/report/introduction/methodology_data_plane_throughput/methodology_data_plane_throughput.rst index 764e198d0f..3df54b3963 100644 --- a/docs/report/introduction/methodology_data_plane_throughput/methodology_data_plane_throughput.rst +++ b/docs/report/introduction/methodology_data_plane_throughput/methodology_data_plane_throughput.rst @@ -75,7 +75,7 @@ trending jobs enabling detection of performance anomalies (regressions, progressions) resulting from data plane code changes. MRR tests are also used for VPP per patch performance jobs verifying -patch performance vs. parent. CSIT reports include MRR throughput +patch performance vs parent. CSIT reports include MRR throughput comparisons between releases and test environments. Small frame sizes only (64b/78B, IMIX). diff --git a/docs/report/vpp_performance_tests/comparisons/2n-clx_vs_3n-hsw_testbeds.rst b/docs/report/vpp_performance_tests/comparisons/2n-clx_vs_3n-hsw_testbeds.rst index 5dccf34f69..4837b6409b 100644 --- a/docs/report/vpp_performance_tests/comparisons/2n-clx_vs_3n-hsw_testbeds.rst +++ b/docs/report/vpp_performance_tests/comparisons/2n-clx_vs_3n-hsw_testbeds.rst @@ -1,5 +1,5 @@ -2n-Clx vs. 3n-Hsw Testbeds --------------------------- +2n-Clx vs 3n-Hsw Testbeds +------------------------- Relative comparison of |vpp-release| packet throughput (NDR, PDR and MRR) is calculated for the same tests executed on 3-Node Skylake (3n- diff --git a/docs/report/vpp_performance_tests/comparisons/2n-skx_vs_2n-clx_testbeds.rst b/docs/report/vpp_performance_tests/comparisons/2n-skx_vs_2n-clx_testbeds.rst index 328fa1968c..3b9abb80f4 100644 --- a/docs/report/vpp_performance_tests/comparisons/2n-skx_vs_2n-clx_testbeds.rst +++ b/docs/report/vpp_performance_tests/comparisons/2n-skx_vs_2n-clx_testbeds.rst @@ -1,5 +1,5 @@ -2n-Skx vs. 2n-Clx Testbeds --------------------------- +2n-Skx vs 2n-Clx Testbeds +------------------------- Relative comparison of |vpp-release| packet throughput (NDR, PDR and MRR) is calculated for the same tests executed on 2-Node Skylake (2n- diff --git a/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_2n-skx_testbeds.rst b/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_2n-skx_testbeds.rst index b71e2bb988..bb4a23ef5a 100644 --- a/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_2n-skx_testbeds.rst +++ b/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_2n-skx_testbeds.rst @@ -1,8 +1,8 @@ .. _vpp_compare_topologies_3n-Skx_vs_2n-Skx: -3n-Skx vs. 2n-Skx Testbeds --------------------------- +3n-Skx vs 2n-Skx Testbeds +------------------------- Relative comparison of |vpp-release| packet throughput (NDR, PDR and MRR) is calculated for the same tests executed on 3-Node Skylake (3n- diff --git a/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst b/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst index dd7cb03d43..003f92cea8 100644 --- a/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst +++ b/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst @@ -1,5 +1,5 @@ -3n-Skx vs. 3n-Hsw Testbeds --------------------------- +3n-Skx vs 3n-Hsw Testbeds +------------------------- Relative comparison of |vpp-release| packet throughput (NDR, PDR and MRR) is calculated for the same tests executed on 3-Node Skylake (3n- diff --git a/docs/report/vpp_performance_tests/comparisons/current_vs_previous_release.rst b/docs/report/vpp_performance_tests/comparisons/current_vs_previous_release.rst index 50c4710d87..a65e5d164c 100644 --- a/docs/report/vpp_performance_tests/comparisons/current_vs_previous_release.rst +++ b/docs/report/vpp_performance_tests/comparisons/current_vs_previous_release.rst @@ -1,8 +1,8 @@ .. _vpp_compare_current_vs_previous_release: -Current vs. Previous Release ----------------------------- +Current vs Previous Release +--------------------------- Relative comparison of VPP packet throughput (NDR, PDR and MRR) between |vpp-release| and |vpp-release-1| (measured for |csit-release| and diff --git a/docs/report/vpp_performance_tests/comparisons/soak_vs_ndr.rst b/docs/report/vpp_performance_tests/comparisons/soak_vs_ndr.rst index c219ffd5fe..b4de50b286 100644 --- a/docs/report/vpp_performance_tests/comparisons/soak_vs_ndr.rst +++ b/docs/report/vpp_performance_tests/comparisons/soak_vs_ndr.rst @@ -1,9 +1,9 @@ .. _`soak vs ndr comparison`: -Soak Tests vs. NDR Tests ------------------------- +Soak Tests vs NDR Tests +----------------------- -Relative comparison of |vpp-release| Soak PLRSearch vs. NDR packet throughput +Relative comparison of |vpp-release| Soak PLRSearch vs NDR packet throughput is calculated for the tests executed on 2-Node Skylake physical testbed types, in 1-core configurations. @@ -20,11 +20,11 @@ Comparison tables in ASCII and CSV formats: 2n-skx ~~~~~~ - - `ASCII Soak vs. NDR comparison <../../_static/vpp/soak-vs-ndr-2n-skx.txt>`_ - - `CSV Soak vs. NDR comparison <../../_static/vpp/soak-vs-ndr-2n-skx.csv>`_ + - `ASCII Soak vs NDR comparison <../../_static/vpp/soak-vs-ndr-2n-skx.txt>`_ + - `CSV Soak vs NDR comparison <../../_static/vpp/soak-vs-ndr-2n-skx.csv>`_ 2n-clx ~~~~~~ - - `ASCII Soak vs. NDR comparison <../../_static/vpp/soak-vs-ndr-2n-clx.txt>`_ - - `CSV Soak vs. NDR comparison <../../_static/vpp/soak-vs-ndr-2n-clx.csv>`_ + - `ASCII Soak vs NDR comparison <../../_static/vpp/soak-vs-ndr-2n-clx.txt>`_ + - `CSV Soak vs NDR comparison <../../_static/vpp/soak-vs-ndr-2n-clx.csv>`_ diff --git a/resources/tools/presentation/specification.yaml b/resources/tools/presentation/specification.yaml index a6ff3b0547..1772b7cf3d 100644 --- a/resources/tools/presentation/specification.yaml +++ b/resources/tools/presentation/specification.yaml @@ -811,9 +811,9 @@ - 41 # NDRPDR sel vpp-performance-changes-3n-skx-cmp-2: csit-vpp-perf-report-iterative-2005-3n-skx: - # TODO: Add data - 42 # NDRPDR sel env 4 - 43 # NDRPDR sel env 4 + - 45 # NDRPDR sel env 4 vpp-performance-changes-2n-skx-h1: csit-vpp-perf-verify-1908_1-2n-skx: @@ -853,7 +853,6 @@ - 73 # NDRPDR sel vpp-performance-changes-2n-skx-cmp-2: csit-vpp-perf-report-iterative-2005-2n-skx: - # TODO: Add data - 88 # NDRPDR sel env 4 - 89 # NDRPDR sel env 4 - 90 # NDRPDR sel env 4 @@ -1013,7 +1012,6 @@ - 14 # NDRPDR sel - 21 # NDRPDR sel vpp-performance-changes-3n-tsh-cmp-2: - # TODO: Add data: csit-vpp-perf-report-iterative-2005-3n-tsh: - 23 # NDRPDR sel env 4 @@ -2272,6 +2270,7 @@ - 41 # NDRPDR sel - 42 # NDRPDR sel env 4 - 43 # NDRPDR sel env 4 + - 45 # NDRPDR sel env 4 - 6 # MRR sel csit-vpp-perf-report-coverage-2005-3n-skx: - 10 # perf-full-3n-skx-crypto_I @@ -3196,10 +3195,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-3n-hsw-cmp-2" comparisons: - - title: "Diff1" - reference: 4 - compare: 3 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 3 + compare: 4 + - title: "2005.0-4 vs 2001.0-2" reference: 2 compare: 4 # rca: @@ -3208,12 +3207,14 @@ # - title: "RCA2" # data: "rca/rca-2005-3n-hsw-1t1c-pdr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: PDR throughput on 3n-hsw with xl710.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 3n-hsw 2t2c pdr - type: "table" @@ -3232,10 +3233,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-3n-hsw-cmp-2" comparisons: - - title: "Diff1" - reference: 4 - compare: 3 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 3 + compare: 4 + - title: "2005.0-4 vs 2001.0-2" reference: 2 compare: 4 # rca: @@ -3244,12 +3245,14 @@ # - title: "RCA2" # data: "rca/rca-2005-3n-hsw-2t2c-pdr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: PDR throughput on 3n-hsw with xl710.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 3n-hsw 1t1c ndr - type: "table" @@ -3268,11 +3271,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-3n-hsw-cmp-2" comparisons: - - title: "Diff1" - - title: "Diff1" - reference: 4 - compare: 3 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 3 + compare: 4 + - title: "2005.0-4 vs 2001.0-2" reference: 2 compare: 4 # rca: @@ -3281,12 +3283,14 @@ # - title: "RCA2" # data: "rca/rca-2005-3n-hsw-1t1c-ndr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: NDR throughput on 3n-hsw with xl710.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 3n-hsw 2t2c ndr - type: "table" @@ -3305,10 +3309,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-3n-hsw-cmp-2" comparisons: - - title: "Diff1" - reference: 4 - compare: 3 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 3 + compare: 4 + - title: "2005.0-4 vs 2001.0-2" reference: 2 compare: 4 # rca: @@ -3317,12 +3321,14 @@ # - title: "RCA2" # data: "rca/rca-2005-3n-hsw-2t2c-ndr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: NDR throughput on 3n-hsw with xl710.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 3n-skx 2t1c pdr - type: "table" @@ -3341,10 +3347,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-3n-skx-cmp-2" comparisons: - - title: "Diff1" - reference: 4 - compare: 3 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 3 + compare: 4 + - title: "2005.0-4 vs 2001.0-2" reference: 2 compare: 4 # rca: @@ -3353,12 +3359,14 @@ # - title: "RCA2" # data: "rca/rca-2005-3n-skx-2t1c-pdr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: PDR throughput on 3n-skx with xxv710.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 3n-skx 4t2c pdr - type: "table" @@ -3377,10 +3385,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-3n-skx-cmp-2" comparisons: - - title: "Diff1" - reference: 4 - compare: 3 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 3 + compare: 4 + - title: "2005.0-4 vs 2001.0-2" reference: 2 compare: 4 # rca: @@ -3389,12 +3397,14 @@ # - title: "RCA2" # data: "rca/rca-2005-3n-skx-4t2c-pdr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: PDR throughput on 3n-skx with xxv710.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 3n-skx 2t1c ndr - type: "table" @@ -3413,10 +3423,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-3n-skx-cmp-2" comparisons: - - title: "Diff1" - reference: 4 - compare: 3 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 3 + compare: 4 + - title: "2005.0-4 vs 2001.0-2" reference: 2 compare: 4 # rca: @@ -3425,12 +3435,14 @@ # - title: "RCA2" # data: "rca/rca-2005-3n-skx-2t1c-ndr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: NDR throughput on 3n-skx with xxv710.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 3n-skx 4t2c ndr - type: "table" @@ -3449,10 +3461,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-3n-skx-cmp-2" comparisons: - - title: "Diff1" - reference: 4 - compare: 3 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 3 + compare: 4 + - title: "2005.0-4 vs 2001.0-2" reference: 2 compare: 4 # rca: @@ -3461,12 +3473,14 @@ # - title: "RCA2" # data: "rca/rca-2005-3n-skx-4t2c-ndr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: NDR throughput on 3n-skx with xxv710.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 2n-skx 2t1c pdr - type: "table" @@ -3485,10 +3499,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-2n-skx-cmp-2" comparisons: - - title: "Diff1" - reference: 4 - compare: 3 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 3 + compare: 4 + - title: "2005.0-4 vs 2001.0-2" reference: 2 compare: 4 # rca: @@ -3497,12 +3511,14 @@ # - title: "RCA2" # data: "rca/rca-2005-2n-skx-2t1c-pdr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: PDR throughput on 2n-skx with xxv710.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 2n-skx 4t2c pdr - type: "table" @@ -3521,10 +3537,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-2n-skx-cmp-2" comparisons: - - title: "Diff1" - reference: 4 - compare: 3 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 3 + compare: 4 + - title: "2005.0-4 vs 2001.0-2" reference: 2 compare: 4 # rca: @@ -3533,12 +3549,14 @@ # - title: "RCA2" # data: "rca/rca-2005-2n-skx-4t2c-pdr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: PDR throughput on 2n-skx with xxv710.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 2n-skx 2t1c ndr - type: "table" @@ -3557,10 +3575,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-2n-skx-cmp-2" comparisons: - - title: "Diff1" - reference: 4 - compare: 3 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 3 + compare: 4 + - title: "2005.0-4 vs 2001.0-2" reference: 2 compare: 4 # rca: @@ -3569,12 +3587,14 @@ # - title: "RCA2" # data: "rca/rca-2005-2n-skx-2t1c-ndr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: NDR throughput on 2n-skx with xxv710.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 2n-skx 4t2c ndr - type: "table" @@ -3593,10 +3613,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-2n-skx-cmp-2" comparisons: - - title: "Diff1" - reference: 4 - compare: 3 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 3 + compare: 4 + - title: "2005.0-4 vs 2001.0-2" reference: 2 compare: 4 # rca: @@ -3605,12 +3625,14 @@ # - title: "RCA2" # data: "rca/rca-2005-2n-skx-4t2c-ndr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: NDR throughput on 2n-skx with xxv710.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 2n-clx-xxv710 2t1c pdr - type: "table" @@ -3627,10 +3649,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-2n-clx-cmp-2" comparisons: - - title: "Diff1" - reference: 3 - compare: 2 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 2 + compare: 3 + - title: "2005.0-4 vs 2001.0-2" reference: 1 compare: 3 # rca: @@ -3639,12 +3661,14 @@ # - title: "RCA2" # data: "rca/rca-2005-2n-clx-2t1c-pdr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: PDR throughput on 2n-clx with xxv710.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 2n-clx-xxv710 4t2c pdr - type: "table" @@ -3661,10 +3685,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-2n-clx-cmp-2" comparisons: - - title: "Diff1" - reference: 3 - compare: 2 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 2 + compare: 3 + - title: "2005.0-4 vs 2001.0-2" reference: 1 compare: 3 # rca: @@ -3673,12 +3697,14 @@ # - title: "RCA2" # data: "rca/rca-2005-2n-clx-4t2c-pdr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: PDR throughput on 2n-clx with xxv710.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 2n-clx-xxv710 2t1c ndr - type: "table" @@ -3695,10 +3721,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-2n-clx-cmp-2" comparisons: - - title: "Diff1" - reference: 3 - compare: 2 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 2 + compare: 3 + - title: "2005.0-4 vs 2001.0-2" reference: 1 compare: 3 # rca: @@ -3707,12 +3733,14 @@ # - title: "RCA2" # data: "rca/rca-2005-2n-clx-2t1c-ndr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: NDR throughput on 2n-clx with xxv710.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 2n-clx-xxv710 4t2c ndr - type: "table" @@ -3729,10 +3757,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-2n-clx-cmp-2" comparisons: - - title: "Diff1" - reference: 3 - compare: 2 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 2 + compare: 3 + - title: "2005.0-4 vs 2001.0-2" reference: 1 compare: 3 # rca: @@ -3741,12 +3769,14 @@ # - title: "RCA2" # data: "rca/rca-2005-2n-clx-4t2c-ndr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: NDR throughput on 2n-clx with xxv710.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 2n-clx-x556 2t1c pdr - type: "table" @@ -4033,10 +4063,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-3n-tsh-cmp-2" comparisons: - - title: "Diff1" - reference: 4 - compare: 3 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 3 + compare: 4 + - title: "2005.0-4 vs 2001.0-2" reference: 2 compare: 4 # rca: @@ -4045,12 +4075,14 @@ # - title: "RCA2" # data: "rca/rca-2005-3n-tsh-1t1c-pdr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: PDR throughput on 3n-tsh with x520.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 3n-tsh 2t2c pdr - type: "table" @@ -4069,10 +4101,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-3n-tsh-cmp-2" comparisons: - - title: "Diff1" - reference: 4 - compare: 3 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 3 + compare: 4 + - title: "2005.0-4 vs 2001.0-2" reference: 2 compare: 4 # rca: @@ -4081,12 +4113,14 @@ # - title: "RCA2" # data: "rca/rca-2005-3n-tsh-2t2c-pdr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: PDR throughput on 3n-tsh with x520.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 3n-tsh 1t1c ndr - type: "table" @@ -4105,10 +4139,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-3n-tsh-cmp-2" comparisons: - - title: "Diff1" - reference: 4 - compare: 3 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 3 + compare: 4 + - title: "2005.0-4 vs 2001.0-2" reference: 2 compare: 4 # rca: @@ -4117,12 +4151,14 @@ # - title: "RCA2" # data: "rca/rca-2005-3n-tsh-1t1c-ndr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: NDR throughput on 3n-tsh with x520.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Performance Changes 3n-tsh 2t2c ndr - type: "table" @@ -4141,10 +4177,10 @@ - title: "2005.0-4" data-set: "vpp-performance-changes-3n-tsh-cmp-2" comparisons: - - title: "Diff1" - reference: 4 - compare: 3 - - title: "Diff2" + - title: "2005.0-4 vs 2005.0-3" + reference: 3 + compare: 4 + - title: "2005.0-4 vs 2001.0-2" reference: 2 compare: 4 # rca: @@ -4153,12 +4189,14 @@ # - title: "RCA2" # data: "rca/rca-2005-3n-tsh-2t2c-ndr-rca2.yaml" legend: - - "Legend to Table" - - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' - - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.' - - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.' -# - 'RCA1 - root cause analysis reference for Diff1.' -# - 'RCA2 - root cause analysis reference for Diff2.' + - 'Legend to Table:' + - '' + - ' Test Case: Test cases executed for: NDR throughput on 3n-tsh with x520.' + - ' YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).' + - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.' + - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.' +# - ' RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.' +# - ' RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.' # VPP Comparison Across Testbeds 3n-hsw to 3n-skx 1c ndr - type: "table" |