From cc7a7eea385534107bfdae744797d68782488a5e Mon Sep 17 00:00:00 2001 From: Maciek Konstantynowicz Date: Wed, 8 May 2019 12:13:42 +0100 Subject: report: edits of methodology throughput sections Change-Id: I9ed1c2d67bc53f98d36ad17afb9153de5e944b7f Signed-off-by: Maciek Konstantynowicz --- docs/report/introduction/methodology.rst | 4 +- .../introduction/methodology_bmrr_throughput.rst | 54 ------------ .../methodology_data_plane_throughput.rst | 97 +++++++++++++++++----- .../introduction/methodology_mlrsearch_tests.rst | 1 - .../introduction/methodology_mrr_throughput.rst | 55 ++++++++++++ docs/report/introduction/methodology_plrsearch.rst | 2 +- 6 files changed, 133 insertions(+), 80 deletions(-) delete mode 100644 docs/report/introduction/methodology_bmrr_throughput.rst create mode 100644 docs/report/introduction/methodology_mrr_throughput.rst (limited to 'docs') diff --git a/docs/report/introduction/methodology.rst b/docs/report/introduction/methodology.rst index e9d87075b9..45cd6bc598 100644 --- a/docs/report/introduction/methodology.rst +++ b/docs/report/introduction/methodology.rst @@ -11,7 +11,8 @@ Test Methodology methodology_vpp_features methodology_data_plane_throughput methodology_mlrsearch_tests - methodology_bmrr_throughput + methodology_mrr_throughput + methodology_plrsearch methodology_packet_latency methodology_multi_core_speedup methodology_vpp_startup_settings @@ -22,7 +23,6 @@ Test Methodology methodology_vpp_device_functional methodology_ipsec_on_intel_qat methodology_trex_traffic_generator - methodology_plrsearch .. methodology_http_tcp_with_wrk_tool diff --git a/docs/report/introduction/methodology_bmrr_throughput.rst b/docs/report/introduction/methodology_bmrr_throughput.rst deleted file mode 100644 index 7bef3a4aaf..0000000000 --- a/docs/report/introduction/methodology_bmrr_throughput.rst +++ /dev/null @@ -1,54 +0,0 @@ -(B)MRR Throughput ------------------ - -Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests, -as they provide a maximum "raw" throughput benchmark for development and -testing community. MRR tests measure the packet forwarding rate under -the maximum load offered by traffic generator over a set trial duration, -regardless of packet loss. Maximum load for specified Ethernet frame -size is set to the bi-directional link rate. - -In |csit-release| MRR test code has been updated with a configurable -burst MRR parameters: trial duration and number of trials in a single -burst. This enabled a new Burst MRR (BMRR) methodology for more precise -performance trending. - -Current parameters for BMRR tests: - -- Ethernet frame sizes: 64B (78B for IPv6), IMIX, 1518B, 9000B; all - quoted sizes include frame CRC, but exclude per frame transmission - overhead of 20B (preamble, inter frame gap). - -- Maximum load offered: 10GE, 25GE and 40GE link (sub-)rates depending on NIC - tested, with the actual packet rate depending on frame size, - transmission overhead and traffic generator NIC forwarding capacity. - - - For 10GE NICs the maximum packet rate load is 2* 14.88 Mpps for 64B, - a 10GE bi-directional link rate. - - For 25GE NICs the maximum packet rate load is 2* 18.75 Mpps for 64B, - a 25GE bi-directional link sub-rate limited by TG 25GE NIC used, - XXV710. - - For 40GE NICs the maximum packet rate load is 2* 18.75 Mpps for 64B, - a 40GE bi-directional link sub-rate limited by TG 40GE NIC used, - XL710. Packet rate for other tested frame sizes is limited by PCIe - Gen3 x8 bandwidth limitation of ~50Gbps. - -- Trial duration: 1 sec. - -- Number of trials per burst: 10. - -Similarly to NDR/PDR throughput tests, MRR test should be reporting -bi-directional link rate (or NIC rate, if lower) if tested VPP -configuration can handle the packet rate higher than bi-directional link -rate, e.g. large packet tests and/or multi-core tests. - -MRR tests are currently used for FD.io CSIT continuous performance -trending and for comparison between releases. Daily trending job tests -subset of frame sizes, focusing on 64B (78B for IPv6) for all tests and -IMIX for selected tests (vhost, memif). - -MRR-like measurements are being used to establish starting conditions -for experimental Probabilistic Loss Ratio Search (PLRsearch) used for -soak testing, aimed at verifying continuous system performance over an -extended period of time, hours, days, weeks, months. PLRsearch code is -currently in experimental phase in FD.io CSIT project. diff --git a/docs/report/introduction/methodology_data_plane_throughput.rst b/docs/report/introduction/methodology_data_plane_throughput.rst index 10d85e991c..2b5b085452 100644 --- a/docs/report/introduction/methodology_data_plane_throughput.rst +++ b/docs/report/introduction/methodology_data_plane_throughput.rst @@ -1,33 +1,86 @@ Data Plane Throughput --------------------- -Network data plane packet and bandwidth throughput are measured in -accordance with :rfc:`2544`, using FD.io CSIT Multiple Loss Ratio search -(MLRsearch), an optimized throughput search algorithm, that measures -SUT/DUT packet throughput rates at different Packet Loss Ratio (PLR) -values. +Network data plane packet and bandwidth throughput are measured using +multiple methods in order to obtain representative and repeatable +results across the large set of performance test cases implemented and +executed within CSIT. Following throughput test methods are used: -Following MLRsearch values are measured across a range of L2 frame sizes -and reported: +#. MLRsearch: Multiple Loss Ratio search -- NON DROP RATE (NDR): packet and bandwidth throughput at PLR=0%. + - **Description**: MLRsearch discovers multiple packet throughput + rates in a single search, reducing the overall test execution + time compared to a binary search. Each rate associated with a + distinct Packet Loss Ratio (PLR) criteria. In FD.io CSIT two + throughput rates are discovered: Non-Drop Rate (NDR, with zero + packet loss, PLR=0) and Partial Drop Rate (PDR, with PLR<0.5%). + MLRsearch is compliant with :rfc:`2544`. + - **Usage**: MLRsearch tests are run to discover NDR and PDR rates + for each VPP and DPDK release covered by CSIT report. Results for + small frame sizes (64b/78B, IMIX) are presented in packet + throughput graphs (Box-and-Whisker Plots) with NDR and PDR rates + plotted against the test cases covering popular VPP packet paths. + Each test is executed at least 10 times to verify measurements + repeatability and results are compared between releases and test + environments. NDR and PDR packet and bandwidth throughput results + for all frame sizes and for all tests are presented in detailed + results tables. + - **References**: See :ref:`mlrsearch_algorithm` for more detailed + description of MLRsearch tests. MLRsearch is being standardized + in IETF with `draft-vpolak-mkonstan-mlrsearch + `_. - - **Aggregate packet rate**: NDR_LOWER - pps. - - **Aggregate bandwidth rate**: NDR_LOWER Gbps. +#. MRR Measurements: Maximum Receive Rate -- PARTIAL DROP RATE (PDR): packet and bandwidth throughput at PLR=0.5%. + - **Description**: MRR tests are complementary to MLRsearch tests, + as they provide a maximum “raw” throughput benchmark for + development and testing community. MRR tests measure the packet + forwarding rate under the maximum load offered by traffic + generator (dependent on link type and NIC model) over a set trial + duration, regardless of packet loss. Maximum load for specified + Ethernet frame size is set to the bi-directional link rate. + - **Usage**: MRR tests are much faster than MLRsearch as they rely + on a single trial or a small set of trials with very short + duration. It is this property that makes them suitable for + continuous execution in daily performance trending jobs enabling + detection of performance anomalies (regressions, progressions) + resulting from data plane code changes. MRR tests are also used + for VPP per patch performance jobs verifying patch performance + vs. parent. CSIT reports include MRR throughput comparisons + between releases and test environments. Small frame sizes only + (64b/78B, IMIX). + - **References**: See :ref:`mrr_throughput` for more detailed + description of MRR tests configuration used for daily performance + trending jobs. VPP per patch test methodology is available on + `FD.io CSIT trending pages + `_. - - **Aggregate packet rate**: PDR_LOWER - pps. - - **Aggregate bandwidth rate**: PDR_LOWER Gbps. +#. PLRsearch: Probabilistic Loss Ratio search -NDR and PDR are measured for the following L2 frame sizes (untagged -Ethernet): + - **Description**: PLRsearch discovers a packet throughput rate + associated with configured Packet Loss Ratio (PLR) criteria for + tests run over an extended period of time a.k.a. soak testing. + PLRsearch assumes that system under test is probabilistic in + nature, and not deterministic. + - **Usage**: PLRsearch are run to discover a sustained throughput + for PLR=10^-7 (close to NDR) for VPP release covered by CSIT + report. Results for small frame sizes (64b/78B) are presented in + packet throughput graphs (Box Plots) for a small subset of + baseline tests. Each soak test lasts 2hrs and is executed at + least twice. Results are compared against NDR and PDR rates + discovered with MLRsearch. + - **References**: See :ref:`plrsearch_algorithm` for more detailed + description of PLRsearch tests. PLRsearch is being standardized + in IETF with `draft-vpolak-bmwg-plrsearch + `_. -- IPv4 payload: 64B, IMIX (28x64B, 16x570B, 4x1518B), 1518B, 9000B. -- IPv6 payload: 78B, IMIX (28x78B, 16x570B, 4x1518B), 1518B, 9000B. +All of the listed data plane throughput test methodologies share +following properties: -All rates are reported from external Traffic Generator perspective. +- Tested L2 frame sizes (untagged Ethernet): + + - IPv4 payload: 64B, IMIX (28x64B, 16x570B, 4x1518B), 1518B, 9000B. + - IPv6 payload: 78B, IMIX (28x78B, 16x570B, 4x1518B), 1518B, 9000B. + +- All measured rates are aggregate bi-directional rates reported from + external Traffic Generator perspective. \ No newline at end of file diff --git a/docs/report/introduction/methodology_mlrsearch_tests.rst b/docs/report/introduction/methodology_mlrsearch_tests.rst index 15b9ee0290..6c2adfcd06 100644 --- a/docs/report/introduction/methodology_mlrsearch_tests.rst +++ b/docs/report/introduction/methodology_mlrsearch_tests.rst @@ -1,4 +1,3 @@ - .. _mlrsearch_algorithm: MLRsearch Tests diff --git a/docs/report/introduction/methodology_mrr_throughput.rst b/docs/report/introduction/methodology_mrr_throughput.rst new file mode 100644 index 0000000000..824ac887f7 --- /dev/null +++ b/docs/report/introduction/methodology_mrr_throughput.rst @@ -0,0 +1,55 @@ +.. _mrr_throughput: + +MRR Throughput +-------------- + +Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests, +as they provide a maximum "raw" throughput benchmark for development and +testing community. MRR tests measure the packet forwarding rate under +the maximum load offered by traffic generator over a set trial duration, +regardless of packet loss. Maximum load for specified Ethernet frame +size is set to the bi-directional link rate. + +In |csit-release| MRR test code has been updated with a configurable +burst MRR parameters: trial duration and number of trials in a single +burst. This enabled more precise performance trending. + +Current parameters for MRR trending tests: + +- Ethernet frame sizes: 64B (78B for IPv6), IMIX, 1518B, 9000B; all + quoted sizes include frame CRC, but exclude per frame transmission + overhead of 20B (preamble, inter frame gap). + +- Maximum load offered: 10GE, 25GE and 40GE link (sub-)rates depending on NIC + tested, with the actual packet rate depending on frame size, + transmission overhead and traffic generator NIC forwarding capacity. + + - For 10GE NICs the maximum packet rate load is 2* 14.88 Mpps for 64B, + a 10GE bi-directional link rate. + - For 25GE NICs the maximum packet rate load is 2* 18.75 Mpps for 64B, + a 25GE bi-directional link sub-rate limited by TG 25GE NIC used, + XXV710. + - For 40GE NICs the maximum packet rate load is 2* 18.75 Mpps for 64B, + a 40GE bi-directional link sub-rate limited by TG 40GE NIC used, + XL710. Packet rate for other tested frame sizes is limited by PCIe + Gen3 x8 bandwidth limitation of ~50Gbps. + +- Trial duration: 1 sec. + +- Number of trials per burst: 10. + +Similarly to NDR/PDR throughput tests, MRR test should be reporting +bi-directional link rate (or NIC rate, if lower) if tested VPP +configuration can handle the packet rate higher than bi-directional link +rate, e.g. large packet tests and/or multi-core tests. + +MRR tests are currently used for FD.io CSIT continuous performance +trending and for comparison between releases. Daily trending job tests +subset of frame sizes, focusing on 64B (78B for IPv6) for all tests and +IMIX for selected tests (vhost, memif). + +MRR-like measurements are being used to establish starting conditions +for experimental Probabilistic Loss Ratio Search (PLRsearch) used for +soak testing, aimed at verifying continuous system performance over an +extended period of time, hours, days, weeks, months. PLRsearch code is +currently in experimental phase in FD.io CSIT project. diff --git a/docs/report/introduction/methodology_plrsearch.rst b/docs/report/introduction/methodology_plrsearch.rst index 65ac590222..fdd03472c3 100644 --- a/docs/report/introduction/methodology_plrsearch.rst +++ b/docs/report/introduction/methodology_plrsearch.rst @@ -1,4 +1,4 @@ -.. _`PLRsearch algorithm`: +.. _plrsearch_algorithm: PLRsearch ^^^^^^^^^ -- cgit 1.2.3-korg