diff options
| -rw-r--r-- | docs/report/introduction/methodology.rst | 4 | ||||
| -rw-r--r-- | docs/report/introduction/methodology_data_plane_throughput.rst | 97 | ||||
| -rw-r--r-- | docs/report/introduction/methodology_mlrsearch_tests.rst | 1 | ||||
| -rw-r--r-- | docs/report/introduction/methodology_mrr_throughput.rst (renamed from docs/report/introduction/methodology_bmrr_throughput.rst) | 11 | ||||
| -rw-r--r-- | docs/report/introduction/methodology_plrsearch.rst | 2 |
5 files changed, 84 insertions, 31 deletions
diff --git a/docs/report/introduction/methodology.rst b/docs/report/introduction/methodology.rst index e9d87075b9..45cd6bc598 100644 --- a/docs/report/introduction/methodology.rst +++ b/docs/report/introduction/methodology.rst @@ -11,7 +11,8 @@ Test Methodology methodology_vpp_features methodology_data_plane_throughput methodology_mlrsearch_tests - methodology_bmrr_throughput + methodology_mrr_throughput + methodology_plrsearch methodology_packet_latency methodology_multi_core_speedup methodology_vpp_startup_settings @@ -22,7 +23,6 @@ Test Methodology methodology_vpp_device_functional methodology_ipsec_on_intel_qat methodology_trex_traffic_generator - methodology_plrsearch .. methodology_http_tcp_with_wrk_tool diff --git a/docs/report/introduction/methodology_data_plane_throughput.rst b/docs/report/introduction/methodology_data_plane_throughput.rst index 10d85e991c..2b5b085452 100644 --- a/docs/report/introduction/methodology_data_plane_throughput.rst +++ b/docs/report/introduction/methodology_data_plane_throughput.rst @@ -1,33 +1,86 @@ Data Plane Throughput --------------------- -Network data plane packet and bandwidth throughput are measured in -accordance with :rfc:`2544`, using FD.io CSIT Multiple Loss Ratio search -(MLRsearch), an optimized throughput search algorithm, that measures -SUT/DUT packet throughput rates at different Packet Loss Ratio (PLR) -values. +Network data plane packet and bandwidth throughput are measured using +multiple methods in order to obtain representative and repeatable +results across the large set of performance test cases implemented and +executed within CSIT. Following throughput test methods are used: -Following MLRsearch values are measured across a range of L2 frame sizes -and reported: +#. MLRsearch: Multiple Loss Ratio search -- NON DROP RATE (NDR): packet and bandwidth throughput at PLR=0%. + - **Description**: MLRsearch discovers multiple packet throughput + rates in a single search, reducing the overall test execution + time compared to a binary search. Each rate associated with a + distinct Packet Loss Ratio (PLR) criteria. In FD.io CSIT two + throughput rates are discovered: Non-Drop Rate (NDR, with zero + packet loss, PLR=0) and Partial Drop Rate (PDR, with PLR<0.5%). + MLRsearch is compliant with :rfc:`2544`. + - **Usage**: MLRsearch tests are run to discover NDR and PDR rates + for each VPP and DPDK release covered by CSIT report. Results for + small frame sizes (64b/78B, IMIX) are presented in packet + throughput graphs (Box-and-Whisker Plots) with NDR and PDR rates + plotted against the test cases covering popular VPP packet paths. + Each test is executed at least 10 times to verify measurements + repeatability and results are compared between releases and test + environments. NDR and PDR packet and bandwidth throughput results + for all frame sizes and for all tests are presented in detailed + results tables. + - **References**: See :ref:`mlrsearch_algorithm` for more detailed + description of MLRsearch tests. MLRsearch is being standardized + in IETF with `draft-vpolak-mkonstan-mlrsearch + <https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch>`_. - - **Aggregate packet rate**: NDR_LOWER <bi-directional packet rate> - pps. - - **Aggregate bandwidth rate**: NDR_LOWER <bi-directional bandwidth - rate> Gbps. +#. MRR Measurements: Maximum Receive Rate -- PARTIAL DROP RATE (PDR): packet and bandwidth throughput at PLR=0.5%. + - **Description**: MRR tests are complementary to MLRsearch tests, + as they provide a maximum “raw” throughput benchmark for + development and testing community. MRR tests measure the packet + forwarding rate under the maximum load offered by traffic + generator (dependent on link type and NIC model) over a set trial + duration, regardless of packet loss. Maximum load for specified + Ethernet frame size is set to the bi-directional link rate. + - **Usage**: MRR tests are much faster than MLRsearch as they rely + on a single trial or a small set of trials with very short + duration. It is this property that makes them suitable for + continuous execution in daily performance trending jobs enabling + detection of performance anomalies (regressions, progressions) + resulting from data plane code changes. MRR tests are also used + for VPP per patch performance jobs verifying patch performance + vs. parent. CSIT reports include MRR throughput comparisons + between releases and test environments. Small frame sizes only + (64b/78B, IMIX). + - **References**: See :ref:`mrr_throughput` for more detailed + description of MRR tests configuration used for daily performance + trending jobs. VPP per patch test methodology is available on + `FD.io CSIT trending pages + <https://docs.fd.io/csit/master/trending/methodology/perpatch_performance_tests.html>`_. - - **Aggregate packet rate**: PDR_LOWER <bi-directional packet rate> - pps. - - **Aggregate bandwidth rate**: PDR_LOWER <bi-directional bandwidth - rate> Gbps. +#. PLRsearch: Probabilistic Loss Ratio search -NDR and PDR are measured for the following L2 frame sizes (untagged -Ethernet): + - **Description**: PLRsearch discovers a packet throughput rate + associated with configured Packet Loss Ratio (PLR) criteria for + tests run over an extended period of time a.k.a. soak testing. + PLRsearch assumes that system under test is probabilistic in + nature, and not deterministic. + - **Usage**: PLRsearch are run to discover a sustained throughput + for PLR=10^-7 (close to NDR) for VPP release covered by CSIT + report. Results for small frame sizes (64b/78B) are presented in + packet throughput graphs (Box Plots) for a small subset of + baseline tests. Each soak test lasts 2hrs and is executed at + least twice. Results are compared against NDR and PDR rates + discovered with MLRsearch. + - **References**: See :ref:`plrsearch_algorithm` for more detailed + description of PLRsearch tests. PLRsearch is being standardized + in IETF with `draft-vpolak-bmwg-plrsearch + <https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch>`_. -- IPv4 payload: 64B, IMIX (28x64B, 16x570B, 4x1518B), 1518B, 9000B. -- IPv6 payload: 78B, IMIX (28x78B, 16x570B, 4x1518B), 1518B, 9000B. +All of the listed data plane throughput test methodologies share +following properties: -All rates are reported from external Traffic Generator perspective. +- Tested L2 frame sizes (untagged Ethernet): + + - IPv4 payload: 64B, IMIX (28x64B, 16x570B, 4x1518B), 1518B, 9000B. + - IPv6 payload: 78B, IMIX (28x78B, 16x570B, 4x1518B), 1518B, 9000B. + +- All measured rates are aggregate bi-directional rates reported from + external Traffic Generator perspective.
\ No newline at end of file diff --git a/docs/report/introduction/methodology_mlrsearch_tests.rst b/docs/report/introduction/methodology_mlrsearch_tests.rst index 15b9ee0290..6c2adfcd06 100644 --- a/docs/report/introduction/methodology_mlrsearch_tests.rst +++ b/docs/report/introduction/methodology_mlrsearch_tests.rst @@ -1,4 +1,3 @@ - .. _mlrsearch_algorithm: MLRsearch Tests diff --git a/docs/report/introduction/methodology_bmrr_throughput.rst b/docs/report/introduction/methodology_mrr_throughput.rst index 7bef3a4aaf..824ac887f7 100644 --- a/docs/report/introduction/methodology_bmrr_throughput.rst +++ b/docs/report/introduction/methodology_mrr_throughput.rst @@ -1,5 +1,7 @@ -(B)MRR Throughput ------------------ +.. _mrr_throughput: + +MRR Throughput +-------------- Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests, as they provide a maximum "raw" throughput benchmark for development and @@ -10,10 +12,9 @@ size is set to the bi-directional link rate. In |csit-release| MRR test code has been updated with a configurable burst MRR parameters: trial duration and number of trials in a single -burst. This enabled a new Burst MRR (BMRR) methodology for more precise -performance trending. +burst. This enabled more precise performance trending. -Current parameters for BMRR tests: +Current parameters for MRR trending tests: - Ethernet frame sizes: 64B (78B for IPv6), IMIX, 1518B, 9000B; all quoted sizes include frame CRC, but exclude per frame transmission diff --git a/docs/report/introduction/methodology_plrsearch.rst b/docs/report/introduction/methodology_plrsearch.rst index 65ac590222..fdd03472c3 100644 --- a/docs/report/introduction/methodology_plrsearch.rst +++ b/docs/report/introduction/methodology_plrsearch.rst @@ -1,4 +1,4 @@ -.. _`PLRsearch algorithm`: +.. _plrsearch_algorithm: PLRsearch ^^^^^^^^^ |