diff options
| -rw-r--r-- | docs/report/introduction/methodology.rst | 2 | ||||
| -rw-r--r-- | docs/report/introduction/methodology_data_plane_throughput.rst | 86 | ||||
| -rw-r--r-- | docs/report/introduction/methodology_data_plane_throughput/index.rst | 10 | ||||
| -rw-r--r-- | docs/report/introduction/methodology_data_plane_throughput/methodology_data_plane_throughput.rst | 137 | ||||
| -rw-r--r-- | docs/report/introduction/methodology_data_plane_throughput/methodology_mlrsearch_tests.rst (renamed from docs/report/introduction/methodology_mlrsearch_tests.rst) | 0 | ||||
| -rw-r--r-- | docs/report/introduction/methodology_data_plane_throughput/methodology_mrr_throughput.rst | 52 | ||||
| -rw-r--r-- | docs/report/introduction/methodology_data_plane_throughput/methodology_plrsearch.rst (renamed from docs/report/introduction/methodology_plrsearch.rst) | 0 | ||||
| -rw-r--r-- | docs/report/introduction/methodology_mrr_throughput.rst | 55 |
8 files changed, 200 insertions, 142 deletions
diff --git a/docs/report/introduction/methodology.rst b/docs/report/introduction/methodology.rst index 37ff946ff7..506fa73b5b 100644 --- a/docs/report/introduction/methodology.rst +++ b/docs/report/introduction/methodology.rst @@ -9,7 +9,7 @@ Test Methodology methodology_vpp_forwarding_modes methodology_tunnel_encapsulations methodology_vpp_features - methodology_data_plane_throughput + methodology_data_plane_throughput/index methodology_mlrsearch_tests methodology_mrr_throughput methodology_plrsearch diff --git a/docs/report/introduction/methodology_data_plane_throughput.rst b/docs/report/introduction/methodology_data_plane_throughput.rst deleted file mode 100644 index 2b5b085452..0000000000 --- a/docs/report/introduction/methodology_data_plane_throughput.rst +++ /dev/null @@ -1,86 +0,0 @@ -Data Plane Throughput ---------------------- - -Network data plane packet and bandwidth throughput are measured using -multiple methods in order to obtain representative and repeatable -results across the large set of performance test cases implemented and -executed within CSIT. Following throughput test methods are used: - -#. MLRsearch: Multiple Loss Ratio search - - - **Description**: MLRsearch discovers multiple packet throughput - rates in a single search, reducing the overall test execution - time compared to a binary search. Each rate associated with a - distinct Packet Loss Ratio (PLR) criteria. In FD.io CSIT two - throughput rates are discovered: Non-Drop Rate (NDR, with zero - packet loss, PLR=0) and Partial Drop Rate (PDR, with PLR<0.5%). - MLRsearch is compliant with :rfc:`2544`. - - **Usage**: MLRsearch tests are run to discover NDR and PDR rates - for each VPP and DPDK release covered by CSIT report. Results for - small frame sizes (64b/78B, IMIX) are presented in packet - throughput graphs (Box-and-Whisker Plots) with NDR and PDR rates - plotted against the test cases covering popular VPP packet paths. - Each test is executed at least 10 times to verify measurements - repeatability and results are compared between releases and test - environments. NDR and PDR packet and bandwidth throughput results - for all frame sizes and for all tests are presented in detailed - results tables. - - **References**: See :ref:`mlrsearch_algorithm` for more detailed - description of MLRsearch tests. MLRsearch is being standardized - in IETF with `draft-vpolak-mkonstan-mlrsearch - <https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch>`_. - -#. MRR Measurements: Maximum Receive Rate - - - **Description**: MRR tests are complementary to MLRsearch tests, - as they provide a maximum “raw” throughput benchmark for - development and testing community. MRR tests measure the packet - forwarding rate under the maximum load offered by traffic - generator (dependent on link type and NIC model) over a set trial - duration, regardless of packet loss. Maximum load for specified - Ethernet frame size is set to the bi-directional link rate. - - **Usage**: MRR tests are much faster than MLRsearch as they rely - on a single trial or a small set of trials with very short - duration. It is this property that makes them suitable for - continuous execution in daily performance trending jobs enabling - detection of performance anomalies (regressions, progressions) - resulting from data plane code changes. MRR tests are also used - for VPP per patch performance jobs verifying patch performance - vs. parent. CSIT reports include MRR throughput comparisons - between releases and test environments. Small frame sizes only - (64b/78B, IMIX). - - **References**: See :ref:`mrr_throughput` for more detailed - description of MRR tests configuration used for daily performance - trending jobs. VPP per patch test methodology is available on - `FD.io CSIT trending pages - <https://docs.fd.io/csit/master/trending/methodology/perpatch_performance_tests.html>`_. - -#. PLRsearch: Probabilistic Loss Ratio search - - - **Description**: PLRsearch discovers a packet throughput rate - associated with configured Packet Loss Ratio (PLR) criteria for - tests run over an extended period of time a.k.a. soak testing. - PLRsearch assumes that system under test is probabilistic in - nature, and not deterministic. - - **Usage**: PLRsearch are run to discover a sustained throughput - for PLR=10^-7 (close to NDR) for VPP release covered by CSIT - report. Results for small frame sizes (64b/78B) are presented in - packet throughput graphs (Box Plots) for a small subset of - baseline tests. Each soak test lasts 2hrs and is executed at - least twice. Results are compared against NDR and PDR rates - discovered with MLRsearch. - - **References**: See :ref:`plrsearch_algorithm` for more detailed - description of PLRsearch tests. PLRsearch is being standardized - in IETF with `draft-vpolak-bmwg-plrsearch - <https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch>`_. - -All of the listed data plane throughput test methodologies share -following properties: - -- Tested L2 frame sizes (untagged Ethernet): - - - IPv4 payload: 64B, IMIX (28x64B, 16x570B, 4x1518B), 1518B, 9000B. - - IPv6 payload: 78B, IMIX (28x78B, 16x570B, 4x1518B), 1518B, 9000B. - -- All measured rates are aggregate bi-directional rates reported from - external Traffic Generator perspective.
\ No newline at end of file diff --git a/docs/report/introduction/methodology_data_plane_throughput/index.rst b/docs/report/introduction/methodology_data_plane_throughput/index.rst new file mode 100644 index 0000000000..f87283810f --- /dev/null +++ b/docs/report/introduction/methodology_data_plane_throughput/index.rst @@ -0,0 +1,10 @@ +Data Plane Throughput +===================== + +.. toctree:: + + methodology_data_plane_throughput + methodology_mlrsearch_tests + methodology_mrr_throughput + methodology_plrsearch + diff --git a/docs/report/introduction/methodology_data_plane_throughput/methodology_data_plane_throughput.rst b/docs/report/introduction/methodology_data_plane_throughput/methodology_data_plane_throughput.rst new file mode 100644 index 0000000000..762a7c2675 --- /dev/null +++ b/docs/report/introduction/methodology_data_plane_throughput/methodology_data_plane_throughput.rst @@ -0,0 +1,137 @@ +Data Plane Throughput Tests +--------------------------- + +Network data plane throughput is measured using multiple test methods in +order to obtain representative and repeatable results across the large +set of performance test cases implemented and executed within CSIT. + +Following throughput test methods are used: + +- MLRsearch - Multiple Loss Ratio search +- MRR - Maximum Receive Rate +- PLRsearch - Probabilistic Loss Ratio search + +Description of each test method is followed by generic test properties +shared by all methods. + +MLRsearch Tests +^^^^^^^^^^^^^^^ + +Description +~~~~~~~~~~~ + +Multiple Loss Ratio search (MLRsearch) tests discover multiple packet +throughput rates in a single search, reducing the overall test execution +time compared to a binary search. Each rate associated with a distinct +Packet Loss Ratio (PLR) criteria. In FD.io CSIT two throughput rates are +discovered: Non-Drop Rate (NDR, with zero packet loss, PLR=0) and +Partial Drop Rate (PDR, with PLR<0.5%). MLRsearch is compliant with +:rfc:`2544`. + +Usage +~~~~~ + +MLRsearch tests are run to discover NDR and PDR rates for each VPP and +DPDK release covered by CSIT report. Results for small frame sizes +(64b/78B, IMIX) are presented in packet throughput graphs +(Box-and-Whisker Plots) with NDR and PDR rates plotted against the test +cases covering popular VPP packet paths. + +Each test is executed at least 10 times to verify measurements +repeatability and results are compared between releases and test +environments. NDR and PDR packet and bandwidth throughput results for +all frame sizes and for all tests are presented in detailed results +tables. + +Details +~~~~~~~ + +See :ref:`mlrsearch_algorithm` section for more detail. MLRsearch is +being standardized in IETF in `draft-vpolak-mkonstan-mlrsearch +<https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch>`_. + +MRR Tests +^^^^^^^^^ + +Description +~~~~~~~~~~~ + +Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests, +as they provide a maximum “raw” throughput benchmark for development and +testing community. + +MRR tests measure the packet forwarding rate under the maximum load +offered by traffic generator (dependent on link type and NIC model) over +a set trial duration, regardless of packet loss. Maximum load for +specified Ethernet frame size is set to the bi-directional link rate. + +MRR tests are much faster than MLRsearch as they rely on a single trial +or a small set of trials with very short duration. It is this property +that makes them suitable for continuous execution in daily performance +trending jobs enabling detection of performance anomalies (regressions, +progressions) resulting from data plane code changes. + +MRR tests are also used for VPP per patch performance jobs verifying +patch performance vs. parent. CSIT reports include MRR throughput +comparisons between releases and test environments. Small frame sizes +only (64b/78B, IMIX). + +Details +~~~~~~~ + +See :ref:`mrr_throughput` section for more detail about MRR tests +configuration. + +FD.io CSIT performance dashboard includes complete description of +`daily performance trending tests +<https://docs.fd.io/csit/master/trending/methodology/performance_tests.html>`_ +and `VPP per patch tests +<https://docs.fd.io/csit/master/trending/methodology/perpatch_performance_tests.html>`_. + +PLRsearch Tests +^^^^^^^^^^^^^^^ + +Description +~~~~~~~~~~~ + +Probabilistic Loss Ratio search (PLRsearch) tests discovers a packet +throughput rate associated with configured Packet Loss Ratio (PLR) +criteria for tests run over an extended period of time a.k.a. soak +testing. PLRsearch assumes that system under test is probabilistic in +nature, and not deterministic. + +Usage +~~~~~ + +PLRsearch are run to discover a sustained throughput for PLR=10^-7 +(close to NDR) for VPP release covered by CSIT report. Results for small +frame sizes (64b/78B) are presented in packet throughput graphs (Box +Plots) for a small subset of baseline tests. + +Each soak test lasts 2hrs and is executed at least twice. Results are +compared against NDR and PDR rates discovered with MLRsearch. + +Details +~~~~~~~ + +See :ref:`plrsearch_algorithm` section for more detail. PLRsearch is +being standardized in IETF in `draft-vpolak-bmwg-plrsearch +<https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch>`_. + +Generic Test Properties +^^^^^^^^^^^^^^^^^^^^^^^ + +All data plane throughput test methodologies share following generic +properties: + +- Tested L2 frame sizes (untagged Ethernet): + + - IPv4 payload: 64B, IMIX (28x64B, 16x570B, 4x1518B), 1518B, 9000B. + - IPv6 payload: 78B, IMIX (28x78B, 16x570B, 4x1518B), 1518B, 9000B. + - All quoted sizes include frame CRC, but exclude per frame + transmission overhead of 20B (preamble, inter frame gap). + +- Offered packet load is always bi-directional and symmetric. +- All measured and reported packet and bandwidth rates are aggregate + bi-directional rates reported from external Traffic Generator + perspective. diff --git a/docs/report/introduction/methodology_mlrsearch_tests.rst b/docs/report/introduction/methodology_data_plane_throughput/methodology_mlrsearch_tests.rst index 6c2adfcd06..6c2adfcd06 100644 --- a/docs/report/introduction/methodology_mlrsearch_tests.rst +++ b/docs/report/introduction/methodology_data_plane_throughput/methodology_mlrsearch_tests.rst diff --git a/docs/report/introduction/methodology_data_plane_throughput/methodology_mrr_throughput.rst b/docs/report/introduction/methodology_data_plane_throughput/methodology_mrr_throughput.rst new file mode 100644 index 0000000000..fd4baca2f3 --- /dev/null +++ b/docs/report/introduction/methodology_data_plane_throughput/methodology_mrr_throughput.rst @@ -0,0 +1,52 @@ +.. _mrr_throughput: + +MRR Throughput +-------------- + +Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests, +as they provide a maximum "raw" throughput benchmark for development and +testing community. MRR tests measure the packet forwarding rate under +the maximum load offered by traffic generator over a set trial duration, +regardless of packet loss. + +MRR tests are currently used for following test jobs: + +- Report performance comparison: 64B, IMIX for vhost, memif. +- Daily performance trending: 64B, IMIX for vhost, memif. +- Per-patch performance verification: 64B. +- PLRsearch soaking tests: 64B. + +Maximum offered load for specific L2 Ethernet frame size is set to +either the maximum bi-directional link rate or tested NIC model +capacity, as follows: + +- For 10GE NICs the maximum packet rate load is 2x14.88 Mpps for 64B, a + 10GE bi-directional link rate. +- For 25GE NICs the maximum packet rate load is 2x18.75 Mpps for 64B, a + 25GE bi-directional link sub-rate limited by 25GE NIC used on TRex TG, + XXV710. +- For 40GE NICs the maximum packet rate load is 2x18.75 Mpps for 64B, a + 40GE bi-directional link sub-rate limited by 40GE NIC used on TRex + TG,XL710. Packet rate for other tested frame sizes is limited by + PCIeGen3 x8 bandwidth limitation of ~50Gbps. + +MRR test code implements multiple bursts of offered packet load and has +two configurable burst parameters: individual trial duration and number +of trials in a single burst. This enables more precise performance +trending by providing more results data for analysis. + +Burst parameter settings vary between different tests using MRR: + +- MRR individual trial duration: + + - Report performance comparison: 1 sec. + - Daily performance trending: 1 sec. + - Per-patch performance verification: 10 sec. + - PLRsearch soaking tests: 5.2 sec. + +- Number of MRR trials per burst: + + - Report performance comparison: 10. + - Daily performance trending: 10. + - Per-patch performance verification: 5. + - PLRsearch soaking tests: 1.
\ No newline at end of file diff --git a/docs/report/introduction/methodology_plrsearch.rst b/docs/report/introduction/methodology_data_plane_throughput/methodology_plrsearch.rst index fdd03472c3..fdd03472c3 100644 --- a/docs/report/introduction/methodology_plrsearch.rst +++ b/docs/report/introduction/methodology_data_plane_throughput/methodology_plrsearch.rst diff --git a/docs/report/introduction/methodology_mrr_throughput.rst b/docs/report/introduction/methodology_mrr_throughput.rst deleted file mode 100644 index 824ac887f7..0000000000 --- a/docs/report/introduction/methodology_mrr_throughput.rst +++ /dev/null @@ -1,55 +0,0 @@ -.. _mrr_throughput: - -MRR Throughput --------------- - -Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests, -as they provide a maximum "raw" throughput benchmark for development and -testing community. MRR tests measure the packet forwarding rate under -the maximum load offered by traffic generator over a set trial duration, -regardless of packet loss. Maximum load for specified Ethernet frame -size is set to the bi-directional link rate. - -In |csit-release| MRR test code has been updated with a configurable -burst MRR parameters: trial duration and number of trials in a single -burst. This enabled more precise performance trending. - -Current parameters for MRR trending tests: - -- Ethernet frame sizes: 64B (78B for IPv6), IMIX, 1518B, 9000B; all - quoted sizes include frame CRC, but exclude per frame transmission - overhead of 20B (preamble, inter frame gap). - -- Maximum load offered: 10GE, 25GE and 40GE link (sub-)rates depending on NIC - tested, with the actual packet rate depending on frame size, - transmission overhead and traffic generator NIC forwarding capacity. - - - For 10GE NICs the maximum packet rate load is 2* 14.88 Mpps for 64B, - a 10GE bi-directional link rate. - - For 25GE NICs the maximum packet rate load is 2* 18.75 Mpps for 64B, - a 25GE bi-directional link sub-rate limited by TG 25GE NIC used, - XXV710. - - For 40GE NICs the maximum packet rate load is 2* 18.75 Mpps for 64B, - a 40GE bi-directional link sub-rate limited by TG 40GE NIC used, - XL710. Packet rate for other tested frame sizes is limited by PCIe - Gen3 x8 bandwidth limitation of ~50Gbps. - -- Trial duration: 1 sec. - -- Number of trials per burst: 10. - -Similarly to NDR/PDR throughput tests, MRR test should be reporting -bi-directional link rate (or NIC rate, if lower) if tested VPP -configuration can handle the packet rate higher than bi-directional link -rate, e.g. large packet tests and/or multi-core tests. - -MRR tests are currently used for FD.io CSIT continuous performance -trending and for comparison between releases. Daily trending job tests -subset of frame sizes, focusing on 64B (78B for IPv6) for all tests and -IMIX for selected tests (vhost, memif). - -MRR-like measurements are being used to establish starting conditions -for experimental Probabilistic Loss Ratio Search (PLRsearch) used for -soak testing, aimed at verifying continuous system performance over an -extended period of time, hours, days, weeks, months. PLRsearch code is -currently in experimental phase in FD.io CSIT project. |