aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorMaciek Konstantynowicz <mkonstan@cisco.com>2019-05-08 12:13:42 +0100
committerPeter Mikus <pmikus@cisco.com>2019-05-08 13:24:10 +0000
commitcc7a7eea385534107bfdae744797d68782488a5e (patch)
treeb80bf0b2faad7156f06e0da6e2f66acde5cc3cd6
parent23185b233e4dd7984a404aa54d5dd0da2502074b (diff)
report: edits of methodology throughput sections
Change-Id: I9ed1c2d67bc53f98d36ad17afb9153de5e944b7f Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
-rw-r--r--docs/report/introduction/methodology.rst4
-rw-r--r--docs/report/introduction/methodology_data_plane_throughput.rst97
-rw-r--r--docs/report/introduction/methodology_mlrsearch_tests.rst1
-rw-r--r--docs/report/introduction/methodology_mrr_throughput.rst (renamed from docs/report/introduction/methodology_bmrr_throughput.rst)11
-rw-r--r--docs/report/introduction/methodology_plrsearch.rst2
5 files changed, 84 insertions, 31 deletions
diff --git a/docs/report/introduction/methodology.rst b/docs/report/introduction/methodology.rst
index e9d87075b9..45cd6bc598 100644
--- a/docs/report/introduction/methodology.rst
+++ b/docs/report/introduction/methodology.rst
@@ -11,7 +11,8 @@ Test Methodology
methodology_vpp_features
methodology_data_plane_throughput
methodology_mlrsearch_tests
- methodology_bmrr_throughput
+ methodology_mrr_throughput
+ methodology_plrsearch
methodology_packet_latency
methodology_multi_core_speedup
methodology_vpp_startup_settings
@@ -22,7 +23,6 @@ Test Methodology
methodology_vpp_device_functional
methodology_ipsec_on_intel_qat
methodology_trex_traffic_generator
- methodology_plrsearch
..
methodology_http_tcp_with_wrk_tool
diff --git a/docs/report/introduction/methodology_data_plane_throughput.rst b/docs/report/introduction/methodology_data_plane_throughput.rst
index 10d85e991c..2b5b085452 100644
--- a/docs/report/introduction/methodology_data_plane_throughput.rst
+++ b/docs/report/introduction/methodology_data_plane_throughput.rst
@@ -1,33 +1,86 @@
Data Plane Throughput
---------------------
-Network data plane packet and bandwidth throughput are measured in
-accordance with :rfc:`2544`, using FD.io CSIT Multiple Loss Ratio search
-(MLRsearch), an optimized throughput search algorithm, that measures
-SUT/DUT packet throughput rates at different Packet Loss Ratio (PLR)
-values.
+Network data plane packet and bandwidth throughput are measured using
+multiple methods in order to obtain representative and repeatable
+results across the large set of performance test cases implemented and
+executed within CSIT. Following throughput test methods are used:
-Following MLRsearch values are measured across a range of L2 frame sizes
-and reported:
+#. MLRsearch: Multiple Loss Ratio search
-- NON DROP RATE (NDR): packet and bandwidth throughput at PLR=0%.
+ - **Description**: MLRsearch discovers multiple packet throughput
+ rates in a single search, reducing the overall test execution
+ time compared to a binary search. Each rate associated with a
+ distinct Packet Loss Ratio (PLR) criteria. In FD.io CSIT two
+ throughput rates are discovered: Non-Drop Rate (NDR, with zero
+ packet loss, PLR=0) and Partial Drop Rate (PDR, with PLR<0.5%).
+ MLRsearch is compliant with :rfc:`2544`.
+ - **Usage**: MLRsearch tests are run to discover NDR and PDR rates
+ for each VPP and DPDK release covered by CSIT report. Results for
+ small frame sizes (64b/78B, IMIX) are presented in packet
+ throughput graphs (Box-and-Whisker Plots) with NDR and PDR rates
+ plotted against the test cases covering popular VPP packet paths.
+ Each test is executed at least 10 times to verify measurements
+ repeatability and results are compared between releases and test
+ environments. NDR and PDR packet and bandwidth throughput results
+ for all frame sizes and for all tests are presented in detailed
+ results tables.
+ - **References**: See :ref:`mlrsearch_algorithm` for more detailed
+ description of MLRsearch tests. MLRsearch is being standardized
+ in IETF with `draft-vpolak-mkonstan-mlrsearch
+ <https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch>`_.
- - **Aggregate packet rate**: NDR_LOWER <bi-directional packet rate>
- pps.
- - **Aggregate bandwidth rate**: NDR_LOWER <bi-directional bandwidth
- rate> Gbps.
+#. MRR Measurements: Maximum Receive Rate
-- PARTIAL DROP RATE (PDR): packet and bandwidth throughput at PLR=0.5%.
+ - **Description**: MRR tests are complementary to MLRsearch tests,
+ as they provide a maximum “raw” throughput benchmark for
+ development and testing community. MRR tests measure the packet
+ forwarding rate under the maximum load offered by traffic
+ generator (dependent on link type and NIC model) over a set trial
+ duration, regardless of packet loss. Maximum load for specified
+ Ethernet frame size is set to the bi-directional link rate.
+ - **Usage**: MRR tests are much faster than MLRsearch as they rely
+ on a single trial or a small set of trials with very short
+ duration. It is this property that makes them suitable for
+ continuous execution in daily performance trending jobs enabling
+ detection of performance anomalies (regressions, progressions)
+ resulting from data plane code changes. MRR tests are also used
+ for VPP per patch performance jobs verifying patch performance
+ vs. parent. CSIT reports include MRR throughput comparisons
+ between releases and test environments. Small frame sizes only
+ (64b/78B, IMIX).
+ - **References**: See :ref:`mrr_throughput` for more detailed
+ description of MRR tests configuration used for daily performance
+ trending jobs. VPP per patch test methodology is available on
+ `FD.io CSIT trending pages
+ <https://docs.fd.io/csit/master/trending/methodology/perpatch_performance_tests.html>`_.
- - **Aggregate packet rate**: PDR_LOWER <bi-directional packet rate>
- pps.
- - **Aggregate bandwidth rate**: PDR_LOWER <bi-directional bandwidth
- rate> Gbps.
+#. PLRsearch: Probabilistic Loss Ratio search
-NDR and PDR are measured for the following L2 frame sizes (untagged
-Ethernet):
+ - **Description**: PLRsearch discovers a packet throughput rate
+ associated with configured Packet Loss Ratio (PLR) criteria for
+ tests run over an extended period of time a.k.a. soak testing.
+ PLRsearch assumes that system under test is probabilistic in
+ nature, and not deterministic.
+ - **Usage**: PLRsearch are run to discover a sustained throughput
+ for PLR=10^-7 (close to NDR) for VPP release covered by CSIT
+ report. Results for small frame sizes (64b/78B) are presented in
+ packet throughput graphs (Box Plots) for a small subset of
+ baseline tests. Each soak test lasts 2hrs and is executed at
+ least twice. Results are compared against NDR and PDR rates
+ discovered with MLRsearch.
+ - **References**: See :ref:`plrsearch_algorithm` for more detailed
+ description of PLRsearch tests. PLRsearch is being standardized
+ in IETF with `draft-vpolak-bmwg-plrsearch
+ <https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch>`_.
-- IPv4 payload: 64B, IMIX (28x64B, 16x570B, 4x1518B), 1518B, 9000B.
-- IPv6 payload: 78B, IMIX (28x78B, 16x570B, 4x1518B), 1518B, 9000B.
+All of the listed data plane throughput test methodologies share
+following properties:
-All rates are reported from external Traffic Generator perspective.
+- Tested L2 frame sizes (untagged Ethernet):
+
+ - IPv4 payload: 64B, IMIX (28x64B, 16x570B, 4x1518B), 1518B, 9000B.
+ - IPv6 payload: 78B, IMIX (28x78B, 16x570B, 4x1518B), 1518B, 9000B.
+
+- All measured rates are aggregate bi-directional rates reported from
+ external Traffic Generator perspective. \ No newline at end of file
diff --git a/docs/report/introduction/methodology_mlrsearch_tests.rst b/docs/report/introduction/methodology_mlrsearch_tests.rst
index 15b9ee0290..6c2adfcd06 100644
--- a/docs/report/introduction/methodology_mlrsearch_tests.rst
+++ b/docs/report/introduction/methodology_mlrsearch_tests.rst
@@ -1,4 +1,3 @@
-
.. _mlrsearch_algorithm:
MLRsearch Tests
diff --git a/docs/report/introduction/methodology_bmrr_throughput.rst b/docs/report/introduction/methodology_mrr_throughput.rst
index 7bef3a4aaf..824ac887f7 100644
--- a/docs/report/introduction/methodology_bmrr_throughput.rst
+++ b/docs/report/introduction/methodology_mrr_throughput.rst
@@ -1,5 +1,7 @@
-(B)MRR Throughput
------------------
+.. _mrr_throughput:
+
+MRR Throughput
+--------------
Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests,
as they provide a maximum "raw" throughput benchmark for development and
@@ -10,10 +12,9 @@ size is set to the bi-directional link rate.
In |csit-release| MRR test code has been updated with a configurable
burst MRR parameters: trial duration and number of trials in a single
-burst. This enabled a new Burst MRR (BMRR) methodology for more precise
-performance trending.
+burst. This enabled more precise performance trending.
-Current parameters for BMRR tests:
+Current parameters for MRR trending tests:
- Ethernet frame sizes: 64B (78B for IPv6), IMIX, 1518B, 9000B; all
quoted sizes include frame CRC, but exclude per frame transmission
diff --git a/docs/report/introduction/methodology_plrsearch.rst b/docs/report/introduction/methodology_plrsearch.rst
index 65ac590222..fdd03472c3 100644
--- a/docs/report/introduction/methodology_plrsearch.rst
+++ b/docs/report/introduction/methodology_plrsearch.rst
@@ -1,4 +1,4 @@
-.. _`PLRsearch algorithm`:
+.. _plrsearch_algorithm:
PLRsearch
^^^^^^^^^