report: further edits of methodology throughput sections

Change-Id: I571d1a47743eb31ee10caf3f3336ac7437daf878
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>

8 files changed, 200 insertions, 142 deletions

diff --git a/docs/report/introduction/methodology.rst b/docs/report/introduction/methodology.rst
index 37ff946ff7..506fa73b5b 100644
--- a/docs/report/introduction/methodology.rst
+++ b/docs/report/introduction/methodology.rst
@@ -9,7 +9,7 @@ Test Methodology
     methodology_vpp_forwarding_modes
     methodology_tunnel_encapsulations
     methodology_vpp_features
-    methodology_data_plane_throughput
+    methodology_data_plane_throughput/index
     methodology_mlrsearch_tests
     methodology_mrr_throughput
     methodology_plrsearch
diff --git a/docs/report/introduction/methodology_data_plane_throughput.rst b/docs/report/introduction/methodology_data_plane_throughput.rst
deleted file mode 100644
index 2b5b085452..0000000000
--- a/docs/report/introduction/methodology_data_plane_throughput.rst
+++ /dev/null
@@ -1,86 +0,0 @@
-Data Plane Throughput
----------------------
-
-Network data plane packet and bandwidth throughput are measured using
-multiple methods in order to obtain representative and repeatable
-results across the large set of performance test cases implemented and
-executed within CSIT. Following throughput test methods are used:
-
-#. MLRsearch: Multiple Loss Ratio search
-
-   - **Description**: MLRsearch discovers multiple packet throughput
-     rates in a single search, reducing the overall test execution
-     time compared to a binary search. Each rate associated with a
-     distinct Packet Loss Ratio (PLR) criteria. In FD.io CSIT two
-     throughput rates are discovered: Non-Drop Rate (NDR, with zero
-     packet loss, PLR=0) and Partial Drop Rate (PDR, with PLR<0.5%).
-     MLRsearch is compliant with :rfc:`2544`.
-   - **Usage**: MLRsearch tests are run to discover NDR and PDR rates
-     for each VPP and DPDK release covered by CSIT report. Results for
-     small frame sizes (64b/78B, IMIX) are presented in packet
-     throughput graphs (Box-and-Whisker Plots) with NDR and PDR rates
-     plotted against the test cases covering popular VPP packet paths.
-     Each test is executed at least 10 times to verify measurements
-     repeatability and results are compared between releases and test
-     environments. NDR and PDR packet and bandwidth throughput results
-     for all frame sizes and for all tests are presented in detailed
-     results tables.
-   - **References**: See :ref:`mlrsearch_algorithm` for more detailed
-     description of MLRsearch tests. MLRsearch is being standardized
-     in IETF with `draft-vpolak-mkonstan-mlrsearch
-     <https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch>`_.
-
-#. MRR Measurements: Maximum Receive Rate
-
-   - **Description**: MRR tests are complementary to MLRsearch tests,
-     as they provide a maximum “raw” throughput benchmark for
-     development and testing community. MRR tests measure the packet
-     forwarding rate under the maximum load offered by traffic
-     generator (dependent on link type and NIC model) over a set trial
-     duration, regardless of packet loss. Maximum load for specified
-     Ethernet frame size is set to the bi-directional link rate.
-   - **Usage**: MRR tests are much faster than MLRsearch as they rely
-     on a single trial or a small set of trials with very short
-     duration. It is this property that makes them suitable for
-     continuous execution in daily performance trending jobs enabling
-     detection of performance anomalies (regressions, progressions)
-     resulting from data plane code changes. MRR tests are also used
-     for VPP per patch performance jobs verifying patch performance
-     vs. parent. CSIT reports include MRR throughput comparisons
-     between releases and test environments. Small frame sizes only
-     (64b/78B, IMIX).
-   - **References**: See :ref:`mrr_throughput` for more detailed
-     description of MRR tests configuration used for daily performance
-     trending jobs. VPP per patch test methodology is available on
-     `FD.io CSIT trending pages
-     <https://docs.fd.io/csit/master/trending/methodology/perpatch_performance_tests.html>`_.
-
-#. PLRsearch: Probabilistic Loss Ratio search
-
-   - **Description**: PLRsearch discovers a packet throughput rate
-     associated with configured Packet Loss Ratio (PLR) criteria for
-     tests run over an extended period of time a.k.a. soak testing.
-     PLRsearch assumes that system under test is probabilistic in
-     nature, and not deterministic.
-   - **Usage**: PLRsearch are run to discover a sustained throughput
-     for PLR=10^-7 (close to NDR) for VPP release covered by CSIT
-     report. Results for small frame sizes (64b/78B) are presented in
-     packet throughput graphs (Box Plots) for a small subset of
-     baseline tests. Each soak test lasts 2hrs and is executed at
-     least twice. Results are compared against NDR and PDR rates
-     discovered with MLRsearch.
-   - **References**: See :ref:`plrsearch_algorithm` for more detailed
-     description of PLRsearch tests. PLRsearch is being standardized
-     in IETF with `draft-vpolak-bmwg-plrsearch
-     <https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch>`_.
-
-All of the listed data plane throughput test methodologies share
-following properties:
-
-- Tested L2 frame sizes (untagged Ethernet):
-
-  - IPv4 payload: 64B, IMIX (28x64B, 16x570B, 4x1518B), 1518B, 9000B.
-  - IPv6 payload: 78B, IMIX (28x78B, 16x570B, 4x1518B), 1518B, 9000B.
-
-- All measured rates are aggregate bi-directional rates reported from
-  external Traffic Generator perspective.
\ No newline at end of file
diff --git a/docs/report/introduction/methodology_data_plane_throughput/index.rst b/docs/report/introduction/methodology_data_plane_throughput/index.rst
new file mode 100644
index 0000000000..f87283810f
--- /dev/null
+++ b/docs/report/introduction/methodology_data_plane_throughput/index.rst
@@ -0,0 +1,10 @@
+Data Plane Throughput
+=====================
+
+.. toctree::
+
+    methodology_data_plane_throughput
+    methodology_mlrsearch_tests
+    methodology_mrr_throughput
+    methodology_plrsearch
+
diff --git a/docs/report/introduction/methodology_data_plane_throughput/methodology_data_plane_throughput.rst b/docs/report/introduction/methodology_data_plane_throughput/methodology_data_plane_throughput.rst
new file mode 100644
index 0000000000..762a7c2675
--- /dev/null
+++ b/docs/report/introduction/methodology_data_plane_throughput/methodology_data_plane_throughput.rst
@@ -0,0 +1,137 @@
+Data Plane Throughput Tests
+---------------------------
+
+Network data plane throughput is measured using multiple test methods in
+order to obtain representative and repeatable results across the large
+set of performance test cases implemented and executed within CSIT.
+
+Following throughput test methods are used:
+
+- MLRsearch - Multiple Loss Ratio search
+- MRR - Maximum Receive Rate
+- PLRsearch - Probabilistic Loss Ratio search
+
+Description of each test method is followed by generic test properties
+shared by all methods.
+
+MLRsearch Tests
+^^^^^^^^^^^^^^^
+
+Description
+~~~~~~~~~~~
+
+Multiple Loss Ratio search (MLRsearch) tests discover multiple packet
+throughput rates in a single search, reducing the overall test execution
+time compared to a binary search. Each rate associated with a distinct
+Packet Loss Ratio (PLR) criteria. In FD.io CSIT two throughput rates are
+discovered: Non-Drop Rate (NDR, with zero packet loss, PLR=0) and
+Partial Drop Rate (PDR, with PLR<0.5%). MLRsearch is compliant with
+:rfc:`2544`.
+
+Usage
+~~~~~
+
+MLRsearch tests are run to discover NDR and PDR rates for each VPP and
+DPDK release covered by CSIT report. Results for small frame sizes
+(64b/78B, IMIX) are presented in packet throughput graphs
+(Box-and-Whisker Plots) with NDR and PDR rates plotted against the test
+cases covering popular VPP packet paths.
+
+Each test is executed at least 10 times to verify measurements
+repeatability and results are compared between releases and test
+environments. NDR and PDR packet and bandwidth throughput results for
+all frame sizes and for all tests are presented in detailed results
+tables.
+
+Details
+~~~~~~~
+
+See :ref:`mlrsearch_algorithm` section for more detail. MLRsearch is
+being standardized in IETF in `draft-vpolak-mkonstan-mlrsearch
+<https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch>`_.
+
+MRR Tests
+^^^^^^^^^
+
+Description
+~~~~~~~~~~~
+
+Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests,
+as they provide a maximum “raw” throughput benchmark for development and
+testing community.
+
+MRR tests measure the packet forwarding rate under the maximum load
+offered by traffic generator (dependent on link type and NIC model) over
+a set trial duration, regardless of packet loss. Maximum load for
+specified Ethernet frame size is set to the bi-directional link rate.
+
+MRR tests are much faster than MLRsearch as they rely on a single trial
+or a small set of trials with very short duration. It is this property
+that makes them suitable for continuous execution in daily performance
+trending jobs enabling detection of performance anomalies (regressions,
+progressions) resulting from data plane code changes.
+
+MRR tests are also used for VPP per patch performance jobs verifying
+patch performance vs. parent. CSIT reports include MRR throughput
+comparisons between releases and test environments. Small frame sizes
+only (64b/78B, IMIX).
+
+Details
+~~~~~~~
+
+See :ref:`mrr_throughput` section for more detail about MRR tests
+configuration.
+
+FD.io CSIT performance dashboard includes complete description of
+`daily performance trending tests
+<https://docs.fd.io/csit/master/trending/methodology/performance_tests.html>`_
+and `VPP per patch tests
+<https://docs.fd.io/csit/master/trending/methodology/perpatch_performance_tests.html>`_.
+
+PLRsearch Tests
+^^^^^^^^^^^^^^^
+
+Description
+~~~~~~~~~~~
+
+Probabilistic Loss Ratio search (PLRsearch) tests discovers a packet
+throughput rate associated with configured Packet Loss Ratio (PLR)
+criteria for tests run over an extended period of time a.k.a. soak
+testing. PLRsearch assumes that system under test is probabilistic in
+nature, and not deterministic.
+
+Usage
+~~~~~
+
+PLRsearch are run to discover a sustained throughput for PLR=10^-7
+(close to NDR) for VPP release covered by CSIT report. Results for small
+frame sizes (64b/78B) are presented in packet throughput graphs (Box
+Plots) for a small subset of baseline tests.
+
+Each soak test lasts 2hrs and is executed at least twice. Results are
+compared against NDR and PDR rates discovered with MLRsearch.
+
+Details
+~~~~~~~
+
+See :ref:`plrsearch_algorithm` section for more detail. PLRsearch is
+being standardized in IETF in `draft-vpolak-bmwg-plrsearch
+<https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch>`_.
+
+Generic Test Properties
+^^^^^^^^^^^^^^^^^^^^^^^
+
+All data plane throughput test methodologies share following generic
+properties:
+
+- Tested L2 frame sizes (untagged Ethernet):
+
+  - IPv4 payload: 64B, IMIX (28x64B, 16x570B, 4x1518B), 1518B, 9000B.
+  - IPv6 payload: 78B, IMIX (28x78B, 16x570B, 4x1518B), 1518B, 9000B.
+  - All quoted sizes include frame CRC, but exclude per frame
+    transmission overhead of 20B (preamble, inter frame gap).
+
+- Offered packet load is always bi-directional and symmetric.
+- All measured and reported packet and bandwidth rates are aggregate
+  bi-directional rates reported from external Traffic Generator
+  perspective.
diff --git a/docs/report/introduction/methodology_mlrsearch_tests.rst b/docs/report/introduction/methodology_data_plane_throughput/methodology_mlrsearch_tests.rst
index 6c2adfcd06..6c2adfcd06 100644
--- a/docs/report/introduction/methodology_mlrsearch_tests.rst
+++ b/docs/report/introduction/methodology_data_plane_throughput/methodology_mlrsearch_tests.rst
diff --git a/docs/report/introduction/methodology_data_plane_throughput/methodology_mrr_throughput.rst b/docs/report/introduction/methodology_data_plane_throughput/methodology_mrr_throughput.rst
new file mode 100644
index 0000000000..fd4baca2f3
--- /dev/null
+++ b/docs/report/introduction/methodology_data_plane_throughput/methodology_mrr_throughput.rst
@@ -0,0 +1,52 @@
+.. _mrr_throughput:
+
+MRR Throughput
+--------------
+
+Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests,
+as they provide a maximum "raw" throughput benchmark for development and
+testing community. MRR tests measure the packet forwarding rate under
+the maximum load offered by traffic generator over a set trial duration,
+regardless of packet loss.
+
+MRR tests are currently used for following test jobs:
+
+- Report performance comparison: 64B, IMIX for vhost, memif.
+- Daily performance trending: 64B, IMIX for vhost, memif.
+- Per-patch performance verification: 64B.
+- PLRsearch soaking tests: 64B.
+
+Maximum offered load for specific L2 Ethernet frame size is set to
+either the maximum bi-directional link rate or tested NIC model
+capacity, as follows:
+
+- For 10GE NICs the maximum packet rate load is 2x14.88 Mpps for 64B, a
+  10GE bi-directional link rate.
+- For 25GE NICs the maximum packet rate load is 2x18.75 Mpps for 64B, a
+  25GE bi-directional link sub-rate limited by 25GE NIC used on TRex TG,
+  XXV710.
+- For 40GE NICs the maximum packet rate load is 2x18.75 Mpps for 64B, a
+  40GE bi-directional link sub-rate limited by 40GE NIC used on TRex
+  TG,XL710. Packet rate for other tested frame sizes is limited by
+  PCIeGen3 x8 bandwidth limitation of ~50Gbps.
+
+MRR test code implements multiple bursts of offered packet load and has
+two configurable burst parameters: individual trial duration and number
+of trials in a single burst. This enables more precise performance
+trending by providing more results data for analysis.
+
+Burst parameter settings vary between different tests using MRR:
+
+- MRR individual trial duration:
+
+  - Report performance comparison: 1 sec.
+  - Daily performance trending: 1 sec.
+  - Per-patch performance verification: 10 sec.
+  - PLRsearch soaking tests: 5.2 sec.
+
+- Number of MRR trials per burst:
+
+  - Report performance comparison: 10.
+  - Daily performance trending: 10.
+  - Per-patch performance verification: 5.
+  - PLRsearch soaking tests: 1.
\ No newline at end of file
diff --git a/docs/report/introduction/methodology_plrsearch.rst b/docs/report/introduction/methodology_data_plane_throughput/methodology_plrsearch.rst
index fdd03472c3..fdd03472c3 100644
--- a/docs/report/introduction/methodology_plrsearch.rst
+++ b/docs/report/introduction/methodology_data_plane_throughput/methodology_plrsearch.rst
diff --git a/docs/report/introduction/methodology_mrr_throughput.rst b/docs/report/introduction/methodology_mrr_throughput.rst
deleted file mode 100644
index 824ac887f7..0000000000
--- a/docs/report/introduction/methodology_mrr_throughput.rst
+++ /dev/null
@@ -1,55 +0,0 @@
-.. _mrr_throughput:
-
-MRR Throughput
---------------
-
-Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests,
-as they provide a maximum "raw" throughput benchmark for development and
-testing community. MRR tests measure the packet forwarding rate under
-the maximum load offered by traffic generator over a set trial duration,
-regardless of packet loss. Maximum load for specified Ethernet frame
-size is set to the bi-directional link rate.
-
-In |csit-release| MRR test code has been updated with a configurable
-burst MRR parameters: trial duration and number of trials in a single
-burst. This enabled more precise performance trending.
-
-Current parameters for MRR trending tests:
-
-- Ethernet frame sizes: 64B (78B for IPv6), IMIX, 1518B, 9000B; all
-  quoted sizes include frame CRC, but exclude per frame transmission
-  overhead of 20B (preamble, inter frame gap).
-
-- Maximum load offered: 10GE, 25GE and 40GE link (sub-)rates depending on NIC
-  tested, with the actual packet rate depending on frame size,
-  transmission overhead and traffic generator NIC forwarding capacity.
-
-  - For 10GE NICs the maximum packet rate load is 2* 14.88 Mpps for 64B,
-    a 10GE bi-directional link rate.
-  - For 25GE NICs the maximum packet rate load is 2* 18.75 Mpps for 64B,
-    a 25GE bi-directional link sub-rate limited by TG 25GE NIC used,
-    XXV710.
-  - For 40GE NICs the maximum packet rate load is 2* 18.75 Mpps for 64B,
-    a 40GE bi-directional link sub-rate limited by TG 40GE NIC used,
-    XL710. Packet rate for other tested frame sizes is limited by PCIe
-    Gen3 x8 bandwidth limitation of ~50Gbps.
-
-- Trial duration: 1 sec.
-
-- Number of trials per burst: 10.
-
-Similarly to NDR/PDR throughput tests, MRR test should be reporting
-bi-directional link rate (or NIC rate, if lower) if tested VPP
-configuration can handle the packet rate higher than bi-directional link
-rate, e.g. large packet tests and/or multi-core tests.
-
-MRR tests are currently used for FD.io CSIT continuous performance
-trending and for comparison between releases. Daily trending job tests
-subset of frame sizes, focusing on 64B (78B for IPv6) for all tests and
-IMIX for selected tests (vhost, memif).
-
-MRR-like measurements are being used to establish starting conditions
-for experimental Probabilistic Loss Ratio Search (PLRsearch) used for
-soak testing, aimed at verifying continuous system performance over an
-extended period of time, hours, days, weeks, months. PLRsearch code is
-currently in experimental phase in FD.io CSIT project.