diff options
Diffstat (limited to 'docs/report')
21 files changed, 324 insertions, 327 deletions
diff --git a/docs/report/dpdk_performance_tests/overview.rst b/docs/report/dpdk_performance_tests/overview.rst index e6abb53c90..81f5ed5efc 100644 --- a/docs/report/dpdk_performance_tests/overview.rst +++ b/docs/report/dpdk_performance_tests/overview.rst @@ -10,8 +10,8 @@ Logical Topologies ------------------ CSIT DPDK performance tests are executed on physical testbeds described -in :ref:`tested_physical_topologies`. Based on the packet path thru -server SUTs, three distinct logical topology types are used for DPDK DUT +in :ref:`tested_physical_topologies`. Based on the packet path through +server SUTs, one distinct logical topology type is used for DPDK DUT data plane testing: #. NIC-to-NIC switching topologies. diff --git a/docs/report/dpdk_performance_tests/packet_latency_graphs/index.rst b/docs/report/dpdk_performance_tests/packet_latency_graphs/index.rst index 43f557c517..1604fb1576 100644 --- a/docs/report/dpdk_performance_tests/packet_latency_graphs/index.rst +++ b/docs/report/dpdk_performance_tests/packet_latency_graphs/index.rst @@ -5,14 +5,14 @@ Plotted results are generated from a single execution of CSIT NDR discovery test. Box plots are used to show the Minimum, Median and Maximum packet latency per test. -*Title of each graph* is a regex (regular expression) matching all -throughput test cases plotted on this graph, *X-axis labels* are indices -of individual test suites executed by -FD.io test executor dpdk performance jobs that created result output file -used as data source for the graph, *Y-axis labels* are measured packet Latency -[uSec] values, and the *Graph legend* lists the plotted test suites and their -indices. Latency is reported for concurrent symmetric bi-directional flows, -separately for each direction: i) West-to-East: +*Title of each graph* describes the packet path, testbed topology, processor +architecture, packet size, number of cores and threads used by workers and +DUT configuration, *X-axis labels* are indices of individual test suites +executed by FD.io test executor dpdk performance jobs that created result output +file used as data source for the graph, *Y-axis labels* are measured packet +Latency [uSec] values, and the *Graph legend* lists the plotted test suites and +their indices. Latency is reported for concurrent symmetric bi-directional +flows, separately for each direction: i) West-to-East: TGint1-to-SUT1-to-SUT2-to-TGint2, and ii) East-to-West: TGint2-to-SUT2-to-SUT1-to-TGint1. diff --git a/docs/report/dpdk_performance_tests/packet_latency_graphs/ip4.rst b/docs/report/dpdk_performance_tests/packet_latency_graphs/ip4.rst index 819e4a9ebd..21ddba1c9b 100644 --- a/docs/report/dpdk_performance_tests/packet_latency_graphs/ip4.rst +++ b/docs/report/dpdk_performance_tests/packet_latency_graphs/ip4.rst @@ -2,7 +2,7 @@ L3fwd ===== This section includes summary graphs of L3FWD Phy-to-Phy performance with packet -routed forwarding measured at 50% of discovered NDR throughput rate. Latency is +routed forwarding measured at 100% of discovered NDR throughput rate. Latency is reported for L3FWD running in multiple configurations of L3FWD pmd thread(s), a.k.a. L3FWD data plane thread(s), and their physical CPU core(s) placement. diff --git a/docs/report/dpdk_performance_tests/packet_latency_graphs/l2.rst b/docs/report/dpdk_performance_tests/packet_latency_graphs/l2.rst index 6d7ea30c98..c07c35d458 100644 --- a/docs/report/dpdk_performance_tests/packet_latency_graphs/l2.rst +++ b/docs/report/dpdk_performance_tests/packet_latency_graphs/l2.rst @@ -2,7 +2,7 @@ Testpmd ======= This section includes summary graphs of Testpmd Phy-to-Phy packet -latency with L2 Ethernet Interface Loop measured at 50% of discovered +latency with L2 Ethernet Interface Loop measured at 100% of discovered NDR throughput rate. Latency is reported for Testpmd running in multiple configurations of Testpmd pmd thread(s), a.k.a. Testpmd data plane thread(s), and their physical CPU core(s) placement. diff --git a/docs/report/dpdk_performance_tests/packet_throughput_graphs/index.rst b/docs/report/dpdk_performance_tests/packet_throughput_graphs/index.rst index cfa75419e6..5274e6be9a 100644 --- a/docs/report/dpdk_performance_tests/packet_throughput_graphs/index.rst +++ b/docs/report/dpdk_performance_tests/packet_throughput_graphs/index.rst @@ -17,10 +17,10 @@ top edge of the box. If multiple samples match only two values, and all samples fall between them, then no whiskers are plotted. If all samples have the same value, only a horizontal line is plotted. -*Title of each graph* is a regex (regular expression) matching all -throughput test cases plotted on this graph, *X-axis labels* are indices -of individual test suites executed by -FD.io test executor dpdk performance jobs that created result output +*Title of each graph* describes the packet path, testbed topology, processor +architecture, packet size, number of cores and threads used by workers and +DUT configuration, *X-axis labels* are indices by individual test suites +executed by FD.io test executor dpdk performance jobs that created result output files used as data sources for the graph, *Y-axis labels* are measured Packets Per Second [pps] values, and the *Graph legend* lists the plotted test suites and their indices. diff --git a/docs/report/introduction/methodology.rst b/docs/report/introduction/methodology.rst index 483cbb7295..ff5714c259 100644 --- a/docs/report/introduction/methodology.rst +++ b/docs/report/introduction/methodology.rst @@ -35,8 +35,286 @@ Ethernet): All rates are reported from external Traffic Generator perspective. -Description of MLRsearch algorithm is provided in -:ref:`mlrsearch_algorithm`. +.. _mlrsearch_algorithm: + +MLRsearch Algorithm +------------------- + +Multiple Loss Rate search (MLRsearch) is a new search algorithm +implemented in FD.io CSIT project. MLRsearch discovers multiple packet +throughput rates in a single search, with each rate associated with a +distinct Packet Loss Ratio (PLR) criteria. + +Two throughput measurements used in FD.io CSIT are Non-Drop Rate (NDR, +with zero packet loss, PLR=0) and Partial Drop Rate (PDR, with packet +loss rate not greater than the configured non-zero PLR). MLRsearch +discovers NDR and PDR in a single pass reducing required execution time +compared to separate binary searches for NDR and PDR. MLRsearch reduces +execution time even further by relying on shorter trial durations +of intermediate steps, with only the final measurements +conducted at the specified final trial duration. +This results in the shorter overall search +execution time when compared to a standard NDR/PDR binary search, +while guaranteeing the same or similar results. + +If needed, MLRsearch can be easily adopted to discover more throughput rates +with different pre-defined PLRs. + +.. Note:: All throughput rates are *always* bi-directional + aggregates of two equal (symmetric) uni-directional packet rates + received and reported by an external traffic generator. + +Overview +~~~~~~~~ + +The main properties of MLRsearch: + +- MLRsearch is a duration aware multi-phase multi-rate search algorithm. + + - Initial phase determines promising starting interval for the search. + - Intermediate phases progress towards defined final search criteria. + - Final phase executes measurements according to the final search + criteria. + +- *Initial phase*: + + - Uses link rate as a starting transmit rate and discovers the Maximum + Receive Rate (MRR) used as an input to the first intermediate phase. + +- *Intermediate phases*: + + - Start with initial trial duration (in the first phase) and converge + geometrically towards the final trial duration (in the final phase). + - Track two values for NDR and two for PDR. + + - The values are called (NDR or PDR) lower_bound and upper_bound. + - Each value comes from a specific trial measurement + (most recent for that transmit rate), + and as such the value is associated with that measurement's duration and loss. + - A bound can be invalid, for example if NDR lower_bound + has been measured with nonzero loss. + - Invalid bounds are not real boundaries for the searched value, + but are needed to track interval widths. + - Valid bounds are real boundaries for the searched value. + - Each non-initial phase ends with all bounds valid. + + - Start with a large (lower_bound, upper_bound) interval width and + geometrically converge towards the width goal (measurement resolution) + of the phase. Each phase halves the previous width goal. + - Use internal and external searches: + + - External search - measures at transmit rates outside the (lower_bound, + upper_bound) interval. Activated when a bound is invalid, + to search for a new valid bound by doubling the interval width. + It is a variant of `exponential search`_. + - Internal search - `binary search`_, measures at transmit rates within the + (lower_bound, upper_bound) valid interval, halving the interval width. + +- *Final phase* is executed with the final test trial duration, and the final + width goal that determines resolution of the overall search. + Intermediate phases together with the final phase are called non-initial phases. + +The main benefits of MLRsearch vs. binary search include: + +- In general MLRsearch is likely to execute more search trials overall, but + less trials at a set final duration. +- In well behaving cases it greatly reduces (>50%) the overall duration + compared to a single PDR (or NDR) binary search duration, + while finding multiple drop rates. +- In all cases MLRsearch yields the same or similar results to binary search. +- Note: both binary search and MLRsearch are susceptible to reporting + non-repeatable results across multiple runs for very bad behaving + cases. + +Caveats: + +- Worst case MLRsearch can take longer than a binary search e.g. in case of + drastic changes in behaviour for trials at varying durations. + +Search Implementation +~~~~~~~~~~~~~~~~~~~~~ + +Following is a brief description of the current MLRsearch +implementation in FD.io CSIT. + +Input Parameters +```````````````` + +#. *maximum_transmit_rate* - maximum packet transmit rate to be used by + external traffic generator, limited by either the actual Ethernet + link rate or traffic generator NIC model capabilities. Sample + defaults: 2 * 14.88 Mpps for 64B 10GE link rate, + 2 * 18.75 Mpps for 64B 40GE NIC maximum rate. +#. *minimum_transmit_rate* - minimum packet transmit rate to be used for + measurements. MLRsearch fails if lower transmit rate needs to be + used to meet search criteria. Default: 2 * 10 kpps (could be higher). +#. *final_trial_duration* - required trial duration for final rate + measurements. Default: 30 sec. +#. *initial_trial_duration* - trial duration for initial MLRsearch phase. + Default: 1 sec. +#. *final_relative_width* - required measurement resolution expressed as + (lower_bound, upper_bound) interval width relative to upper_bound. + Default: 0.5%. +#. *packet_loss_ratio* - maximum acceptable PLR search criteria for + PDR measurements. Default: 0.5%. +#. *number_of_intermediate_phases* - number of phases between the initial + phase and the final phase. Impacts the overall MLRsearch duration. + Less phases are required for well behaving cases, more phases + may be needed to reduce the overall search duration for worse behaving cases. + Default (2). (Value chosen based on limited experimentation to date. + More experimentation needed to arrive to clearer guidelines.) + +Initial phase +````````````` + +1. First trial measures at maximum rate and discovers MRR. + + a. *in*: trial_duration = initial_trial_duration. + b. *in*: offered_transmit_rate = maximum_transmit_rate. + c. *do*: single trial. + d. *out*: measured loss ratio. + e. *out*: mrr = measured receive rate. + +2. Second trial measures at MRR and discovers MRR2. + + a. *in*: trial_duration = initial_trial_duration. + b. *in*: offered_transmit_rate = MRR. + c. *do*: single trial. + d. *out*: measured loss ratio. + e. *out*: mrr2 = measured receive rate. + +3. Third trial measures at MRR2. + + a. *in*: trial_duration = initial_trial_duration. + b. *in*: offered_transmit_rate = MRR2. + c. *do*: single trial. + d. *out*: measured loss ratio. + +Non-initial phases +`````````````````` + +1. Main loop: + + a. *in*: trial_duration for the current phase. + Set to initial_trial_duration for the first intermediate phase; + to final_trial_duration for the final phase; + or to the element of interpolating geometric sequence + for other intermediate phases. + For example with two intermediate phases, trial_duration + of the second intermediate phase is the geometric average + of initial_strial_duration and final_trial_duration. + b. *in*: relative_width_goal for the current phase. + Set to final_relative_width for the final phase; + doubled for each preceding phase. + For example with two intermediate phases, + the first intermediate phase uses quadruple of final_relative_width + and the second intermediate phase uses double of final_relative_width. + c. *in*: ndr_interval, pdr_interval from the previous main loop iteration + or the previous phase. + If the previous phase is the initial phase, both intervals have + lower_bound = MRR2, uper_bound = MRR. + Note that the initial phase is likely to create intervals with invalid bounds. + d. *do*: According to the procedure described in point 2, + either exit the phase (by jumping to 1.g.), + or prepare new transmit rate to measure with. + e. *do*: Perform the trial measurement at the new transmit rate + and trial_duration, compute its loss ratio. + f. *do*: Update the bounds of both intervals, based on the new measurement. + The actual update rules are numerous, as NDR external search + can affect PDR interval and vice versa, but the result + agrees with rules of both internal and external search. + For example, any new measurement below an invalid lower_bound + becomes the new lower_bound, while the old measurement + (previously acting as the invalid lower_bound) + becomes a new and valid upper_bound. + Go to next iteration (1.c.), taking the updated intervals as new input. + g. *out*: current ndr_interval and pdr_interval. + In the final phase this is also considered + to be the result of the whole search. + For other phases, the next phase loop is started + with the current results as an input. + +2. New transmit rate (or exit) calculation (for 1.d.): + + - If there is an invalid bound then prepare for external search: + + - *If* the most recent measurement at NDR lower_bound transmit rate + had the loss higher than zero, then + the new transmit rate is NDR lower_bound + decreased by two NDR interval widths. + - Else, *if* the most recent measurement at PDR lower_bound + transmit rate had the loss higher than PLR, then + the new transmit rate is PDR lower_bound + decreased by two PDR interval widths. + - Else, *if* the most recent measurement at NDR upper_bound + transmit rate had no loss, then + the new transmit rate is NDR upper_bound + increased by two NDR interval widths. + - Else, *if* the most recent measurement at PDR upper_bound + transmit rate had the loss lower or equal to PLR, then + the new transmit rate is PDR upper_bound + increased by two PDR interval widths. + - If interval width is higher than the current phase goal: + + - Else, *if* NDR interval does not meet the current phase width goal, + prepare for internal search. The new transmit rate is + (NDR lower bound + NDR upper bound) / 2. + - Else, *if* PDR interval does not meet the current phase width goal, + prepare for internal search. The new transmit rate is + (PDR lower bound + PDR upper bound) / 2. + - Else, *if* some bound has still only been measured at a lower duration, + prepare to re-measure at the current duration (and the same transmit rate). + The order of priorities is: + + - NDR lower_bound, + - PDR lower_bound, + - NDR upper_bound, + - PDR upper_bound. + - *Else*, do not prepare any new rate, to exit the phase. + This ensures that at the end of each non-initial phase + all intervals are valid, narrow enough, and measured + at current phase trial duration. + +Implementation Deviations +~~~~~~~~~~~~~~~~~~~~~~~~~ + +This document so far has been describing a simplified version of MLRsearch algorithm. +The full algorithm as implemented contains additional logic, +which makes some of the details (but not general ideas) above incorrect. +Here is a short description of the additional logic as a list of principles, +explaining their main differences from (or additions to) the simplified description, +but without detailing their mutual interaction. + +1. *Logarithmic transmit rate.* + In order to better fit the relative width goal, + the interval doubling and halving is done differently. + For example, the middle of 2 and 8 is 4, not 5. +2. *Optimistic maximum rate.* + The increased rate is never higher than the maximum rate. + Upper bound at that rate is always considered valid. +3. *Pessimistic minimum rate.* + The decreased rate is never lower than the minimum rate. + If a lower bound at that rate is invalid, + a phase stops refining the interval further (until it gets re-measured). +4. *Conservative interval updates.* + Measurements above current upper bound never update a valid upper bound, + even if drop ratio is low. + Measurements below current lower bound always update any lower bound + if drop ratio is high. +5. *Ensure sufficient interval width.* + Narrow intervals make external search take more time to find a valid bound. + If the new transmit increased or decreased rate would result in width + less than the current goal, increase/decrease more. + This can happen if the measurement for the other interval + makes the current interval too narrow. + Similarly, take care the measurements in the initial phase + create wide enough interval. +6. *Timeout for bad cases.* + The worst case for MLRsearch is when each phase converges to intervals + way different than the results of the previous phase. + Rather than suffer total search time several times larger + than pure binary search, the implemented tests fail themselves + when the search takes too long (given by argument *timeout*). Maximum Receive Rate MRR ------------------------ @@ -391,3 +669,8 @@ The initial tests are designed as follows: - Connection close after set test duration time. - Resulting flow sequence: >Syn, <Syn-Ack, >Ack, >Req[1], <Rep[1], .., >Req[n], <Rep[n], >Fin, <Fin, >Ack. + +.. _binary search: https://en.wikipedia.org/wiki/Binary_search +.. _exponential search: https://en.wikipedia.org/wiki/Exponential_search +.. _estimation of standard deviation: https://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation +.. _simplified error propagation formula: https://en.wikipedia.org/wiki/Propagation_of_uncertainty#Simplification diff --git a/docs/report/vpp_performance_tests/documentation/index.rst b/docs/report/vpp_performance_tests/documentation/index.rst index 39a32ba74f..200ac409f3 100644 --- a/docs/report/vpp_performance_tests/documentation/index.rst +++ b/docs/report/vpp_performance_tests/documentation/index.rst @@ -4,6 +4,5 @@ Documentation .. toctree:: containers - mlr_search documentation diff --git a/docs/report/vpp_performance_tests/documentation/mlr_search.rst b/docs/report/vpp_performance_tests/documentation/mlr_search.rst deleted file mode 100644 index 1a093f5ba6..0000000000 --- a/docs/report/vpp_performance_tests/documentation/mlr_search.rst +++ /dev/null @@ -1,285 +0,0 @@ -.. _mlrsearch_algorithm: - -MLRsearch Algorithm -=================== - -Multiple Loss Rate search (MLRsearch) is a new search algorithm -implemented in FD.io CSIT project. MLRsearch discovers multiple packet -throughput rates in a single search, with each rate associated with a -distinct Packet Loss Ratio (PLR) criteria. - -Two throughput measurements used in FD.io CSIT are Non-Drop Rate (NDR, -with zero packet loss, PLR=0) and Partial Drop Rate (PDR, with packet -loss rate not greater than the configured non-zero PLR). MLRsearch -discovers NDR and PDR in a single pass reducing required execution time -compared to separate binary searches for NDR and PDR. MLRsearch reduces -execution time even further by relying on shorter trial durations -of intermediate steps, with only the final measurements -conducted at the specified final trial duration. -This results in the shorter overall search -execution time when compared to a standard NDR/PDR binary search, -while guaranteeing the same or similar results. - -If needed, MLRsearch can be easily adopted to discover more throughput rates -with different pre-defined PLRs. - -.. Note:: All throughput rates are *always* bi-directional - aggregates of two equal (symmetric) uni-directional packet rates - received and reported by an external traffic generator. - -Overview ---------- - -The main properties of MLRsearch: - -- MLRsearch is a duration aware multi-phase multi-rate search algorithm. - - - Initial phase determines promising starting interval for the search. - - Intermediate phases progress towards defined final search criteria. - - Final phase executes measurements according to the final search - criteria. - -- *Initial phase*: - - - Uses link rate as a starting transmit rate and discovers the Maximum - Receive Rate (MRR) used as an input to the first intermediate phase. - -- *Intermediate phases*: - - - Start with initial trial duration (in the first phase) and converge - geometrically towards the final trial duration (in the final phase). - - Track two values for NDR and two for PDR. - - - The values are called (NDR or PDR) lower_bound and upper_bound. - - Each value comes from a specific trial measurement - (most recent for that transmit rate), - and as such the value is associated with that measurement's duration and loss. - - A bound can be invalid, for example if NDR lower_bound - has been measured with nonzero loss. - - Invalid bounds are not real boundaries for the searched value, - but are needed to track interval widths. - - Valid bounds are real boundaries for the searched value. - - Each non-initial phase ends with all bounds valid. - - - Start with a large (lower_bound, upper_bound) interval width and - geometrically converge towards the width goal (measurement resolution) - of the phase. Each phase halves the previous width goal. - - Use internal and external searches: - - - External search - measures at transmit rates outside the (lower_bound, - upper_bound) interval. Activated when a bound is invalid, - to search for a new valid bound by doubling the interval width. - It is a variant of `exponential search`_. - - Internal search - `binary search`_, measures at transmit rates within the - (lower_bound, upper_bound) valid interval, halving the interval width. - -- *Final phase* is executed with the final test trial duration, and the final - width goal that determines resolution of the overall search. - Intermediate phases together with the final phase are called non-initial phases. - -The main benefits of MLRsearch vs. binary search include: - -- In general MLRsearch is likely to execute more search trials overall, but - less trials at a set final duration. -- In well behaving cases it greatly reduces (>50%) the overall duration - compared to a single PDR (or NDR) binary search duration, - while finding multiple drop rates. -- In all cases MLRsearch yields the same or similar results to binary search. -- Note: both binary search and MLRsearch are susceptible to reporting - non-repeatable results across multiple runs for very bad behaving - cases. - -Caveats: - -- Worst case MLRsearch can take longer than a binary search e.g. in case of - drastic changes in behaviour for trials at varying durations. - -Search Implementation ---------------------- - -Following is a brief description of the current MLRsearch -implementation in FD.io CSIT. - -Input Parameters -```````````````` - -#. *maximum_transmit_rate* - maximum packet transmit rate to be used by - external traffic generator, limited by either the actual Ethernet - link rate or traffic generator NIC model capabilities. Sample - defaults: 2 * 14.88 Mpps for 64B 10GE link rate, - 2 * 18.75 Mpps for 64B 40GE NIC maximum rate. -#. *minimum_transmit_rate* - minimum packet transmit rate to be used for - measurements. MLRsearch fails if lower transmit rate needs to be - used to meet search criteria. Default: 2 * 10 kpps (could be higher). -#. *final_trial_duration* - required trial duration for final rate - measurements. Default: 30 sec. -#. *initial_trial_duration* - trial duration for initial MLRsearch phase. - Default: 1 sec. -#. *final_relative_width* - required measurement resolution expressed as - (lower_bound, upper_bound) interval width relative to upper_bound. - Default: 0.5%. -#. *packet_loss_ratio* - maximum acceptable PLR search criteria for - PDR measurements. Default: 0.5%. -#. *number_of_intermediate_phases* - number of phases between the initial - phase and the final phase. Impacts the overall MLRsearch duration. - Less phases are required for well behaving cases, more phases - may be needed to reduce the overall search duration for worse behaving cases. - Default (2). (Value chosen based on limited experimentation to date. - More experimentation needed to arrive to clearer guidelines.) - -Initial phase -````````````` - -1. First trial measures at maximum rate and discovers MRR. - - a. *in*: trial_duration = initial_trial_duration. - b. *in*: offered_transmit_rate = maximum_transmit_rate. - c. *do*: single trial. - d. *out*: measured loss ratio. - e. *out*: mrr = measured receive rate. - -2. Second trial measures at MRR and discovers MRR2. - - a. *in*: trial_duration = initial_trial_duration. - b. *in*: offered_transmit_rate = MRR. - c. *do*: single trial. - d. *out*: measured loss ratio. - e. *out*: mrr2 = measured receive rate. - -3. Third trial measures at MRR2. - - a. *in*: trial_duration = initial_trial_duration. - b. *in*: offered_transmit_rate = MRR2. - c. *do*: single trial. - d. *out*: measured loss ratio. - -Non-initial phases -`````````````````` - -1. Main loop: - - a. *in*: trial_duration for the current phase. - Set to initial_trial_duration for the first intermediate phase; - to final_trial_duration for the final phase; - or to the element of interpolating geometric sequence - for other intermediate phases. - For example with two intermediate phases, trial_duration - of the second intermediate phase is the geometric average - of initial_strial_duration and final_trial_duration. - b. *in*: relative_width_goal for the current phase. - Set to final_relative_width for the final phase; - doubled for each preceding phase. - For example with two intermediate phases, - the first intermediate phase uses quadruple of final_relative_width - and the second intermediate phase uses double of final_relative_width. - c. *in*: ndr_interval, pdr_interval from the previous main loop iteration - or the previous phase. - If the previous phase is the initial phase, both intervals have - lower_bound = MRR2, uper_bound = MRR. - Note that the initial phase is likely to create intervals with invalid bounds. - d. *do*: According to the procedure described in point 2, - either exit the phase (by jumping to 1.g.), - or prepare new transmit rate to measure with. - e. *do*: Perform the trial measurement at the new transmit rate - and trial_duration, compute its loss ratio. - f. *do*: Update the bounds of both intervals, based on the new measurement. - The actual update rules are numerous, as NDR external search - can affect PDR interval and vice versa, but the result - agrees with rules of both internal and external search. - For example, any new measurement below an invalid lower_bound - becomes the new lower_bound, while the old measurement - (previously acting as the invalid lower_bound) - becomes a new and valid upper_bound. - Go to next iteration (1.c.), taking the updated intervals as new input. - g. *out*: current ndr_interval and pdr_interval. - In the final phase this is also considered - to be the result of the whole search. - For other phases, the next phase loop is started - with the current results as an input. - -2. New transmit rate (or exit) calculation (for 1.d.): - - - If there is an invalid bound then prepare for external search: - - - *If* the most recent measurement at NDR lower_bound transmit rate - had the loss higher than zero, then - the new transmit rate is NDR lower_bound - decreased by two NDR interval widths. - - Else, *if* the most recent measurement at PDR lower_bound - transmit rate had the loss higher than PLR, then - the new transmit rate is PDR lower_bound - decreased by two PDR interval widths. - - Else, *if* the most recent measurement at NDR upper_bound - transmit rate had no loss, then - the new transmit rate is NDR upper_bound - increased by two NDR interval widths. - - Else, *if* the most recent measurement at PDR upper_bound - transmit rate had the loss lower or equal to PLR, then - the new transmit rate is PDR upper_bound - increased by two PDR interval widths. - - If interval width is higher than the current phase goal: - - - Else, *if* NDR interval does not meet the current phase width goal, - prepare for internal search. The new transmit rate is - (NDR lower bound + NDR upper bound) / 2. - - Else, *if* PDR interval does not meet the current phase width goal, - prepare for internal search. The new transmit rate is - (PDR lower bound + PDR upper bound) / 2. - - Else, *if* some bound has still only been measured at a lower duration, - prepare to re-measure at the current duration (and the same transmit rate). - The order of priorities is: - - - NDR lower_bound, - - PDR lower_bound, - - NDR upper_bound, - - PDR upper_bound. - - *Else*, do not prepare any new rate, to exit the phase. - This ensures that at the end of each non-initial phase - all intervals are valid, narrow enough, and measured - at current phase trial duration. - -Implementation Deviations -------------------------- - -This document so far has been describing a simplified version of MLRsearch algorithm. -The full algorithm as implemented contains additional logic, -which makes some of the details (but not general ideas) above incorrect. -Here is a short description of the additional logic as a list of principles, -explaining their main differences from (or additions to) the simplified description, -but without detailing their mutual interaction. - -1. *Logarithmic transmit rate.* - In order to better fit the relative width goal, - the interval doubling and halving is done differently. - For example, the middle of 2 and 8 is 4, not 5. -2. *Optimistic maximum rate.* - The increased rate is never higher than the maximum rate. - Upper bound at that rate is always considered valid. -3. *Pessimistic minimum rate.* - The decreased rate is never lower than the minimum rate. - If a lower bound at that rate is invalid, - a phase stops refining the interval further (until it gets re-measured). -4. *Conservative interval updates.* - Measurements above current upper bound never update a valid upper bound, - even if drop ratio is low. - Measurements below current lower bound always update any lower bound - if drop ratio is high. -5. *Ensure sufficient interval width.* - Narrow intervals make external search take more time to find a valid bound. - If the new transmit increased or decreased rate would result in width - less than the current goal, increase/decrease more. - This can happen if the measurement for the other interval - makes the current interval too narrow. - Similarly, take care the measurements in the initial phase - create wide enough interval. -6. *Timeout for bad cases.* - The worst case for MLRsearch is when each phase converges to intervals - way different than the results of the previous phase. - Rather than suffer total search time several times larger - than pure binary search, the implemented tests fail themselves - when the search takes too long (given by argument *timeout*). - -.. _binary search: https://en.wikipedia.org/wiki/Binary_search -.. _exponential search: https://en.wikipedia.org/wiki/Exponential_search -.. _estimation of standard deviation: https://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation -.. _simplified error propagation formula: https://en.wikipedia.org/wiki/Propagation_of_uncertainty#Simplification diff --git a/docs/report/vpp_performance_tests/packet_latency_graphs/container_memif.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/container_memif.rst index 230b1045b1..0f90e0c832 100644 --- a/docs/report/vpp_performance_tests/packet_latency_graphs/container_memif.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/container_memif.rst @@ -2,7 +2,7 @@ LXC/DRC Container Memif ======================= This section includes summary graphs of VPP Phy-to-Phy packet latency -with Container memif Connections measured at 50% of discovered NDR throughput +with Container memif Connections measured at 100% of discovered NDR throughput rate. Latency is reported for VPP running in multiple configurations of VPP worker thread(s), a.k.a. VPP data plane thread(s), and their physical CPU core(s) placement. diff --git a/docs/report/vpp_performance_tests/packet_latency_graphs/container_orchestrated.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/container_orchestrated.rst index cda1d3698a..8f5f6538e1 100644 --- a/docs/report/vpp_performance_tests/packet_latency_graphs/container_orchestrated.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/container_orchestrated.rst @@ -2,7 +2,7 @@ K8s Container Memif =================== This section includes summary graphs of VPP Phy-to-Phy packet latency -with CContainer Orchestrated Topologies measured at 50% of discovered NDR +with CContainer Orchestrated Topologies measured at 100% of discovered NDR throughput rate. Latency is reported for VPP running in multiple configurations of VPP worker thread(s), a.k.a. VPP data plane thread(s), and their physical CPU core(s) placement. diff --git a/docs/report/vpp_performance_tests/packet_latency_graphs/index.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/index.rst index ab44222f63..878df78ea8 100644 --- a/docs/report/vpp_performance_tests/packet_latency_graphs/index.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/index.rst @@ -5,14 +5,14 @@ Plotted results are generated from a single execution of CSIT NDR discovery test. Box plots are used to show the Minimum, Median and Maximum packet latency per test. -*Title of each graph* is a regex (regular expression) matching all -throughput test cases plotted on this graph, *X-axis labels* are indices -of individual test suites executed by -FD.io test executor vpp performance jobs that created result output file -used as data source for the graph, *Y-axis labels* are measured packet Latency -[uSec] values, and the *Graph legend* lists the plotted test suites and their -indices. Latency is reported for concurrent symmetric bi-directional flows, -separately for each direction: i) West-to-East: +*Title of each graph* describes the packet path, testbed topology, processor +architecture, packet size, number of cores and threads used by workers and +DUT configuration, *X-axis labels* are indices of individual test suites +executed by FD.io test executor vpp performance jobs that created result output +file used as data source for the graph, *Y-axis labels* are measured packet +Latency [uSec] values, and the *Graph legend* lists the plotted test suites and +their indices. Latency is reported for concurrent symmetric bi-directional +flows, separately for each direction: i) West-to-East: TGint1-to-SUT1-to-SUT2-to-TGint2, and ii) East-to-West: TGint2-to-SUT2-to-SUT1-to-TGint1. diff --git a/docs/report/vpp_performance_tests/packet_latency_graphs/ip4.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/ip4.rst index a7f352498e..5864a934fb 100644 --- a/docs/report/vpp_performance_tests/packet_latency_graphs/ip4.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/ip4.rst @@ -2,7 +2,7 @@ IPv4 Routing ============ This section includes summary graphs of VPP Phy-to-Phy packet latency -with IPv4 Routed-Forwarding measured at 50% of discovered NDR throughput +with IPv4 Routed-Forwarding measured at 100% of discovered NDR throughput rate. Latency is reported for VPP running in multiple configurations of VPP worker thread(s), a.k.a. VPP data plane thread(s), and their physical CPU core(s) placement. diff --git a/docs/report/vpp_performance_tests/packet_latency_graphs/ip4_tunnels.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/ip4_tunnels.rst index e45755e684..ca444502d3 100644 --- a/docs/report/vpp_performance_tests/packet_latency_graphs/ip4_tunnels.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/ip4_tunnels.rst @@ -2,7 +2,7 @@ IPv4 Tunnels ============ This section includes summary graphs of VPP Phy-to-Phy packet latency -with IPv4 Overlay Tunnels measured at 50% of discovered NDR throughput +with IPv4 Overlay Tunnels measured at 100% of discovered NDR throughput rate. Latency is reported for VPP running in multiple configurations of VPP worker thread(s), a.k.a. VPP data plane thread(s), and their physical CPU core(s) placement. diff --git a/docs/report/vpp_performance_tests/packet_latency_graphs/ip6.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/ip6.rst index 4640d6ed7a..2d0c23839b 100644 --- a/docs/report/vpp_performance_tests/packet_latency_graphs/ip6.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/ip6.rst @@ -2,7 +2,7 @@ IPv6 Routing ============ This section includes summary graphs of VPP Phy-to-Phy packet latency -with IPv6 Routed-Forwarding measured at 50% of discovered NDR throughput +with IPv6 Routed-Forwarding measured at 100% of discovered NDR throughput rate. Latency is reported for VPP running in multiple configurations of VPP worker thread(s), a.k.a. VPP data plane thread(s), and their physical CPU core(s) placement. diff --git a/docs/report/vpp_performance_tests/packet_latency_graphs/ip6_tunnels.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/ip6_tunnels.rst index d86db3e45d..60b4d61e59 100644 --- a/docs/report/vpp_performance_tests/packet_latency_graphs/ip6_tunnels.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/ip6_tunnels.rst @@ -2,7 +2,7 @@ IPv6 Tunnels ============ This section includes summary graphs of VPP Phy-to-Phy packet latency -with IPv6 Overlay Tunnels measured at 50% of discovered NDR throughput +with IPv6 Overlay Tunnels measured at 100% of discovered NDR throughput rate. Latency is reported for VPP running in multiple configurations of VPP worker thread(s), a.k.a. VPP data plane thread(s), and their physical CPU core(s) placement. diff --git a/docs/report/vpp_performance_tests/packet_latency_graphs/ipsec.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/ipsec.rst index bd7a871c1b..c4797e1e31 100644 --- a/docs/report/vpp_performance_tests/packet_latency_graphs/ipsec.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/ipsec.rst @@ -3,7 +3,7 @@ IPSec IPv4 Routing This section includes summary graphs of VPP Phy-to-Phy packet latency with IPSec encryption used in combination with IPv4 routed-forwarding, -with latency measured at 50% of discovered NDR throughput rate. VPP +with latency measured at 100% of discovered NDR throughput rate. VPP IPSec encryption is accelerated using DPDK cryptodev library driving Intel Quick Assist (QAT) crypto PCIe hardware cards. Latency is reported for VPP running in multiple configurations of VPP worker thread(s), diff --git a/docs/report/vpp_performance_tests/packet_latency_graphs/l2.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/l2.rst index 4638ef2325..53af627195 100644 --- a/docs/report/vpp_performance_tests/packet_latency_graphs/l2.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/l2.rst @@ -2,7 +2,7 @@ L2 Ethernet Switching ===================== This section includes summary graphs of VPP Phy-to-Phy packet latency -with L2 Ethernet switching measured at 50% of discovered NDR throughput +with L2 Ethernet switching measured at 100% of discovered NDR throughput rate. Latency is reported for VPP running in multiple configurations of VPP worker thread(s), a.k.a. VPP data plane thread(s), and their physical CPU core(s) placement. diff --git a/docs/report/vpp_performance_tests/packet_latency_graphs/srv6.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/srv6.rst index 2cb010f8b1..a7084c24d4 100644 --- a/docs/report/vpp_performance_tests/packet_latency_graphs/srv6.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/srv6.rst @@ -2,7 +2,7 @@ SRv6 Routing ============ This section includes summary graphs of VPP Phy-to-Phy packet latency -with SRv6 measured at 50% of discovered NDR throughput +with SRv6 measured at 100% of discovered NDR throughput rate. Latency is reported for VPP running in multiple configurations of VPP worker thread(s), a.k.a. VPP data plane thread(s), and their physical CPU core(s) placement. diff --git a/docs/report/vpp_performance_tests/packet_latency_graphs/vm_vhost.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/vm_vhost.rst index dbeca9e75b..a756bd69c8 100644 --- a/docs/report/vpp_performance_tests/packet_latency_graphs/vm_vhost.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/vm_vhost.rst @@ -3,7 +3,7 @@ KVM VMs vhost-user This section includes summary graphs of VPP Phy-to-VM(s)-to-Phy packet latency with with VM virtio and VPP vhost-user virtual interfaces -measured at 50% of discovered NDR throughput rate. Latency is reported +measured at 100% of discovered NDR throughput rate. Latency is reported for VPP running in multiple configurations of VPP worker thread(s), a.k.a. VPP data plane thread(s), and their physical CPU core(s) placement. diff --git a/docs/report/vpp_performance_tests/packet_latency_graphs/vts.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/vts.rst index 977b926dcf..15d5309ddd 100644 --- a/docs/report/vpp_performance_tests/packet_latency_graphs/vts.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/vts.rst @@ -3,7 +3,7 @@ VTS This section includes summary graphs of VPP Phy-to-VM(s)-to-Phy packet latency with with VM virtio and VPP vhost-user virtual interfaces -measured at 50% of discovered NDR throughput rate. Latency is reported +measured at 100% of discovered NDR throughput rate. Latency is reported for VPP running in multiple configurations of VPP worker thread(s), a.k.a. VPP data plane thread(s), and their physical CPU core(s) placement. diff --git a/docs/report/vpp_performance_tests/packet_throughput_graphs/index.rst b/docs/report/vpp_performance_tests/packet_throughput_graphs/index.rst index aa03a669f1..5c11206175 100644 --- a/docs/report/vpp_performance_tests/packet_throughput_graphs/index.rst +++ b/docs/report/vpp_performance_tests/packet_throughput_graphs/index.rst @@ -17,10 +17,10 @@ top edge of the box. If multiple samples match only two values, and all samples fall between them, then no whiskers are plotted. If all samples have the same value, only a horizontal line is plotted. -*Title of each graph* is a regex (regular expression) matching all -throughput test cases plotted on this graph, *X-axis labels* are indices -of individual test suites executed by -FD.io test executor vpp performance jobs that created result output +*Title of each graph* describes the packet path, testbed topology, processor +architecture, packet size, number of cores and threads used by workers and +DUT configuration, *X-axis labels* are indices of individual test suites +executed by FD.io test executor vpp performance jobs that created result output files used as data sources for the graph, *Y-axis labels* are measured Packets Per Second [pps] values, and the *Graph legend* lists the plotted test suites and their indices. |